Most Software Testing is Too Late

Everyone agree? Maybe not, perhaps all the testing you've seen/overseen/ heard of was completed at exactly the right time, but when was that: 10mins before live deployment? 10 days before? 10mins after? When all the build was completed and a representative live-like environment was available?

If you're a software quality professional, you might think you're always late to the party and having to pull those hours to get things done to hit a live date. If you're not, you might think testing is the last in the line of "Design-Build-Test" in the standard Software Development LifeCycle and always takes too long but it's in the right place.

OK, let's wind back, what am I getting at here? My original title for this article was "Let's talk about late inspection" so what is "late inspection" and what does it have to do with Software Testing being too late?

"Late inspection" is a term coined in response to the work of Dr D Edwards Deming. Deming is famous for what he learned in his work in Japanese manufacturing, which was later applied to US automotive manufacturing and which he detailed in his book "Quality Productivity and Competitive Position, Out of the Crisis "

Deming observed that some manufacturing plants put their quality processes at the end of the line, with a dedicated team trying to find and right all the wrongs the products (such as cars) had picked up along the way. The cliche is of QA engineers with big rubber mallets bashing body panels and doors to make them fit. This was slow, inefficient, but, most importantly did not produce a high quality product at the end.

"Inspection is too late. The quality, good or bad, is already in the product."

Software development is not strictly akin to manufacturing or assembly, these days it described as a form of engineering, it has an intellectual quality and the problems solved can differ from day to day and project to project. However, putting the inspection last and trying to test-the-quality-in is a failing that it frequently has in common with those 1960s and 70s assembly lines.

"But Tim, you're talking crazy: the thing must be built to be tested and it can't be built without a little design, even if it's a squiggle on a white board or a set of post its. So, Design Build, Test?"

I hear you say. Well, that sequencing isn't set in stone, but we'll come back to that.

Even in traditional waterfall development we have distinct test types in phases. We do not wait for every last thing to be built and jammed together before we begin testing. Like that assembly line we test components to make sure they meet tolerances before they get bolted together. We do Unit and Component testing. The thing is we tend not to do enough, perhaps just enough to get something to compile and deploy and often because we still think the "real" test is the integrated one: End to End, the Pre-Prod or the SIT.

Agile methods should make us think about testing earlier because "the test team" isn't an entity to be handed off to and should be engaged earlier. Still we see a lot of the work to prove or explore the quality and function of a system under development left until some time close to when it's planned to be deployed.

Some engineers will know and apply the test pyramid. Martin Fowler has perhaps done the most to popularise this concept. It simply states that most testing should be done at the cheaper and faster base of the pyramid where the components under test are small and unintegrated and the and fewer tests as you move towards the top where the UI (if there is one) and all integrating components are in place.

The testing pyramid is an article of faith in modern QA approaches, it supports that early inspection we're looking for; its anti pattern is an inverted pyramid or ice cream cone - where we see the least testing done at the early stages and increasing amounts of work, money and time being expended as we build into End to End tests.

There's more to this story: Doing more testing at the component end of the pyramid means doing more of the thinking up front, this might seem to be that dread, over used testing term "shift left"

But what it really means is the doing the thinking at the right time, testing early instead of testing late. Test Driven Development and its progeny Behaviour Driven Development both seek to achieve this by changing that sequence of Design, Build, Test. Sort of...

Testing before we build?

In these methods tests (or assertions) make up the design. We lose a whole translation step where a design is produced, interpreted one way by a developer, another way by a tester and then the 2 find out who was right.

Instead, the design is expressed as tests and then the code is written to pass those tests. Like the tests we'd have written post design, maybe post build have travelled back in time to.. The time when they can be most useful.

It can be argued that tests are always the best expression of the design: perhaps "just enough" design is done to build something, but to produce the tests and have a good expectation that once they have been applied and the resulting bugs correct, the software will be fit for live, you have to really examine the assumptions and fill in the blanks that were left.

Putting the thinking in at the right time pays huge dividends in: building a quality process (enabling Continuous integration and deployment), saving time, money, risk and adding predictability to our software. It allows exploratory types of testing to properly focus on discovery rather than finding what should be known or engineered out defects.

Adoption of design techniques like TDD and BDD or Model Based Testing are big moves in the right direction, but to move from our "Always late" model of testing does not require a huge change in the our ways of working it doesn't require us to adopt any particular agile method or tool set.

It can begin with the simplest of steps; make "how will I know that this is functional/performant/secure/resilient/Etc?" the first questions that come after "As a ----- I want-- because" then you will have the opportunity to see that the answers are examined at the first opportunity rather than the questions remaining unexplored until you lumber into the mythical lands of the Representative Test Environment.

That may be a land we visit in a future blog.