Unit Testing for Continuous Deployment

The first in a series of posts that will talk about building a deployment pipeline from the start to the finish, biased very much to delivering a set top box into a digital television system because that is what I do.

In this post I want to cover the entry point to the pipeline, which should be production quality code delivered to trunk. What I discovered whilst working through this with my teams is that unit testing is a very broad and misunderstood topic (including by me) so I want to try and define what it means to me today, and how that can be applied by development teams.

In amongst all the noise of the blogosphere there are some pretty good definitions of a unit test out there in the wild and I find that this one aligns to what I think we need as a quality gate at the start of the pipeline.

The tests must be fast to run and they have to be trustworthy to allow developers to deliver in complete safety. I would extend the definition to ensure that unit tests can be run at the desk by a developer in a very short period of time (cumulatively or incrementally). Unit tests will ideally be run many times during feature implementation as development teams check in changes and verify that there are no unexpected regressions.

The Legendary Martin Fowler blogged about unit testing in 2014. He pointed to some terminology introduced by Jay Fields and discusses Solitary Tests versus Sociable Tests as well as capturing some of the less desirable alternatives.

Timothy Fitz is also a great source of inspiration and has a lot of hands on experience at the sharp end of continuous deployment. He ran a workshop I recently attended and coined the phrase "Microtest" which was a useful way of discussing Unit Tests without all of the baggage that goes along with that definition. His blog carries the same message as quoted below.

(Unit tests) should be your bread and butter. I may be bastardizing the term “Unit Test” since I don’t care about how many classes they touch, I just mean they don’t touch external state and they don’t rely on a specific environment. That means they don’t use randomness or time either. If you follow those rules it’s actually hard to write a test. You should be able to run thousands in a few seconds.

Our goal is to improve the velocity of value through the pipeline, enabling many smaller changes to be delivered to trunk and continuously integrated - and all whilst maintaining the production quality code on trunk. We measure our success by development teams becoming empowered to deliver without fear.

So, to extend the definition a little, A good unit test is:

• Able to be fully automated
• Has full control over all the pieces running (Use mocks or stubs to achieve this isolation when needed)
• Can be run in any order if part of many other tests
• Runs sociably rather than in isolation
• Runs in memory (no DB or File access, for example)
• Consistently returns the same result (You always run the same test, so no random numbers, for example. save those for integration or range tests)
• Runs fast (1000s of tests in a few seconds)
• Tests a single logical concept in the system
• Readable
• Maintainable
• Trustworthy (when you see its result, you don’t need to debug the code just to be sure)

I'm very interested in the opinions and experiences of other teams regardless of whether you are testing embedded consumer devices, web applications, or something else.