Test Classification - An Objective approach towards a Subjective topic

                           PART1: InMemory Tests

Today it’s not only very important for all organisations to invest into the right amount of automated tests at all levels but it’s also very important that we do understand the boundaries of each test category and validate the right boundaries to get the desired confidence every time when each of these tests pass.

In my previous article regarding the Test Pyramid, One of key factors which I mentioned on which the success of your automation strategy would heavily rely upon is your understanding of the different tests that you are developing and at what point of your testing lifecycle you decide to execute them in order to get the most value out of them.

Also, the topic of test classification as I have found is highly subjective, vast and it differs from person to person and also varies depending on the type of application under test. From my past experiences, I have learnt that the best value you can bring out from these tests is by applying the objectivity of your application under test and figuring out which are the most valuable ones for the cause.

The classification from here on, is going to be a version to my objectivity which has always helped me in achieving the best results for the projects which I have managed.

Unit Tests: Unit tests have the narrowest scope of all tests in your test suite. The number of unit tests would largely outnumber any other type of test. Unit tests are heavily mocked.

As depicted in the diagram, Unit tests concentrate down to the level of individual functions/methods inside your class and any outside dependencies should be mocked.

Mocking candidates for Unit tests:

• other classes in same service
• Any external service calls
• Any databases, message brokers, cache stores etc.

Mocking Options for Unit tests:

• Mockito
• Jmockit
• In memory databases ( H2 Database ) etc.

Benefits:

• Fastest feedback and execution time
• Failures can directly point to malfunctioning Units
• Most helpful in providing confidence while refactoring major pieces of code

Integration Tests: Integration tests have a larger scope when compared to unit tests. These would be lesser in number than the unit tests but the coverage and boundary of these tests increases. Integration tests also rely on heavy mocking.

Unlike unit tests where we were concentrating on individual functions inside classes. Integration tests are targeted more at verifying functionalities that span across multiple classes and functions inside the service and also testing at functionalities which depend on external service responses for their completion. The mocking still remains heavy in these tests as well.

Mocking candidates for Integration tests:

• Any external service calls
• Any databases, message brokers, cache stores etc.

Mocking Options for Integration tests:

• Mockito
• Jmockit
• In memory databases ( H2 Database ) etc.

Benefits:

• Fast feedback
• Provides confidence on business logic at feature level
• Failures would still be agnostic of any external factors and would always point to internal malfunctioning code

Contract Tests: After gaining sufficient confidence within the service boundaries via the unit & integration tests, with contract tests we move on to the next phase and a very important phase of validation i.e. validating the contracts between services. These tests are targeted at validating that any change in the service logic doesn’t break any of the contracts that we have with the external world.

With the increasing number of microservices and increasing number of contracts between those services, it has been found more valuable & logical to look at these tests as ‘Consumer Driven Contract Tests aka CDC’s’.

As the name suggests, these tests are typically written by the consumer services and each consumer service has to make sure that they are validating the contracts that their service depends on for its own functional operation. The producer on the other hand has to make sure that if it changes any code, it has to validate that it doesn't break any contract which it has with all it’s consumers.

This tests as it seems from above are pretty complicated ones to manage, develop and to make sure that both consumers & producers are independently developed and released without the above breakages.

But thanks to the open source community we have a framework in ‘PACT.io’ which helps us develop and test CDC’s and integrates seamlessly with leading continuous integration systems.

The most important thing that we should remember while developing a CDC test is that we should only be concentrating on validating contracts and should avoid making any data validations like we generally do in any other type of test.

For e.g. Let’s take a simple scenario,

When service-a calls an service-b endpoint named ‘getUserDetails’ with a unique userid. service-b is supposed to respond back with a json containing the name, address and mobile of the requested user.

Now, service-a would be writing a CDC to validate this contract and the validation points in the CDC should check,

• If the response type is ‘application/json’
• If we have the required keys with the expected key names in the response i.e. name, address and mobile
• If the datatype of the values are as expected i.e. string, varchar, int, long etc.

What we should not validate or care about in CDC is,

• Is the returned name ‘Kohitij’ or ‘xyz’
• Does he live on ‘Street1’ or ‘abc’
• Is his mobile ‘1234’ or ‘9876’

The contracts tests are comparatively a newer concept if you compare to any other layers of tests. But are more valuable in terms of proactively flagging these contract breaches especially in a microservice heavy ecosystem.

NOTE: All the test categories that we covered till now i.e. unit tests, integration tests and contract tests did heavy use of mocks and because of which they are also called as ‘ in-memory’ tests i.e. they don’t require the service to be actually deployed and running on any instance for executing any of these tests. Hence these tests are very fast and part of build phase itself, letting us fail the deployable artefact generation incase of failures in any of the tests.

PART2: Testing against a deployed instance