Test-aware microservices (deep testing)

The following article describes a specific approach for testing single operations and complex workflows in a microservice-based architecture. The typical scenario will be the following:

• Cloud farm infrastructure
• REST-based microservice architecture
• Internal and public microservices
• Single-entry point: API gateway

The following diagram depicts all the mentioned aspects of the tested architecture:

The article attempts explaining the benefits of designing testable microservices from the ground up rather than using heterogeneous strategies for testing them. It also offers an alternative for testing servers without recurring to the User Interface.

The proposal is untied to a specific category of tests and can be used in, but not constrained to:

• Functional testing
• Performance testing
• Integration testing
• Exploratory testing
• Contract testing
• End-to-end testing

Testing of monolithic versus microservices architectures

The testing of a microservice architecture differs from monolithic systems in several aspects related to the inherent features of both design styles:

• Monolithic systems typically interchange information at memory level while microservices do it by using the network infrastructure.
• The data is handled in a single database in monolithic systems while each microservice may have an isolated data repository.
• Data models (POCO/POJO objects) are defined in a single library while each microservice defines its own, following the separation-of-concerns (SoC) principle.
• Monoliths share a single code-base and development rules while microservice components are built asynchronously and possibly using different tools.

Considering those aspects and despite the difficulty of handling changes in a monolithic system performed by different development teams, if the system is big enough, it is easier to test server applications designed as monoliths.

A common challenge for both architectures is the testing of workflows: The test is not unitary and needs to keep track of the data flowing across different stages (multiple input and output data packages).

Operations and workflows

A microservices-based system is divided into areas of responsibility: each microservice performs a different task, using its own process and persisting in its own storage. This means that even a simple operation will traverse several public and internal services.

For example, a data retrieval operation can touch the following microservices before returning to the client-side:

1. API Gateway
2. Authorization
3. Specific data service
4. Billing
5. Audit
6. Analytics

From the client perspective, an operation is simple and unitary, as it is unaware of all the internal traffic. I like describing this process as a Breakout game, bouncing internally until touching all the required microservices, as shown in the following animation:

A workflow will perform similar processing multiple times, as it was bouncing in the white paddle. The unitary operations will be linked somehow by a transaction or operation number.

Testing techniques

Data accessibility

There are multiple techniques for accessing data for testing purposes, among others:

• Using the UI for accessing server information
• Performing API calls to the gateway and reading the response
• Using server monitoring tools to sniff HTTP calls to microservices
• Examining HTTP log files at each server
• Querying the databases directly

An important factor is the location of the test harness: if it is located at the client-side, there will be more challenges trying to verify the operations performed in the most inner part of the cloud farm. On the contrary, tests controlled somewhere on the server-side have better access to internal microservices.

Microservice instances and test dependencies

A test can be performed over both production and test environments. For a microservice architecture, it is possible to combine several techniques like:

• A production microservice invoking a test microservice, or vice-versa
• A mock instance providing minimal functionality for the test purposes
• A stub instance providing specific data for the test, either static or dynamic

A combination of the techniques mentioned above will define the general testing strategy.

Unifying all of it

Test initiation

As mentioned before, a test can be launched from the client- or server-side, with different pros and cons. One of the biggest challenges for performing deep server testing is that secure systems isolate as most services as possible from the client. Those services are called here 'internal' as opposed to 'public'.

Although it is easy to start a test from the client-side, by activating a UI operation or sending an API call using an HTTP tool like Postman, retrieving intermediate results from messages bouncing across internal services can discourage the client-side approach.

The following diagram depicts the launch scenarios mentioned above:

The HTTP handler

Retrieving all the calls associated with a specific test-case requires monitoring the HTTP activity in the involved microservices. Cloud services, server VMs or Docker containers provide an alternative for logging the HTTP requests and responses, but the information may be unstructured, insufficient and difficult to retrieve and maintain (read https://httpd.apache.org/docs/2.4/logs.html).

All the major server frameworks offer the alternative to put an HTTP handler (H) in the request and response pipelines. This allows us to include a proprietary logging mechanism, as shown in the picture.

The logged calls may be constrained to those calls related to a specific operation or workflow transaction to be tested and stored along with some relevant data like server name (instance), microservice name, operation or transaction identifier, date/time, etc. along with the HTTP message content.

It will be necessary to send some identifiers for the operation or transaction being tested for enabling the logging and facilitate the grouping of information for every test. HTTP custom headers may cause a low impact. For example, all calls related to a single test instance:

• X-Test-Case-ID: F6456
• X-Test-Instance: da27f70c-b2f3-46be-b896-606fd6fe0287

These headers shall be propagated to all the derived calls to other microservices related to the same operation. This feature is crucial for a microservice to become testable.

The Testing microservice

The responsibility of storing and retrieving the testing logs shall lie on a component accessible from all microservices and the test controller. This is a good candidate for a new microservice that we will call the Testing Microservice, as described in the diagram below:

This microservice's API can support the following operations, among others:

• Record HTTP request and response operations (URL, verb, headers, body) from other microservices
• Retrieve all the operations performed for a single test instance
• Handle call templates (see next section)
• Determine the test result based on a call template
• Remove old test data

Call templates

A sequence of an expected HTTP request and response calls can be stored in a template associated with a test case, and then compared against the bunch of calls associated with a test instance.

A call template can specify expected strings in the fundamental parts of HTTP calls (URL, verb, headers, body) as literals or variables.

For a simple case, it may be enough to check for fixed URL strings and HTTP response codes. More complex templates can indicate variations in the URL (usually resource ID), specific body parts in XML, JSON or even binary streams, or a range of response codes like 3XX.

Regular expressions may help to evaluate variable texts and also preserve them into named capturing groups (see https://www.regular-expressions.info/named.html) for being reused in further expressions.

Following the operation mentioned in the Operations and Workflow section, a call template in JSON format may look like this:

Notice the template above belongs to a single client-side operation. However, it can be extended to support entire workflows by adding more requests from the client-side and keep posting the same HTTP header for the Test-Instance or another transaction identifier.

Other tests may contain additional constraints like:

• Field-specific values and value ranges in text bodies like XML and JSON
• Time spans per call or test case, for performance tests
• Schema validation for contract tests
• Golden image files for comparing binary results (raw data, images)
• Diagnostic-only values passed as HTTP headers

Differences with other API testing tools

There are several API testing tools that can automate the calls to one or many microservices by reading a template and compare the result code or content, but they are limited to the public scope.

Using the sample case above, performing GET /document/123 and receiving a 200 response may be insufficient, although satisfactory for common API testing tools. This operation won't tell if the workflow went through the internal Authorization, Billing and Audit microservices behind the Gateway.

Conclusions

An implementation made of uniform microservices can be extended to support an explicit testing infrastructure as opposite to ad-hoc and heuristic testing techniques.

A dedicated microservice can handle (store, retrieve, evaluate) the test data flowing deep inside a cloud farm, which may not accessible from the client-side.

Call templates are a simple and effective way to evaluate the result of tests (a bunch of HTTP calls) over either simple operations and complex workflows.