What is Model-Based Testing ?

MBT is where you create a mockup of the system being tested, along with a way to generate random, yet realistic, test data. The mockup’s job is to try to produce the same output as the actual system. Then the two outputs are compared.

Differences mean one of two things:

1. The model needs to be updated.

2. The actual system isn’t behaving as expected.

Let’s take a look at a simple example to further describe this.

Building the Data Generator

Having good test data is the first half of this method. But we also don’t want the effort to be spent creating a bunch of data sets. Instead, let’s make a data generator. The generator’s job will be to make data that the system would expect. It can’t be gibberish, otherwise we’ll be using a bunch invalid data, hence the “random yet realistic” description before.

Let’s say your system expects a person record in JSON format, with a name, address, city, state and ZIP code. The generator would then create valid JSON, but could choose to include (or not) each piece of info. The generator can be created with a switch statement, along with picking random values out of arrays of data (state, names, ZIP codes, etc.).

 Building the Model 

For the data generator example above, let’s say the system checks to see that a valid person record was provided, in the following ways:

1. Accept names less than or equal to 50 characters.

2. Accept addresses less than or equal to 35 characters.

3. Accept valid combinations of city, state, ZIP.

4. Throw appropriate error for any other case.

 Even if the system doing these checks is actually a collection of small microservices, that’s fine. A model can be built fairly simply, and take the form of a single if/else statement. 

Comparing Results 

The final step is comparing the outputs of the model versus the system. If you gave a record with a city of St. Louis, and a ZIP code of 00000 (which is invalid), an accurate model would expect an error. If the actual system doesn’t, then you may have a bug.

It’s possible to also create a kind of heatmap to see which parts were different when they were compared. Then, as dozens or hundreds of comparisons start showing a common theme of the city and ZIP code being involved, the more apparent it’ll become that something weird is happening in that logic.

 Advantages and Tips

The biggest advantage to this approach is that the more complex the system is, the better. The chances of both the model and the system being wrong in the exact same way are near impossible, so you have a much greater assurance that a zero-difference comparison means that that part of the system is behaving correctly.

Plus, because your model and generator can each live in a giant program, you can use something like Cucumber to generate code for you. This makes understanding the code—along with the maintenance—much easier, since you can read exactly what the code will be doing.