Mutation testing with Pitest

Mutation testing promises to help ensure quality tests. It does this by making changes to a code base and running all tests. If all is well, some changes in code should result in failing tests. So making a bunch of changes like inverting the condition in an if-statement, should cause the tests to fail. If not, the test isn't good enough.

The Wikipedia page above shows a list of frameworks that gave me pause; most of these frameworks are badly outdated. The most active projects I found are:

• Stryker which works for Javascript, C# and Scala, but not for Java
• Jumble which was last updated in 2015
• and [Pitest]

Pitest should work fine for Java, so I decided to try it out.

The code in this blog lives lives here on GitHub. The version for this blog is tagged with [part1].

The class under test is `MyClass`. It defines a single method `myTestMethod`. This method makes a decision based on an input parameter.

public boolean myTestMethod(int i) {
    if (i > 10) return true;
    return false;
}

The test code in MyControllerTest shows what looks like a perfectly fine test:

@Test
public void testMyTestMethod() {
    MyClass myClass = new MyClass();
    assertTrue(myClass.myTestMethod(100));
    assertFalse(myClass.myTestMethod(0));
}

Both branches of the if in `myTestMethod` are tested, but what happens if we would change the condition of the if?

The test cases are not sufficient because we can change `if (i > 10)` to `if (i > 11)` and all would still be green.

Pitest catches this problem in our test case:

1. changed conditional boundary → SURVIVED
2. negated conditional → KILLED
3. replaced return of integer sized value with (x == 0 ? 1 : 0) → KILLED
1. replaced return of integer sized value with (x == 0 ? 1 : 0) → KILLED

as shown by the line

1. changed conditional boundary → SURVIVED

The boundary for the if statement can be changed but the test still passes.

Given this test code it wasn't possible to negate the condition though, because in that case the test fails:

2. negated conditional → KILLED

OK, so this is where my wife, working as a tester, was rather unimpressed with my results and said

duh, your test is broken because it didn't check the boundary condition.

Fair enough, I guess... Still, these things happen, right?

To add Pitest to your project you need to add the following in your pom.xml file:

<build>
    <plugins>
        <plugin>
            <groupId>org.pitest</groupId>
            <artifactId>pitest-maven</artifactId>
            <version>1.4.3</version>
            <dependencies>
                <dependency>
                    <groupId>org.pitest</groupId>
                    <artifactId>pitest-junit5-plugin</artifactId>
                    <version>0.10</version>
                </dependency>
            </dependencies>
        </plugin>
...

(or remove the <dependency> if you're using Junit 4.2) and run the tests:

mvn clean test org.pitest:pitest-maven:mutationCoverage

The results are in `target/pit-reports`.