Improve your tests and codebase by „Mutation Testing” – part 1

All experienced developers have a good understanding of how significant for the quality of the product is a source code covered by high quality test suites. Depending on a company or even a team, we can come across many different test procedures and test frameworks. Regardless of that, at the end of the day, we are faced with a codebase covered by tests.

Overview

All experienced developers have a good understanding of how significant for the quality of the product is a source code covered by high quality test suites. Depending on a company or even a team, we can come across many different test procedures and test frameworks. Regardless of that, at the end of the day, we are faced with a codebase covered by tests. We are required to implement strict testing procedures in order to protect our code against bugs. At the same time, we assume that, if the test output is green, the code works as expected. But, how do we know for sure that a test we created is good enough? How can one be convinced that during implementation of new features the system still works according to the requirements? What if a new team member with less experience, introduces a change in the existing logic? How to prevent the situation, where we need to reassess a whole bunch of altered code over and over again?

Correlations between code and test

Let us take a look at a typical daily development process. When we start to think deeply on a test procedure, we can come up with the following thought process.

Our code should pass the tests, however it should NOT pass it for the wrong reasons. Mutation testing is a known solution to improve continuous testing of our code. It does not provide a magic cure for all testing issues, but certainly it increases the testing reliability and allows us to be more confident and comfortable with our code.

Quick theoretical introduction to Mutation Testing

This article does not attempt to rephrase the whole available literature. If you are looking for more details, please refer to the references provided at the end.

• How does the history of Mutation Testing look like for Java environment?

• What is a program mutation?
  It is a technique for assessing the adequacy of test sets.
• What is a mutation?
  It is the act of changing a program.
• What is a mutant?
  It is a small change which we are introducing into source code.
• What does “Mutant killed” mean?
  It means that your tests were able to catch the artificial change.
• What does “Mutant survived” mean?
  It means that your tests did not catch the artificial change.
• How does the above definitions work together?

• What does the idea behind Mutation testing stand by? Competent Programmer Hypothesis – programmers tend to make simple mistakes.Coupling effect – tests that detect simple, small errors can detect more complex ones, derived from the combination of other errors.
• What is PiTest?
  It is a Mutation testing framework.

Mutation Testing example using PiTest

In our example we use an application dedicated for airlines, which applies discounts on luggage. We start with simple code and then introduce some changes. To avoid unnecessary complexity, the structure of the code is simplified. Let’s consider the following code:

public class HandLuggageDiscountApplier {
    private static final double MAX_PERCENTAGE_DISCOUNT = 0.1;
    public double applyDiscount(double price, double discount){
        if (discount <= MAX_PERCENTAGE_DISCOUNT){
            return price - (price * discount);
        }

        return price;
    }
}

We are assuming that an airline provides basic price for hand luggage and a percentage discount, which we should apply under certain conditions. Discount is calculated in another application based on many requirements, but in the end, we want to be sure that it is not greater than some fixed value. Before going further, I will give you a moment – try figure out of how many test cases you need, in order to cover all variations.

As you can observe below all tests are green and the coverage is 100%.

Test cases

class HandLuggageDiscountApplierTest extends Specification {
    HandLuggageDiscountApplier discountApplier = new HandLuggageDiscountApplier()

    def "do not apply discount cause exceed max discount threshold"() {
        when:
        def priceWithDiscount = discountApplier.applyDiscount(100.0, 0.2)

        then:
        priceWithDiscount == 100
    }

    def "apply discount cause is less than max discount threshold"() {
        when:
        def priceWithDiscount = discountApplier.applyDiscount(100.0, 0.05)

        then:
        priceWithDiscount == 95
    }
}

Result of tests

Test coverage

Now, it is necessary to check what PiTest says about that.

Starting from the bottom, take a close look at what information the attached report presents.

• Report was generated by PiTest in version 1.2.4.
• Tests were executed in 361 ms.
• List of active mutators points out to the set of utilized mutators. This is the default set used by PiTest, when no extra configuration was provided.
• PiTest analyzed our code based on active mutators, executing 6 most suitable mutations.
• The part of code marked in red points out the location where mutant survived.

What exactly happened? During the analysis of source code, PiTest noted that the code contains a condition. Then, based on the available mutators PiTest picked mutator from CONDITIONALS_BOUNDARY set. This set replaces relational operator as presented in the table.

PiTest had introduced an artificial change into the codebase. Afterwards, it checked if the test suite was able to catch the mutant. We see in PiTest output, that tests do not check the case when discount is equal to the maximal discount threshold. We will fix that with a new test.

The new test case

def "apply discount cause is equal to max discount threshold"() {
    when:
    def priceWithDiscount = discountApplier.applyDiscount(100.0, 0.1)

    then:
    priceWithDiscount == 90
}

Result of tests

Let us again look on how the PiTest evaluates the test suite. To make it more interesting, I will show you how the console output looks like. As you can see, the PiTest generated 6 mutations, which were all killed. In this situation you can be sure that your tests will catch any change introduced into the source code.

General time consumption

Information about executed mutations and generated mutators

I am convinced that so far you are not impressed by Mutation testing, you can say “I write my code in a TDD style and all the above was obvious to me” – I agree with you, so let us make it a little bit more complex. Imagine that we received another requirements from our customer:

• As an airline, we would like to apply a discount on the check-in luggage
• Because the discount is calculated by different application, we need to introduce rules while applying this discount – max allowed discount is 20%
• When discount is greater or equal to the max allowed discount, then we expect that the max allowed discount would be applied. In another case we should apply discount which comes from the external system.

Below, you can find the implementation of our new requirements covered by the tests.

class CheckInBaggageDiscountApplier {
    private static final double MAX_PERCENTAGE_DISCOUNT = 0.2;

    public double applyDiscount(double price, double discount){
        double newDiscount = discount >= MAX_PERCENTAGE_DISCOUNT ? MAX_PERCENTAGE_DISCOUNT : discount;
        return price - (price * newDiscount);
    }
}

Test cases

class CheckInBaggageDiscountApplierTest extends Specifications {
    CheckInBaggageDiscountApplier discountApplier = new CheckInBaggageDiscountApplier()

    def "apply discount when is less than allowed maximum"() {
        when:
        def priceWithDiscount = discountApplier.applyDiscount(100.0, 0.15)

        then:
        priceWithDiscount == 85
    }

    def "do not apply discount when is equal to the allowed maximum, apply maximum"() {
        when:
        def priceWithDiscount = discountApplier.applyDiscount(100.0, 0.2)

        then:
        priceWithDiscount == 80
    }

    def "do not apply discount when is greather than allowed maximum, apply maximum"() {
        when:
        def priceWithDiscount = discountApplier.applyDiscount(100.0, 0.5)

        then:
        priceWithDiscount == 80
    }
}

Result of tests

Test coverage

This implementation is quite like the HandLuggageDiscountApplier. Test cases also look similar. I will ask you again the same question as before – please look at the code above and try to answer if we have covered all variations? Do you think that these test cases can protect the code against random changes? If you have your answer, check again the PiTest report.

PiTest report

It turns out that the PiTest was able to find a gap in the test suite. When you change condition from “discount >= MAXPERCENTAGEDISCOUNT” to the “discount > MAXPERCENTAGEDISCOUNT”, all tests are still green. You might be surprised, because there is a test which checks a scenario when the discount is equal to the MAXPERCENTAGEDISCOUNT. So how can we test it? – You did it already