Home Blog technologiczny Improve your tests and codebase by „Mutation Testing” – part 2

Improve your tests and codebase by „Mutation Testing” – part 2

Kamil Orłowski

29.01.2020

In the previous article I presented the main idea of Mutation Testing. In order to better understand this concept, so far, my reflections covered only an example of a simple code where Traditional Mutation Operators have been used. In this part, you will get introduced to the concept of subsequent groups of Mutation Operators. I will show you also how to configure PIT with the Intellij IDEA.

Mutators

Literature defines mutators for general use and specific, designed for certain programing languages. For example, 77 Mutation Operators have been defined for C, Fortran has 22 operators and Java has 29.
Below you can find a set of operators, dedicated for Java language.

Mutation operators for Java are divided into two groups. First group is found in the procedural programing languages. Second group consists of operators specific for the object-orientation programing. The following mutation operators were proposed by Yu-Seung Ma, Yong-rae Kwon, and Jeff Offutt – this is important, because you can find more papers, which describe this topic. Furthermore, each Mutation Testing framework implements a different subset of the operators. The following information should illustrate how immense this topic is.

1. Traditional mutation operators – based on faults, due to common programming mistakes made in a procedural programing.

2. Class related mutation operators – based on faults due to redeclaration of class variables and a method in corresponding subclass. Because of the abundance of code to present all possible mutants, below are extensive examples only for Java specific operators. In other cases, I have limited myself to a brief description. If you are interested in the details of this topic, please refer to the references at the end.

2.1. Java specific operators

2.2. Inheritance operators

2.3. Polymorphism and dynamic binding operators – based on faults due to Java polymorphism features. Operators check bindings used in a codebase and a possibility to change these bindings with another incorrect one.

2.4. Method overloading operators – based on faults due to method overloading.

Please keep in mind that two similar operators, when applied to a program written in different languages, will most likely produce a different number of mutators.

PiTest’s mutators

In this section I will present groups of mutators implemented by PiTest (PIT). The framework is still under the development, and subsequent mutators are in the experimental state. To be up to date, please check the main site of the project. I also encourage you to analyze the implementation of mutators, available here.

Below you can find PIT’s Mutation Operators groups. You can use names from the left column to configure PIT in your project – we will come back to this later.

PiTest configuration

In this section, I will show you how you can use PIT together with Intellij, and how to configure PIT’s runner for a better performance. All necessary configuration details are depicted here.

1. How to use PIT
There are a few options – personally I think when you use PIT during the daily development routine, you can minimize costs of Mutation Testing. Because of that I encourage you to utilize PIT plugin dedicated for Intellij.

1.1. Plugin installation
You can download the plugin by yourself or directly from Intellij. Here you can find the PIT’s official site plugin.

1.2. How to use the PIT plugin
First, you should setup a default configuration for all PIT’s runners.

Now, you can create the configuration runner for your feature. Target classes field points out to an exact package. All tests nested inside a package given by you, will be executed by PIT runner. In case you have declared already other runners, the following shortcut may be helpful: Alt + Shift + F10

1.3. How to configure Mutation Operators

By default, PIT uses mutators from DEFAULTS group. In order to use another set of mutators, you should add extra configuration like below. All available names of groups were listed and described at the beginning of the section ‘PiTest’s mutators’.

1.4. Other configuration

You can tune the generation of reports or establish how many threads should be involved into the performing of Mutation tests. All the configuration parameters must be added into Other params field – as in the previous point. Here you can find more available PIT’s configuration parameters.

Mutation testing with other tools

PIT works with the well-known tools like: Maven, Gradle, Eclipse, Junit 5, Spock. At the time when you are reading this article I encourage you to check if PIT supports your favorite tools or programing language – when I started a journey with PIT, I could not find any resources about running PIT with Spock, but when I tried to use it, it turns out that it is possible.

Be aware!

Mutation testing is time-consuming, because every single mutation requires new build which contains an exact defect. Unfortunately, the number of mutations can easily go into the thousands but luckily, PIT allows us to configure different types of mutators, so we can tune it as needed. Moreover, PIT will try to optimize the tests execution by manipulating a binary code directly or ignore nonconverted code.

Conclusions

Let me summarize what we have learned. First part of the article demonstrates the real-world example of how Mutation Testing can help us. Second part shows what exactly happens under the hood. At the end I demonstrated how to start using PIT plugin. This should be enough to implement Mutation Testing in your projects.

Resources to learn more

Books & articles

Aditya P. Mathur – Foundations of Software Testing, 2nd Edition
Filip van Laenen – Mutation testing. Better code by making bugs
Adam Goucher, Tim Riley – Beautiful Testing
Talk in Oxford – Chris Rimmer – What mutation testing is
Article in the ACCU magazine Overload – Filip van Laenen’s – A look at Mutation testing

Video