Parametrized tests – solutions overview (TestNG, JUnit4, JUnit5)

Parametrized tests are a lot easier to maintain. Usually when single requirement changes it’s enough to change one variable in the code. You don’t have to read the whole implementation of the test (If you know what your variable means).

Why do we parametrize tests? 

Effort spent on unit tests maintenance is one of major argument of unit tests sceptics. Of course, always when behavior of code covered by test changes, unit tests also have to be changed. This is not a reason for not testing your code, but it shows that maintenance cost might be an issue and we have to keep it in mind.

Second thing that many people don’t like is… that we are obliged to write unit tests. It’s obvious that it takes time. Often we face problems e.g. when we work with classes which are difficult to test. It would be perfect to cover all cases just be adding few lines of code. And this is exactly the purpose of parametrized tests.

Parametrized tests are a lot easier to maintain. Usually when single requirement changes it’s enough to change one variable in the code. You don’t have to read the whole implementation of the test (If you know what your variable means).

An example…

Let’s consider a password validator. It will have two similar methods. First will return boolean. Second will throw an exception if password is incorrect.

public interface Validator {
    public boolean isValid(String password);
    public void validate(String password) throws ValidationException;
}

Requirements:

• password length between 7 and 21 characters
• includes lower and capital letter
• must not include white spaces
• null causes throwing NullPointerException

I have chosen methods and requirements to present interesting cases.

In the beginning we will create two simple tests and then we will parametrize them using TestNG, JUnit4 and JUnit5 (in that order).

public class ValidatorTest {
    private Validator validator = new ValidatorImpl();

    @Test
    public void validatorShouldReturnFalseForInvalidPassword(){
        String password = "#!$";
        boolean result = validator.isValid(password);
        assertFalse(result);
    }

    @Test
    public void validatorShouldReturnTrueForValidPassword() {
        String password = "testPassword!";
        boolean result = validator.isValid(password);
        assertTrue(result);
    }
}

TestNG, DataProviders and so on…

“Recipe” for parameterization in TestNG:

1. Take method which you want to test.
2. Chose data and make them method arguments.
3. Prepare separated method which will return two-dimensional array of Objects. Single array will be used as arguments list for test method.
4. Add annotations to link data provider and test method.

For correct passwords…

@DataProvider(name = "correct passwords")
public Object[][] getValidPasswords() {
    return new Object[][]{
        {"testPassword!"},
        {"12345sS!"},
        {"123456789012345678Aa"},
        {"aaAAbbBB!"}
    };
}

@Test(dataProvider = "correct passwords")
public void validatorShouldReturnTrueForValidPassword(String validPassword) {
    boolean result = validator.isValid(validPassword);
    assertTrue(result);
}

…and incorrect.

@DataProvider(name = "wrong passwords")
public Object[][] getInvalidPasswords() {
    return new Object[][]{
            {"Aa3456789012345678901"},
            {"Aa34567"},
            {"aa345678"},
            {"AA345678"},
            {"Aa345678\n"},
            {"Aa34 5678"},
    };
}

@Test(dataProvider = "wrong passwords")
public void validatorShouldReturnFalseForInvalidPassword(String invalidPassword) {
    boolean result = validator.isValid(invalidPassword);
    assertFalse(result);
}

Fortunately implementation of data providers is reusable so we can add two parametrized tests for validate method.

@Test(dataProvider = "correct passwords")
public void validatorShouldNotThrowExceptionForValidPassword(String validPassword) throws ValidationException {
    validator.validate(validPassword);
}

@Test(dataProvider = "wrong passwords", expectedExceptions = ValidationException.class)
public void validatorShouldThrowExceptionForInvalidPassword(String invalidPassword) throws ValidationException {
    validator.validate(invalidPassword);
}

And last two tests for handling null.

@Test(expectedExceptions = NullPointerException.class)
public void isValidMethodShouldThrowNullPointerExceptionForNull() {
    validator.isValid(null);
}

@Test(expectedExceptions = NullPointerException.class)
public void validateMethodShouldThrowNullPointerExceptionForNull() throws ValidationException {
    validator.validate(null);
}

JUnit4, Runner Parametrized etc.

JUnit4 provides completely different solution. It parametrizes whole test class, not a single method like in TestNG. We need to turn on Parameterized runner using annotation @RunWith on class and static method annotated with @Parameters that returns data. The data is represented by collection of objects array. What happen with these objects and how to use them in tests? Each array goes to constructor as arguments list and these objects will be set on fields.

Earlier we used two separated data sources: correct and incorrect data. Now we’re going to combine and distinguish them to adjust to given frames.

@RunWith(Parameterized.class)
public class ValidatorTest {
    private Validator validator = new ValidatorImpl();

    @Parameterized.Parameters(name = "password: {0}, valid: {1}")
    public static Collection<Object[]> getPasswords() {
        return Arrays.asList(new Object[][]{
                {"Aa345678901234567890", true},
                {"Aa345678", true},
                {"AbCdEfgH", true},
                {"AbCdEfgH90123", true},
                {"Aa3456789012345678901", false},
                {"Aa34567", false},
                {"aa345678", false},
                {"AA345678", false},
                {"Aa345678\n", false},
                {"Aa34 5678", false},
                {"Aa34\t5678", false},
                {"aB", false}
        });
    }

    private String password;
    private boolean expectedResult;

    public ValidatorTest(String password, boolean expectedResult) {
        this.password = password;
        this.expectedResult = expectedResult;
    }

    @Test
    public void validatorShouldReturnCorrectResult() {
        boolean result = validator.isValid(password);
        assertEquals(expectedResult, result);
    }

    // more tests here
}

First test looks really good, maybe even better than tests from TestNG example. We have one data source. One input and one expected result, but next test case doesn’t look pretty…

@Test
public void validationShouldPassOrThrowException() throws ValidationException {
    boolean exceptionShouldBeThrown = !expectedResult;
    if (exceptionShouldBeThrown) {
        try {
            validator.validate(password);
            fail("Password: " + password + " is incorrect. ValidationException should be thrown");
        } catch (ValidationException e) {
            //success, validation exception thrown
        }
    } else {
        validator.validate(password);
    }
}

Behavior of validator depends on variable called expectedResult. Tests for positive and negative cases would look completely different. To be honest I don’t like this test.

Exclusion of non-parametrized tests

In the end we would like to add tests for null. And there is another issue. Tests will be repeated unnecessarily many times.

@Test(expectedExceptions = NullPointerException.class)
public void isValidMethodShouldThrowNullPointerExceptionForNull(){
    validator.isValid(null);
}

@Test(expectedExceptions = NullPointerException.class)
public void validateMethodShouldThrowNullPointerExceptionForNull() throws ValidationException{
    validator.validate(null);
}

We can create separate class for these tests. Second option is to use runner Enclosed and nested class. Code would look like that:

@RunWith(Enclosed.class)
public class ValidatorTest {

    @RunWith(Parameterized.class)
    public static class ParametrizedTests {

        private Validator validator = new ValidatorImpl();

        @Parameterized.Parameters(name = "password: {0}, valid: {1}")
        public static Collection<Object[]> getPasswords() {
            return Arrays.asList(new Object[][]{
                    {"Aa345678901234567890", true},
                    {"Aa345678", true},
                    {"AbCdEfgH", true},
                    {"AbCdEfgH90123", true},
                    {"Aa3456789012345678901", false},
                    {"Aa34567", false},
                    {"aa345678", false},
                    {"AA345678", false},
                    {"Aa345678\n", false},
                    {"Aa34 5678", false},
                    {"Aa34\t5678", false},
                    {"aB", false}
            });
        }

        private String password;
        private boolean expectedResult;

        public ParametrizedTests(String password, boolean expectedResult) {
            this.password = password;
            this.expectedResult = expectedResult;
        }

        @Test
        public void validatorShouldReturnCorrectResult() {
            boolean result = validator.isValid(password);
            assertEquals(expectedResult, result);
        }

        @Test
        public void validationShouldPassOrThrowException() throws ValidationException {
            boolean exceptionShouldBeThrown = !expectedResult;
            if (exceptionShouldBeThrown) {
                try {
                    validator.validate(password);
                    fail("Password: " + password + " is incorrect. ValidationException should be thrown");
                } catch (ValidationException e) {
                    //success, validation exception thrown
                }
            } else {
                validator.validate(password);
            }
        }
    }

    public static class NotParametrizedTests {
        private Validator validator = new ValidatorImpl();

        @Test(expected = NullPointerException.class)
        public void isValidMethodShouldThrowNullPointerExceptionForNull() {
            validator.isValid(null);
        }

        @Test(expected = NullPointerException.class)
        public void validateMethodShouldThrowNullPointerExceptionForNull() throws ValidationException {
            validator.validate(null);
        }
    }
}

TestNG vs JUnit4

To summarize.

1. Tools for parameterizing in JUnit4 are not elastic enough. Only one set of data per one class.
2. Parameters names are written in constructor, it made code a little bit less readable.
3. More specific tests, which don’t need parametrization, are run many times. To avoid it we need inelegant workarounds.

Junit 5

New is always better.”

Barney Stinson, “How I met your Mother”

Much better support for parametrization provides JUnit 5. We can choose from variety of options. Data source can be a stream, array, method, enum… (more information can be found in the user guide http://junit.org/junit5/docs/current/user-guide/#writing-tests-parameterized-tests).

Maybe I am prejudiced in favour of DataProviders from TestNG and that why I choose providing parameters from methods.

class ValidatorTest {

    Validator validator = new Validator();

    static Stream<String> getValidPasswords() {
        return Stream.of("testPassword!",
                "12345sS!",
                "123456789012345678aA",
                "aaAAbbBB!"
        );
    }

    @ParameterizedTest
    @MethodSource("getValidPasswords")
    void validatorShouldReturnTrueForValidPassword(String validPassword) {
        boolean result = validator.isValid(validPassword);
        assertTrue(result);
    }

    @ParameterizedTest
    @MethodSource("getValidPasswords")
    void validatorShouldNotThrowExceptionForValidPassword(String validPassword) throws ValidationException {
        validator.validate(validPassword);
    }

    static Stream<String> getInvalidPasswords() {
        return Stream.of(
                "Aa3456789012345678901",
                "Aa34567",
                "aa345678",
                "AA345678",
                "Aa345678\n",
                "Aa34 5678"
        );
    }

    @ParameterizedTest
    @MethodSource("getInvalidPasswords")
    void validatorShouldReturnFalseForInvalidPassword(String invalidPassword) {
        boolean result = validator.isValid(invalidPassword);
        assertFalse(result);
    }

    @ParameterizedTest
    @MethodSource("getInvalidPasswords")
    void validatorShouldThrowExceptionForInvalidPassword(String invalidPassword) {
        assertThrows(ValidationException.class,
                () -> validator.validate(invalidPassword));
    }

    @Test
    void isValidMethodShouldThrowNullPointerExceptionForNull() {
        assertThrows(NullPointerException.class,
                () -> validator.isValid(null));
    }

    @Test
    void validateMethodShouldThrowNullPointerExceptionForNull(){
        assertThrows(NullPointerException.class,
                () -> validator.validate(null));
    }
}

Summary

1. In my opinion TestNG is better than Junit4 but JUnit5 is better than TestNG.
2. While choosing test framework it is really worth to consider provided support for tests parametrization.

Happy coding everyone 🙂