Custom formatters in Cucumber

I’ve been working with cucumber-java test framework for some time now and I must admit, I really like what is possible to achieve by using it. Of course, it is quite common in test automation, that more often it is required to do something that is nearly impossible than otherwise.

One of such cases happened to me lately, or at least, I thought it will be hard to overcome. Finally, I discovered one of cucumber features I was not aware of – formatters, which helped me a lot. Formatter (or EventListener) can be used to capture specific events sent by cucumber. Honestly, I had serious problems finding any documentation related to that, so I decided to write this short article to help others understand how formatters work.

Use case

My goal was quite straightforward – I wanted to scan all my feature files to retrieve unique set of tags used in my tests and store it in a file. Initially I thought I will have to write some code to open all feature files one by one and do some content parsing – I was pretty sure there is nothing like annotation processors similar to what we already have in java. Then, I realized that I already use plugins when I run my tests, which produces some output report in form of JSON file. It is configured in my CucumberOptions:

@RunWith(Cucumber.class)
@CucumberOptions(
    plugin = {"pretty",
        "json:target/report.json"},
    features = "src/test/resources/feature")
public class TestRunner {}

I did some reverse engineering and I found classes responsible for those plugins – cucumber.runtime.formatter.JSONFormatter and cucumber.runtime.formatter.PrettyFormatter, stored in cucumber-core module. Obviously, there is no single line of Javadoc in their source code and to be honest, logic placed there wasn’t super easy to understand. Digging up to the Plugin interface payed off – this interface has some Javadoc describing how implementing classes should look like.

Implementation

Since Cucumber 4.0, Formatter interface is deprecated – to create own formatter class you must implement EventListener, which brings single method – setEventPublisher.

public class CustomTagsFormatter implements EventListener {

    private final NiceAppendable out;
    private Set<String> tags = new TreeSet<>();

    public CustomTagsFormatter(Appendable out) {
        this.out = new NiceAppendable(out);
    }

    @Override
    public void setEventPublisher(EventPublisher publisher) {
        publisher.registerHandlerFor(TestCaseStarted.class, this::collectTags);
        publisher.registerHandlerFor(TestRunFinished.class, this::generateJson);
    }

There is also a possibility to use ConcurrentEventListener, which supports parallel test execution.

The method parameter, EventPublisher, allows registering handlers for specific Events, including the ones useful for me:

• cucumber.api.event.TestCaseStarted – triggered before each test scenario
• cucumber.api.event.TestRunFinished – triggered after all feature files are finished

I also added class constructor with Appendable parameter – thanks to that I’m able to set output file from CucumberOptions, just the same as it is done in JSONFormatter. Constructor argument is optional and can be chosen from java.lang.Appendable, java.net.URI, java.net.URL or java.io.File, depending on formatter needs. Only one argument is accepted.

The only thing left is to implement handler methods. For each test scenario started I want to read assigned tags, which is done like this:

private void collectTags(TestCaseStarted event) {
    tags.addAll(event.testCase.getTags().stream()
        .map(PickleTag::getName)
        .collect(Collectors.toSet()));
}

All tags are taken into account – those assigned to feature files as well as those assigned to specific scenarios. At the very end of test run I write everything collected to appendable passed from constructor:

private void generateJson(TestRunFinished event) {
    out.println(String.join(",", tags));
}

Finally, calling my new formatter from CucumberOptions, with output file location as a parameter:

@RunWith(Cucumber.class)
@CucumberOptions(
    plugin = {"pretty",
        "json:target/report.json",
        "html:target/html",
        "config.CustomTagsFormatter:target/tags.txt"},
    features = "src/test/resources/feature")
public class TestRunner {}

The ‘:’ delimiter is used to separate the formatter class and an optional argument.

After successful (or not) run of my test suite I receive nice tags.txt file with all scenario tags placed in the target of the project. Awesome!

Parsing feature files can be also performed ‘offline’ – without running real tests. There is nice option called dryRun to do that. When added to CucumberOptions it triggers all cucumber related configurations but does not execute real tests:

@RunWith(Cucumber.class)
@CucumberOptions(
    plugin = {"config.CustomTagsFormatter:target/tags.txt"},
    features = "src/test/resources/feature",
    dryRun = true)
public class TestRunner {}

Possibilities

My use case was pretty trivial, I must say, but formatter saved me a lot of bizarre coding. Potential possibilities, however, are endless. From writing custom reports, to trigger some special actions after each scenario step – this is probably the most powerful mechanism in cucumber exposed to the user and, sadly, it is not even documented properly. I remember all this time I tried to mix Junit specific annotations and cucumber Before/After hooks to achieve some specific setup in my framework configuration. Now I know how to do it smarter. For your reference, possible events handled by EventPublisher are (javadoc):

• Event – all events.
• TestRunStarted – the first event sent.
• TestSourceRead – sent for each feature file read, contains the feature file source.
• SnippetsSuggestedEvent – sent for each step that could not be matched to a step definition, contains the raw snippets for the step.
• TestCaseStarted – sent before starting the execution of a Test Case(/Pickle/Scenario), contains the Test Case
• TestStepStarted – sent before starting the execution of a Test Step, contains the Test Step
• EmbedEvent – calling scenario.embed in a hook triggers this event.
• WriteEvent – calling scenario.write in a hook triggers this event.
• TestStepFinished – sent after the execution of a Test Step, contains the Test Step and its Result.
• TestCaseFinished – sent after the execution of a Test Case(/Pickle/Scenario), contains the Test Case and its Result.
• TestRunFinished – the last event sent

Each of those events contains some specific data related to what causes it, like source, test case, test results, etc.

Happy testing!