Java – Getting cucumber results into Sonar

Sonar doesn’t seem to like second helpings of cucumber-java test results. Here’s a simple workaround to refresh its appetite.

Cucumber is a great testing tool for Behaviour Driven Development and with the release of cucumber-java it’s pretty easy to start testing Java code against natural language feature specifications.

Chances are though, if Sonar is anywhere in your build chain, the euphoria of getting your first Gherkin feature to run will quickly dissipate when you try a second.

First time it’s easy

Let’s kick off with the Bartender example we used here and write a Cucumber feature to describe the scenario of ordering a Long Island Iced Tea:-

Feature: Good Bartender

Scenario: Ordering a Long Island Iced Tea
  Given I'm thirsty
    And I'm sober
   When I order a Long Island Iced Tea
   Then my glass contains the following
		| Ingredient | Quantity |
		| Gin        | 50ml     |
		| Rum        | 50ml     |
		| Vodka      | 50ml     |
		| Tequila    | 50ml     |
		| Triple Sec | 50ml     |
		| Sour Mix   | 75ml     |
		| Coke       | 75ml     |   		
		| Ice        | 4 cubes  |

We can get this running against our Java classes using cucumber-java and cucumber-junit with a few lines of boilerplate code and three step definitions. It’ll compile, it’ll run and the results will import into Sonar.

But here’s the problem

Unfortunately the Surefire reports generated by the current cucumber-junit (1.1.1 at the time of writing) aren’t that great:-

<testcase time="0" 
            classname="Given I\&apos;m thirsty   " 
            name="Given I\&apos;m thirsty   "/>
<testcase time="0" 
            classname="And I\&apos;m sober    " 
            name="And I\&apos;m sober    "/>
<testcase time="0" 
            classname="When I order a Long Island Iced Tea     " 
            name="When I order a Long Island Iced Tea     "/>
<testcase time="0" 
            classname="Then my glass contains the following      " 
            name="Then my glass contains the following      "/>
<testcase time="0.031" 
            classname="Scenario: Ordering a Long Island Iced Tea  " 
            name="Scenario: Ordering a Long Island Iced Tea  "/>

One minor annoyance of this is that each step in a feature adds to the test count reported by the Maven test runner and by Sonar, resulting in a somewhat unrepresentative test count.

The other problem is that the classname and the name of each testcase match the name of the step. They aren’t qualified by the scenario description.

Second time’s not so sweet

Flushed with success we decide to expand the testing of our bartender’s repertoire:-

Feature: Good Bartender

Scenario: Ordering a Long Island Iced Tea
  Given I'm thirsty
    And I'm sober
   When I order a Long Island Iced Tea
   Then my glass contains the following
		| Ingredient | Quantity |
		| Gin        | 50ml     |
		| Rum        | 50ml     |
		| Vodka      | 50ml     |
		| Tequila    | 50ml     |
		| Triple Sec | 50ml     |
		| Sour Mix   | 75ml     |
		| Coke       | 75ml     |   		
		| Ice        | 4 cubes  |

Scenario: Ordering a Bloody Mary
  Given I'm hung over
    And I've just woken up
   When I order a Bloody Mary
   Then my glass contains the following
		| Ingredient      | Quantity |
		| Tomato Juice    | 150ml    |
		| Vodka           | 50ml     |
		| Tabasco         | 3 dashes |
		| Worcester Sauce | 50ml     |
		| Ice             | 4 cubes  |

Our Maven build runs fine, but this time Sonar doesn’t like the results:

[ERROR] Failed to execute goal org.codehaus.mojo:sonar-maven-plugin:2.0:
sonar (default-cli) on project MyProject: 
Can not execute Sonar: 
Can not add twice the same measure on [default] … -> [Help 1]

The key to the problem here is can not add twice the same measure, the reason being our second scenario is re-using steps from the first. Since the test cases are only identified by the name of the step we’re now getting duplicates.

Getting Sonar to eat its second helpings

One workaround here would be to have a separate JUnit runner for each scenario, though that duplicates a lot of code and contaminates our feature structure. Another would be to add some unique reference to each step but that contaminates all our features and glue code.

We could, of course, fix the problem properly – it is open source after all. The underlying problem is buried quite deep in the interaction between cucumber-junit and JUnit though – it might take a bit more time than you have to hand. If you need a quicker solution, read on.

One thing you might notice about the <testcase/> entries generated by cucumber-junit is that each test name has an ever-lengthening string of spaces after it. These are introduced by the class cucumber.runtime.junit.DescriptionFactory via an interesting field called UNIQUE_HACK. It looks like this was introduced to try and make test case names unique. Unfortunately, at least with Sonar 3.2 with both MySQL and Oracle behind it, the whitespaces added don’t work.

UNIQUE_HACK shouldn’t be necessary with JUnit 4.11 but unfortunately I’m still seeing the same problem. The quick solution is to include the DescriptionFactory class in your project and change UNIQUE_HACK to be an incrementing number. Change UNIQUE_HACK to a long and rework createDescription to do the following:-

public static Description createDescription(String name, Object uniqueId) {
    if (USE_UNIQUE_ID) {
        try {
            return (Description) Utils.invoke(null, CREATE_SUITE_DESCRIPTION, 
                  0, name, uniqueId, Array.newInstance(Annotation.class, 0));
        } catch (Throwable t) {
            throw new CucumberException(t);
        }
    } else {
        UNIQUE_HACK++;
        try {
            return (Description) Utils.invoke(null, CREATE_SUITE_DESCRIPTION, 
                  0, name + " # " + UNIQUE_HACK, 
                  Array.newInstance(Annotation.class, 0));
        } catch (Throwable t) {
            throw new CucumberException(t);
        }
    }
}

Our step names now present with a sequential number, which is a minor annoyance, but at least they’re unique and we can continue to develop and use clean Cucumber tests within our project, and run them through a build chain which includes Sonar, with just that hacked class to remove when we have a better fix.