9.13.2009

Running tests from a Maven test-jar in your build

So one of the less frequently used features of Maven is that it allows you to package and deploy the test classes and resources to the repository. While the documentation suggests it's to reuse the tests, you can only use these as a standard dependency, not to re-execute them.

Well I was recently challenged with the task of re-executing the tests. I won't discuss the motivation behind the request as I'm not certain it's worthwhile. For now let's just assume you've been asked to do this too. I'll save you some time and share some of the things I discovered when trying to accomplish this.

So obviously we'll need to have the test-jar in the repository (see link above). We'll then need to pull it down and have surefire to execute it. The problem is that surefire does not have a "test jarfile" goal. So we'll need to leverage the unpack goal of the dependency mojo. Here is the xml for that:

<build>
<plugins>
...
<plugin>
<groupid>org.apache.maven.plugins</groupid>
<artifactid>maven-dependency-plugin</artifactid>
<executions>
<execution>
<id>unpack</id>
<phase>process-test-classes</phase>
<goals>
<goal>unpack</goal>
</goals>
<configuration>
<artifactitems>
<artifactitem>
<groupid>com.hirn</groupid>
<artifactid>projecta</artifactid>
<version>1.0-SNAPSHOT</version>
<type>test-jar</type>
<outputdirectory>${project.build.directory}/additionaltests/projecta/</outputdirectory>
</artifactitem>
</artifactitems>
</configuration>
</execution>
</executions>
</plugin>
...



I'm assuming some basic Maven knowledge here but please comment if you have more questions about how this is configured. There are a couple personal selections worth discussing here.

The first is the outputDirectory within the artifactItems section. It may be tempting to unzip into src/test/java so that surefire will just execute the tests as a normal part of the cycle. This isn't good though because you could potentially clobber a test that already exists there. You can't create a separate directory within /src/test/java either because the package names wouldn't be correct anymore. I choose to use ${project.build.directory}/additionalTests/ (aka target) so that things get cleaned up naturally. It's also important to unzip each artifactItem into a separate folder to avoid possible namespace collisions.

The second is when to execute the goal. Process test resources seemed like a natural selection here, even though it's slightly against purist Maven lingo as the "resources" directory non class files. Another option could be to use the pre-integration-test. Just as long as it's before the phase we configure surefire actually execute the tests.

Since we extracted the executable .class files to a separate directory, we need to tell surefire when and what to execute. Here's one example:


<build>
<plugins>
...
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<executions>
<execution>
<id>project-a-tests</id>
<phase>test</phase>
<goals>
<goal>test</goal>
</goals>
<configuration>
<testClassesDirectory>${project.build.directory}/additionaltests/projecta</testClassesDirectory>
<reportsDirectory>${project.build.directory}/surefire-reports/projecta</reportsDirectory>
</configuration>
</execution>
</executions>
</plugin>



Pretty straightforward. You have to make sure the phase is declared after the phase where the dependency is unpacked and point it to corresponding directory. The other piece to consider is that you don't overwrite the surefire report produced from running the rest of the tests. I couldn't find a way to aggregate the the report but wither or not you want to is personal preference. Would you even want your CI or reporting tool to track the success of executing these tests? Probably not, but you certainly don't want to affect the output of the tests within the projects.

You can download an example of this to see it in action. Just `mvn install` projecta and then do the same for projectb. You'll see the test run from projecta all over again.

Why I had to do this seemed to be pretty special case. My situation was driven by a peculiar scenario. There was a project A with dependencies A->B->C where A has a defect due to code that lives in C. Rather than create a new B, we just want to touch C and modify A such that A->B,A-C'.

One way to look at it is that there's no code change in B so why build it? The alternative is to say why not? Not only is rebuilding cheap (or should be), version numbers are free and it communicates to other projects using B that they need to upgrade to B'.

So if you have a good reason for executing tests from a deployed test-jar, I'd like to hear it. The whole time I was doing this it felt unnatural. I've never had to do this in over two years of extensive Maven use. The thing I love about Maven is that if it doesn't work automatically, you probably shouldn't be doing it. If this were truly useful, there would be a surefire:execute-jar goal.

6 comments:

  1. To my mind, there is a pretty good reason to execute tests contained in a test jar. It seems natural to write unit tests corresponding to a pure interface api in the test dir of this api. API Implementors should then execute unit tests contained in the test-jar to ensure that the original contracts are fullfiled.

    ReplyDelete
  2. if you have generic integration tests that you want to be able to execute across many projects, then it makes sense to package them into a single jar, instead of maintaining those exact same tests in multiple locations (all of the projects' test directories)

    ReplyDelete
    Replies
    1. I think is spot on. I have one situation where the tests are configurable with XML, so I want the generic "skeleton" to be in one place, and then use it in other projects with a config file.

      Delete
  3. This solution will not work, if the test.jar is not installed. Try to run maven with goal "test" from the top project and you will see that the unpack plugin will fail. (Sorry, i can not speak english)

    ReplyDelete
  4. I was not able to get this to work. I got it to extract the tests to folder in target/additionaltests, but I was not able to get the tests to run. It seems like Maven can't find them even though I did add the folder with the .class files to the classpath. I'm not sure if I did something wrong, and the link to the example is broken. could you post an updated link?

    ReplyDelete
  5. Hi Mike.

    I'll see if I can dig up the code but it's doubtful since I've let this blog turn into a swampland. This worked with Maven 2.0.6 if I remember. Things may have changed.

    If test classes are in the folder you've extracted the test-jar into and you've told surefire to execute them it should be doing it. If you could post your error I'd take a peek at it.

    ReplyDelete