Running tests from a Maven test-jar in your build

So one of the less frequently used features of Maven is that it allows you to package and deploy the test classes and resources to the repository. While the documentation suggests it's to reuse the tests, you can only use these as a standard dependency, not to re-execute them.

Well I was recently challenged with the task of re-executing the tests. I won't discuss the motivation behind the request as I'm not certain it's worthwhile. For now let's just assume you've been asked to do this too. I'll save you some time and share some of the things I discovered when trying to accomplish this.

So obviously we'll need to have the test-jar in the repository (see link above). We'll then need to pull it down and have surefire to execute it. The problem is that surefire does not have a "test jarfile" goal. So we'll need to leverage the unpack goal of the dependency mojo. Here is the xml for that:

<build>
<plugins>
...
<plugin>
<groupid>org.apache.maven.plugins</groupid>
<artifactid>maven-dependency-plugin</artifactid>
<executions>
<execution>
<id>unpack</id>
<phase>process-test-classes</phase>
<goals>
<goal>unpack</goal>
</goals>
<configuration>
<artifactitems>
  <artifactitem>
    <groupid>com.hirn</groupid>
    <artifactid>projecta</artifactid>
    <version>1.0-SNAPSHOT</version>
    <type>test-jar</type>
    <outputdirectory>${project.build.directory}/additionaltests/projecta/</outputdirectory>
  </artifactitem>
</artifactitems>
</configuration>
</execution>
</executions>
</plugin>
...

I'm assuming some basic Maven knowledge here but please comment if you have more questions about how this is configured. There are a couple personal selections worth discussing here.

The first is the outputDirectory within the artifactItems section. It may be tempting to unzip into src/test/java so that surefire will just execute the tests as a normal part of the cycle. This isn't good though because you could potentially clobber a test that already exists there. You can't create a separate directory within /src/test/java either because the package names wouldn't be correct anymore. I choose to use ${project.build.directory}/additionalTests/ (aka target) so that things get cleaned up naturally. It's also important to unzip each artifactItem into a separate folder to avoid possible namespace collisions.

The second is when to execute the goal. Process test resources seemed like a natural selection here, even though it's slightly against purist Maven lingo as the "resources" directory non class files. Another option could be to use the pre-integration-test. Just as long as it's before the phase we configure surefire actually execute the tests.

Since we extracted the executable .class files to a separate directory, we need to tell surefire when and what to execute. Here's one example:

<build>
<plugins>
...
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<executions>
<execution>
<id>project-a-tests</id>
<phase>test</phase>
<goals>
  <goal>test</goal>
</goals>
<configuration>
<testClassesDirectory>${project.build.directory}/additionaltests/projecta</testClassesDirectory>
  <reportsDirectory>${project.build.directory}/surefire-reports/projecta</reportsDirectory>
</configuration>
</execution>
</executions>
</plugin>

Pretty straightforward. You have to make sure the phase is declared after the phase where the dependency is unpacked and point it to corresponding directory. The other piece to consider is that you don't overwrite the surefire report produced from running the rest of the tests. I couldn't find a way to aggregate the the report but wither or not you want to is personal preference. Would you even want your CI or reporting tool to track the success of executing these tests? Probably not, but you certainly don't want to affect the output of the tests within the projects.

You can download an example of this to see it in action. Just `mvn install` projecta and then do the same for projectb. You'll see the test run from projecta all over again.

Why I had to do this seemed to be pretty special case. My situation was driven by a peculiar scenario. There was a project A with dependencies A->B->C where A has a defect due to code that lives in C. Rather than create a new B, we just want to touch C and modify A such that A->B,A-C'.

One way to look at it is that there's no code change in B so why build it? The alternative is to say why not? Not only is rebuilding cheap (or should be), version numbers are free and it communicates to other projects using B that they need to upgrade to B'.

So if you have a good reason for executing tests from a deployed test-jar, I'd like to hear it. The whole time I was doing this it felt unnatural. I've never had to do this in over two years of extensive Maven use. The thing I love about Maven is that if it doesn't work automatically, you probably shouldn't be doing it. If this were truly useful, there would be a surefire:execute-jar goal.