If your unit tests depend on external resources they’re going to fail from time-to-time when those resources aren’t available. Here we look at a few options for preventing your build from breaking.
Ideally unit tests should be self-contained, stubbing or mocking any external resources they depend on and thus ensuring they are repeatable under any and all circumstances.
Of course, in the real world, our tests aren’t always so nicely packaged. On smaller projects, or smaller teams where test resources are scarce, our unit tests may also be taking on an integration-testing role, or we may simply not have the time to stub or mock all the external resources we use. And we might like a heads-up if something has been changed in that external resource which genuinely breaks our code.
So our real-life unit tests may well hit external resources and annoyingly our otherwise valid build may randomly break – perhaps because the external resource isn’t up-and-running or we’re working away from our normal network where they simply aren’t reachable.
Fortunately there are some simple ways we can ensure our builds still complete successfully in these predictable false-positive failure scenarios.
Ignoring failures in your Maven build
If you’re using Maven for your build there a few command-line tricks that can help.
By default, maven will complete testing the module where it encounters the first failure, and then immediately stop and fail the build. For multi-module projects there are a couple of command-line options that force maven to plough on, though still fail the overall build at the end.
- –fail-at-end (or -fae for short) will mark the module where it first encounters test failures as failed, but continue on to build any modules that do not depend on it, failing the overall build at the end. It completely skips any module that depends on the one with test failures, and thus will run none of their tests.
- –fail-never (or -fn for short) will mark the module where it first encountered test failures as failed, but then try to build all other modules whether they depend on it or not. This will normally execute as much of a multi-module build as possible.
Both of these options fail our overall build but allow us to test and complete more of the components. If you want your build to complete successfully, even if there are failed tests, you can add -Dmaven.test.failure.ignore=true to your Maven command line:
mvn clean install -Dmaven.test.failure.ignore=true
This will log the failures but complete the build successfully.
Tests run: 410, Failures: 1, Errors: 1, Skipped: 2 [ERROR] There are test failures. … [INFO] ------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------ [INFO] Total time: 3:22.947s [INFO] Finished at: Fri May 03 09:41:44 BST 2013 [INFO] Final Memory: 18M/149M [INFO] ------------------------------------------------------------
This is a handy one to know in those situations where perhaps you’re working away from your normal network temporarily, or there’s a short, known outage on a service. It doesn’t discriminate the nature of the failures in any way though and it’s certainly not something you would want anywhere near a continuous integration build.
For that, we need a more discriminating, selective solution
Automatically skipping specific tests with unavailable resources
What we need is a programmatic equivalent of JUnit’s @Ignore annotation and fortunately JUnit provides just that in the Assume class. Its methods are used in a similar manner to those in the Assert class, except that failure causes the test to be ignored rather than failed.
For example, let’s say we’re writing a unit test that hits a web-service and we’ve decided we’re going to use the live service for it. As a simple check we could use InetAddress.isReachable() with a reasonable timeout to verify connectivity:-
@Test
public void test() {
Assume.assumeTrue(InetAddress.
getByName("www.devsumo.com").isReachable(5000));
// Some logic that hits our web-service
}
This will work but has a few disadvantages. First of all it could throw an assortment of exceptions (mostly sub-classes of IOException so we’re going to have to enclose it in a try/catch block with a slightly ugly hack to treat the exceptions as failed assumptions:-
@Test
public void test() {
try {
Assume.assumeTrue(InetAddress.
getByName("www.devsumo.com").isReachable(5000));
} catch(Exception e) {
Assume.assumeTrue(false);
}
// Some logic that hits our web-service
}
Secondly isReachable() isn’t really checking that the service is there, merely that there’s low-level connectivity to the host. It won’t tell us if the web-server is down and it might fail behind some proxy servers or corporate firewalls even though the web-service is reachable.
Making better assumptions
We need to be careful about assuming too much here and impinging on what our test is responsible for. We could try to access the WSDL for the web-service for example, but if there’s a change to the web-service that genuinely stops the old URL from working, our test is going to repeatedly skip rather than failing as it needs to. We need to carefully balance the convenience of being able to temporarily skip tests with the risk of skipping situations that should genuinely be marked valid.
As a compromise, let’s try to open an HttpURLConnection to the web-server’s root page, and wrap our check in a re-usable function which will remove the boilerplate code and exception-handling hack.
private boolean checkResource(String url) {
HttpURLConnection urlConnection = null;
try {
urlConnection = (HttpURLConnection)
new URL(url).openConnection();
urlConnection.setRequestMethod("GET");
urlConnection.connect();
return true;
} catch (MalformedURLException e) {
return false;
} catch (IOException e) {
return false;
} finally {
if(urlConnection != null) {
urlConnection.disconnect();
}
}
}
@Test
public void test() {
Assume.assumeTrue(
checkResource("https://www.devsumo.com/"));
// Some logic that hits our web-service
}
This will cause our test to run normally when the required web-server is up and contactable, or be ignored otherwise, and it keeps the content of our test method as uncontaminated as possible.
Our solution is HTTP specific though. We can make it more re-usable by trying to open a Socket connection to a specified host and port instead. While we’re at it we’ll move our resource check to a separate helper class so we can use it in other tests:-
public class SocketResourceValidator {
private static final int CHECK_TIMEOUT = 5000;
public static boolean validate(String host, int port) {
Socket testSocket = new Socket();
try {
SocketAddress testSocketAddress =
new InetSocketAddress(host, port);
testSocket.connect(testSocketAddress, CHECK_TIMEOUT);
return true;
} catch(IOException e) {
return false;
} finally {
if(testSocket.isConnected()) {
try {
testSocket.close();
} catch(Exception e) {
// Log
}
}
}
}
}
Which leaves our test method looking like this with no extra functions in our test class:-
@Test
public void test() {
Assume.assumeTrue(SocketResourceValidator.
validate("www.devsumo.com", 80));
// Some logic that hits our web-service
}
