I spent a lot of time scratching my head at work this week as I looked at
a java project’s maven build output:
Tests run: 1457, Failures: 0, Errors: 1, Skipped: 0
[INFO] ----------------------------------------------------------------
[ERROR] BUILD FAILURE
[INFO] ----------------------------------------------------------------
[INFO] There are test failures.
[INFO] ----------------------------------------------------------------
[INFO] For more information, run Maven with the -e switch
[INFO] ----------------------------------------------------------------
[INFO] Total time: 51 seconds
When I first encountered this, I suspected that I had broken a test with some
new code I had been working on. I opened up the test that was being flagged with an error
and, to my surprise, it ran fine when I ran it individually. I ran it from the command
line with maven and also from within my IDE. By itself, it ran without error.
Each time I ran the full unit test suite, it failed with the same error:
java.lang.IllegalStateException: 0 matchers expected, 2 recorded.
at org.easymock...
This further confused me because the line it was failing on was the equivalent of this:
expect(mymock.getValue()).andReturn(anotherMock).anyTimes();
I wasn’t using any matchers on this line. Since the test worked by itself, I started to
suspect that one of the other tests was somehow polluting the test environment.
This project uses the “fork once” option to run the test suite. So, all of the tests
are executing in the same JVM. As a quick sanity check, I changed the pom.xml file
to “fork always” (new JVM for each test class). When I ran with this configuration,
all of the tests passed (it just took a LOT longer):
[INFO] ----------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ----------------------------------------------------------------
[INFO] Total time: 20 minutes 49 seconds
So, now to track down the test or tests that are causing the unit test suite to fail.
(While the “fork always” worked, I didn’t want to leave it at that and wait 21 minutes
for each round of unit tests!)
Unfortunately, we don’t have the tests running as part of our continuous integration
builds (yet). So, I just started rocking svn log with a date range to look for recent
changes that touched tests executing prior to the test in error.
Finally, I ran across a class that had something like this in it:
expect(obj.execute(isA(Action.class))).andAnswer(new IAnswer<Object>(){
public Object answer() throws Throwable {
.
.
.
.
return EasyMock.anyObject();
}
}).anyTimes();
At first glance, I didn’t see anything wrong. But then I started looking more closely
at EasyMock.anyObject() after googling around for “N matchers expected, M recorded.”
There were several sections like this in one of the test classes. All of them had a
return EasyMock.anyObject();
line. Since anyObject() is an argument matcher, I didn’t think it needed to be
returned by our mocked object. In this case, I was able to simply replace
return EasyMock.anyObject();
with
return null;
After doing this, the test suite started running successfully.
Rocking the EasyMock.anyObject() outside of the normal argument matcher scenario
really hijacked the test suite.
Closing Thoughts
Obviously, we need to get this project’s tests running as part of the continuous integration
process. Not doing so costs us time, broken tests, and a lot of head scratching.
I believe mock objects can be a very valuable component of every project’s test suite.
If you’re not using them in your project, please don’t let this post stop you from doing so.
For every mock slip up we’ve had, we’ve had a ton of bugs caught by unit tests using mock objects.
Just be careful if you start throwing around anyObject()!
Rock the Mock