Jake Herringbone

Making the case for unit tests to live in the code they test

This is perhaps the most controversial idea in Noop. I had an interesting discussion that included Cédric Beust, in which I defended this idea from a lot of doubts, mostly around why it would be needed, and the challenges with tools that expect a conventional code layout. So, I went to spec out exactly how it could work. And I’m happy to report that I think it totally works.

First, what’s wrong with writing unit tests the conventional way?

• Create a separate src-test or src/test source root, which is to contain just tests, so that we can compile the production code independently and enforce that prod code doesn’t have deps on tests.
• Create a new source file in that source root, in a package/namespace mirroring the location of the prod file you want to test.
• Name the new class with a convention like “*Test” so our team understands where to find the unit tests for a file.
• In the test fixture, create an instance of the class-under-test, supplying fakes or mocks for some of the constructor/setter dependencies as needed.
• Start calling methods in the class and sensing whether they did the right thing. Modify the code under test to make some fields or methods package-protected or friendly to the test class so it can white-box test where you need to.
• Optionally mark that change with a comment or annotation like @VisibleForTesting so your team can understand why this field or method isn’t private as they’d expect.
• Get the tests green, and send off your code review. The reviewer needs to flip between the test file and the production file, so they can use the test-as-documentation to understand what you intend the code to do, and help you find corner cases you missed in your test.

Here’s what’s wrong.

• You want to change code in the prod class with the corresponding unit test visible. Maybe you have an IDE plugin like TestDox that can navigate between the two classes based on the naming convention. If not, you have to navigate manually, and maybe change your window layout to show the two side-by-side.
• Even when you can see both prod and test code, you have to scroll around the two files to see the tests that cover the method you’re working on.
• You refactor the prod class, changing the name or package, and the test class now is misnamed or in the wrong place. Maybe the package-protected access breaks so the compiler tells you, or maybe your code reviewer notices. But your refactoring tool probably doesn’t help you, and it’s easy to check in this mistake.
• You changed the production code just to expose some fields for testing. You really want the test to have special access into the class internals that wouldn’t be allowed at production runtime, but the language doesn’t distinguish these two runtimes.
• As the code grows, you refactor the class by extracting some methods and pulling out some responsibilities into their own classes. You have to read through all the tests to understand what behavior should be extracted into a new test. Some tests are no longer unit tests, since they test behavior that’s now an interaction between the original class and the new class, so you might move those tests to a third location.
• It’s really annoying in the code review to correlate the test and prod changes, especially if your code review tool has a high latency when navigating between files.
• The test needs to be written as a class with methods, even though you should never create an instance of that class yourself, nor call any of the methods yourself.
• The test provides fantastic documentation, because it’s executable. The test always tells you the truth, even when comments get outdated. Sadly, that documentation is not found in your generated docs, because your doc tool doesn’t know where to find the tests. Some BDD frameworks have a separate tool to create the “spec” for the class, but it still doesn’t appear in places where the class API appears.

Sucky! Ok, so what are we going to do about it? Here’s what I propose in Noop:

Unit tests are a special entity, declared with a unittest keyword. The keyword is followed by a string literal, which allows you to write your intent in a normal sentence rather than “escaped” as a camel-case method name. They may appear either in a file dedicated to testing, or better, in the class being tested.

class TestThis() {
  String printHello(String name) {
    return "Hello %s!" % name;
  }
 
  unittest "It should print hello" {
    printHello("Fred") should equal("Hello Fred!");
  }
}

The unittest is a member of the class, just like a field or a method, so it naturally has intimate access to any fields or methods. And the test fixture is naturally provided as the “this” reference in the test, allowing you to simply call methods in the class.

Let’s deal with the objections now.

“Um, where does the ‘this’ instance come from?” -> I’m assuming here that the language has built-in dependency injection, so it’s normal to expect instances to be created for you. In the example above, it’s obvious how to make an instance of TestThis, so why should you have to make one yourself?

“Ok, but what about a less trivial constructor? What if the constructor needs some service that we want to mock?” -> Again, dependency injection to the rescue. You need a way to add “bindings” to the DI runtime, so that it understands how to provide instances of whatever objects you need in the constructor. These bindings need be declared in the unittest declaration, before the block that contains the logic and assertions, so that the ‘this’ reference is available in the first line of the block.

“I like having the tests in their own source root, like it’s been since the old days!” -> I think this is just an artifact of language tools that don’t understand tests, so the easiest way to compile the right set of sources, search for tests, and deal with dependencies was to create a separate source root. Starting with a fresh slate today, I don’t see why it’s necessary to continue to appease language tools at the cost of maintaining a separate package hierarchy and all the rest.

“But the production code shouldn’t depend on the tests, and the tests have extra dependencies that the production code doesn’t!” -> the first half is solved because the unittests aren’t methods, so there is no way to reference them in production code. The second half might require that there’s a way to make an import statement that only imports something for use by the tests. Maybe “test import junit.Asserts”? And the language tools need to provide a different set of dependency libraries in the classpath when running the tests vs. running production code

“But you don’t want to ship those tests to production!” -> well, I’m not sure why that’s a big deal, as there’s no way to call into that code without a test runner. But sure, the compiler or other language tools can support a “test” mode, just as many compilers have a “debug” mode to dictate whether the full symbol table is emitted.

“But this mixes testing and development!” -> You should really start doing Test-Driven Development. Testing is part of developing. They should be mixed.

I’m very keen on your comments on this one. Don’t leave me hanging!

2 Comments to Making the case for unit tests to live in the code they test

Olli Aalto
November 15, 2009

I’ve found that when the unittess can be written right next to the code that is being tested makes it easier to maintain the tests and even write them in the first place.

The D programming language has unittests builtin.
http://www.digitalmars.com/d/index.html

http://www.digitalmars.com/d/2.0/class.html#UnitTest

David Linsin
November 15, 2009

Thanks for the great blog post! I always enjoy readings of authors, that are as passionate about testing as I am. So here comes my 2 cents:

I think there’s nothing wrong with the process that we have today, except that the developer needs to care about his work. If he doesn’t – then the process we have today is highly flawed and we need something like you suggested in Noop.

For me, the approach we have today works great! I like writing tests and I like to make sure that my code works. I cannot imagine writing my tests in the production code, although I cannot come up with any objective arguments against it – it just doesn’t feel right.

One more thing: here’s a blog post of mine, about changing your production code – http://dlinsin.blogspot.com/2009/04/using-reflection-in-unit-tests-ii.html

Leave a comment