Tag: Unit tests

Software Craftsmanship and Heroism

A certain restaurant in the Silicon Valley has a sign which says “This is a bad place for a diet. This is a good place for a diet.” There is no contradiction – just 2 different contexts.

So it is with Software Craftsmanship and heroism. The opposite of heroism in software development is a smooth operation high up the CMMI ladder, where everyone knows what to do and no one wastes time or stresses out because the process is moving forward. On the bottom of the CMMI ladder, on the other hand, it’s sometimes chaos, and a developer can be put under pressure to deliver in a hurry because of bad communication or poorly designed and/or poorly tested code encountering demanding clients.

Software craftsmanship is bad for heroism because it reduces the chaos that requires a heroic developer to step in and fix things in a hurry. There’s a software factory pipeline that doesn’t let logic failures reach the client.

Software craftsmanship is good for heroism because sometimes even with a solid agile process and a finely-tuned software factory pipeline and clean, unit-tested code, misconceptions can slip in and a client can do something that the unit tests did not foresee and (rightly) complain about it. And why is craftsmanship good for heroism in this case? Because the code with the error is clean, extensible and unit-testable. If you have an idea for a fix, you can try it out in minutes in a unit test instead of having to code blindly, build, run and test manually and then repeat the process until it’s debugged or you decide to take a different approach. Ultimately, the software craftsperson has to build and run and test manually, but only AFTER the key idea for a solution has been tested and debugged. So with craftsmanship, you may have some shining moments of glory, but the heroic struggle happens gradually, every day, maintaining the quality of the code and the software factory.

Advertisements

To mock or not to mock

That is the question… debated in a recent civilized exchange of comments on this blog.

Like Ringo Star, I’m not afraid to position myself as pro-mock.

Of course mocking is not the goal, just a means of quickly writing tests which are focused on a small piece of logic. The goal is maintainable, thoroughly-tested code which functions as expected.

I’m not religious about mocking. If the method you’re testing has a collaborating class which you can just instantiate without any hassles and it works perfectly in the test from the start, then go for it. No need to mock. But that’s not something I see very often, apart from data structures or purely mathematical functions with no side effects. Most collaborating classes will use the network or disk or exhibit some non-repeatable behavior. At a minimum, there may be behavior that’s dependent on its state and which could evolve as the product evolves. You will likely need an integration test to show that your classes work together, but when you’re unit testing, you should be focused on the method you’re testing and not its collaborators.

For me (and many others), an interface is like a contract. If an interface is injected into the class I’m testing, I assume the object that implements it will do what’s expected, and I don’t care how it’s done. I would rather mock (or stub) the interface than write a test which gets involved with the details of its implementation. The contract doesn’t say the implementation won’t change, it just promises a certain behavior (a result and/ or side effects). If requirements evolve and the contract changes (different behavior, different return type, and/or different arguments), there will be some pain whether or not you use mocks, but it’s often more painful with mocks because many tests have to change (though the changes tend to be simple to implement). On the flip side, if, with no contract change, a bug slips into the implementation of the collaborating class, none of the tests which mock it will fail. If the creator of the collaborating implemented good unit tests, the bug will be spotted instantly in that case. On the other hand, without mocks, all the tests using the real collaborating class could fail at once, in which case it can take a little digging to find the bug.

There’s also the perhaps more frequent case of implementation details changing without a change to the interface. In that case, if the tests use mocks, there’s no issue at all. If, on the other hand, the tests call a constructor of the collaborating class which changes to add a new dependency, or if suddenly the collaborating class requires database access, all the tests without mocks are broken, often with no easy fix other than mocking.

For me, mocks are part of a modular approach to development. You develop little components, and testing quickly and simply with mocks you are confident that each component works exactly as specified. A few integration tests prove that the components can work together. This is the same sort of philosophy used with physical products such as cars. If you change the tire of your car, you don’t lose sleep over the fact that the tire has never been tested on the wheel in question. You assume that the tire has been tested, and you know that the wheel worked with a different tire with the same standards, and that’s that.

See which lines of code are really tested with PIT

PIT (a.k.a. pitest) is a mutation testing tool which is fairly easy to set up (in maven or gradle) for most java applications.

The basic idea with mutation testing is that the tool will iterate over each line of code, changing that 1 line, and then run all the tests against the mutated code base. If the changed line provokes an error in at least one test, it’s considered covered. If no test fails, then the line is not covered.

It’s possible to configure PIT to ignore your slow integration tests or end-to-end tests, which is a good idea, because PIT will run the same tests over and over again. With isolated unit tests, it’s pretty fast and a great coverage tool.

I tried it out on my interface-it project. Here’s a link to my pom.xml file which uses the pitest maven plugin. I struggled a little with the configuration of target classes, and their getting started document was slightly off, but it did not take long to set up.

Here are a few screen captures of the resulting report, which includes drill-down to see each line of code and an explanation of the mutation for the line and the result of the mutation :

Main page

pitestmethod

pitestexplanation

 

Key Software Factory Tools in the Java World

Here are some popular free and/or freemium tools for creating a software factory for a Java project :

I’ve written about these build tools before.

Here’s a chart showing the relative popularity of these 3 tools.

There are many more unit test facilitators available for free, and I’ve developed some lesser-known open-source unit test facilitators myself.

  • Integration test facilitators
    • Arquillian – allows you to run a test on a Java EE server (either a remote server or one configured and launched by your test code)
    • OpenEJB – a lightweight EJB framework implementation that can be launched in a test
    • DBUnit – set up and clean up after tests that use real database connections
    • Docker – actually, Docker is a facilitator for just about any sort of automation you want to do, but I put it in the integration test category because it allows you to spin up a database or other server from a test.

Note: do not mix the Arquillian and OpenEJB together in the same test suite – jar conflicts can ensue when tests are run manually in Eclipse

There are surely other tools I’ve left out.  Feel free to mention your favorites in the comments.

Improved matching error messages in Extended-Mockito

I’ve recently made some improvements to Extended-Mockito in the area of failure messages.

In early versions, messages from a failure to match in a verify() call were a bit cryptic, especially when matching based on lambdas.  This is what you used to get :

Wanted but not invoked:
exampleService.doAThingWithSomeParameters(
    <Extended matchers$$ lambda$ 6/ 1 5 0 2 6 8 5 4 0>,
    <Extended matchers$$ lambda$ 8/ 3 6 1 5 7 1 9 6 8>,
    <custom argument matcher>,
    <Extended matchers$$ lambda$ 1 1/ 2 1 0 5 0 6 4 1 2>
);

In the first of my examples with the new improvements (available starting in version 2.0.78-beta.1 of extended-mockito and transitively in version 0.9.0 of tdd-mixins-core and tdd-mixins-junit4 ) it’s now possible to show more clearly what kind of arguments were expected:

Wanted but not invoked:
exampleService.doAThingWithSomeParameters(
{String containing all of: [Butcher,Baker,Candlestick Maker]},
[All items matching the given Predicate],
[One or more items matching the given Predicate],
SomeBean where val1 > 5
);

For info, the expectation call which gives this failure message is:

verify(expecting).doAThingWithSomeParameters(this.containsAllOf("Butcher", "Baker", "Candlestick Maker"),
				allSetItemsMatch(s -> s.startsWith("A")),
				oneOrMoreListItemsMatch(s -> s.startsWith("B")),
				objectMatches((SomeBean o) -> o.getVal1() > 5, "SomeBean where val1 > 5"));

Does TDD “damage” your design?

I recently came across a couple articles that challenged some of my beliefs about best practices.

In this article, Simon Brown makes the case for components tightly coupling a service with its data access implementation and for testing each component as a unit rather than testing the service with mocked-out data access. Brown also cites David Heinemeir Hansson, the creator of Rails, who has written a couple of incendiary articles discouraging isolated tests and even TDD in general. Heinemeir Hansson goes so far as to suggest that TDD results in “code that is warped out of shape solely to accomodate testing objectives.Ouch.

These are thought-provoking articles written by smart, accomplished engineers, but I disagree with them.

For those unfamiliar with the (volatile and sometimes confusing and controversial) terminology, isolated tests are tests which mock out dependencies of the unit under test. This is done both for performance reasons (which Heinemeir Hansson calls into question) and for focus on the unit (if a service calls the database and the test fails, is the problem in the service or the SQL or the database tables or the network connection?). There’s also a question of the difficulty of setting up and maintaining tests with database dependencies. There are tools for that, but there’s a learning curve and some set-up required (which hopefully can be Dockerized to make life easier). And there’s one more very important reason which I’ll get to later…

Both Brown and Heinemeir Hansson argue against adding what they consider unnecessary layers of indirection. If your design is test-driven, the need for unit tests will nudge you to de-couple things that Brown and Heinemeir Hansson think should remain coupled. The real dilemma is where should we put the inevitable complexity in any design? As an extreme example, to avoid all sorts of “unnecessary” code you could just put all your business logic into stored procedures in the database.

“Gang of Four” member Ralph Johnson described a paradox:

There is no theoretical reason that anything is hard to change about software. If you pick any one aspect of software then you can make it easy to change, but we don’t know how to make everything easy to change. Making something easy to change makes the overall system a little more complex, and making everything easy to change makes the entire system very complex. Complexity is what makes software hard to change. That, and duplication.

TDD, especially the “mockist” variety, nudges us to add layers of indirection to separate responsibilities cleanly. Johnson seems to be implying that doing this systematically can add unnecessary complexity to the system, making it harder to change, paradoxically undermining one of TDD’s goals.

I do not think that lots of loose coupling makes things harder to change. It does increase the number of interfaces, but it makes it easier to swap out implementations or to limit behavior changes to a single class.

And what about the complexity of the test code? Brown and Heinemeir Hansson seem to act as if reducing the complexity of the test code does not matter. Or rather, that you don’t need to write tests for code that’s hard to test because you should just expand the scope of the tests to do verification at the level of whole components.

Here’s where I get back to that other important reason why “isolated” tests are necessary: math. J.B. Rainsberger simply destroys the arguments of the kind that Brown and Heinemeir Hansson make and their emphasis on component-level tests. He points out that there’s an explosive multiplicative effect on the number of tests needed when you test classes in combination. For an oversimplified example, if your service class has 10 execution paths and its calls to your storage class have 10 execution paths on average, testing them as a component, you may need to write as may as 100 tests to get full coverage of the component. Testing them as separate units, you only need 20 tests to get the same coverage. Imagine your component has 10 interdependent classes like that… Do you have the developer bandwidth to write all those tests? If you do write them all, how easy is it to change something in your component? How many of those tests will break if you make one simple change?

So I reject the idea that TDD “damages” the design. If you think TDD would damage your design, maybe you just don’t know how bad your design is, because most of your code is not really tested.

As for Heinemeir Hansson’s contention that it’s outdated thinking to isolate tests from database access, he may be right about performance issues (not everyone has expensive development machines with fancy SSD drives, but there should be a way to run a modest number of database tests quickly). If a class’s single responsibility is closely linked to the database, I’m in favor of unit-testing it against a real database, but any other test that hits a real database should be considered an integration test. Brown proposes a re-shaped, “architecturally-aligned” testing pyramid with fewer unit tests and more integrated component tests. Because of the aforementioned combinatorial effect of coupling classes in the component, that approach would seem to require either writing (and frequently running) a lot more tests or releasing components which are not exhaustively tested.

A Testing Toolbox for Java 8

A few months ago, I presented interface-it, a java-8 tool to generate mixin interfaces. In case you don’t know what that means or why mixins could be useful to you, I wrote a short article which explains it.

One of the key motivations for the interface-it tool was to be able to generate mixins for the latest versions of unit-testing libraries like Mockito and AssertJ. Now you no longer have to worry about that, because I’m doing it for you. And more.

I now have several more projects to present to you.

Presenting tdd-mixins-junit4

Working backwards – if you want to have a great test fixture by adding only one dependency in your build configuration (your Maven Pom, Ivy xml, Gradle, or just a fat jar added to your classpath), use tdd-mixins-junit4. It gives you all the basics you need to do mocking and assertions with fluidity, simplicity and power.

Normally, that’s all you should need for your tests. Mockito allows you to to set up collaborating objects and verify behavior, and the extensions I added make it even easier to handle cases where, for example, you want to mock behavior based on arguments passed to the mock which are generated by your unit under test. As for verifying returned results, AssertJ and JUnit assertions allow you to verify any data returned by the unit under test.

Presenting tdd-mixins-core

If you do not want to use JUnit 4 (maybe you want to use TestNG or an early version of JUnit 5), then you can use tdd-mixins-core, which has everything that tdd-mixins-junit4 has, except the mixin for JUnit assertions and JUnit itself.

Presenting extended-mockito

So these tdd-mixins libraries notably give you mixins for the aforementioned libraries Mockito and AssertJ. As for Mockito, they use my extended-mockito (TODO: link to project) library, which not only provides mixins for classes like Mockito and BDDMockito, but it also provides extra matcher methods to simplify specifying matching arguments for mocked methods. For example:

when(myMockSpamFilter.isSpam(containsOneOrMoreOf("Viagra", "Cialis", "Payday loan")))
            .thenReturn(Boolean.TRUE);

See the project’s home page or the unit tests for more details.

Presenting template-example

As for AssertJ, it’s already quite extended for general purpose use, so there is no extended-assertj project, but if you want to take things farther, I did create a project called template-example, which shows how, with a little tweaking, you can use a Maven plugin to auto-generate custom assertions for your own JavaBeans which are combined with the AssertJ mixin from tdd-mixins-core. These custom assertions allow you to do smooth, fluent assertions for your own data types, allowing this sort of validation call:

assertThat(employee).employer().address().postalCode().startsWith("11");

With these tools, you can more productively write unit tests with powerful assertions and mocking. They give you a fixture that you can set up in any test class by implementing an interface or two – for example:

public class MyTest implements ExtendedMockito, AllAssertions {

Not Included

What’s missing from these tools? I wanted to keep the toolset light, so there are some excellent but more specialized tools which are not included. For example, I have generated a mixin for Jsoup, which is very useful if you need to validate generated HTML, but unless I hear a clamoring for it, I will leave it out of tdd-mixins-core because it adds a dependency that lots of people may not need. Same for extensions to AssertJ – I generated mixins for AssertJ-DB  and for AssertJGuava (UPDATE: also added one for Awaitility), but did not include them in tdd-mixins (you can copy and paste the generated mixins’ source files if you want to use them).

Another library which is useful but which does not lend itself to mixins (because it uses annotations rather than static calls) is Zohhak it simplifies testing methods which return results that depend on a wide variety of possible input values (such as mathematical calculations or business rules).

The What and Why of Mixins in Java 8

A mixin is a type which can be mixed in to (i.e. included in) other types via multiple inheritance. This allows the encapsulation of certain behaviors which cut across the main type hierarchy. In C++ this is fairly standard, though it’s sometimes discouraged because of the potential for the “diamond problem”. In Java 8, mixins can be created using interfaces with default methods.

Why use mixins in Java?

Mixins give you design options which you did not have before. A class can inherit behavior that isn’t in its superclass. For example, you can have Goose implements Honker and Car implements Honker – with no need to implement honk() more than once. Some more practical uses include test fixtures (which I will address in more detail in my next article), and logging.

Default methods in mixins are also very useful for wrapping static method calls, especially in legacy code. This is what my interface-it tool automates. So you can replace static calls with calls to interface methods that can be mocked in tests, and/or overridden for special cases. It allows you to make a procedural design more object-oriented and pull hidden dependencies out into the open. It also lets you avoid using static imports, which can make the code less readable.

Finally, a benefit of mixins is polymorphism.  You can override or extend the default behavior in special cases, which is something you can’t do with static calls.

TDD is our friend, but…

“Animals are our friends!!”, comedian Bob Goldthwait used to shout, adding “But they won’t lend you money.”

One famous definition of legacy code (from Michael Feathers) is “code without tests”. I would extend that definition to say that it’s code which does not have tests for the behavior you want to have. TDD will get you code that is well-tested for the behavior you want when you develop it. If your code does anything innovative, some of the behavior you want will almost certainly change over time.

So TDD is our friend, but it won’t eliminate all legacy code. What it will do is make it easier to know when we accidentally change behavior. It doesn’t necessarily make it easy to change our code’s behavior, either, but it gives us confidence to tackle the change because we know that tests will catch and allow us to fix any little mis-steps before they become more difficult and time-consuming to find and fix, and, importantly, it nudges us constantly to design our code in a way which allows it to be refactored with less trouble and risk.

NIO2 is great… until…

In Java 7, Oracle gave us a shiny new API to do dingy old file-related tasks with less boilerplate code (and with shiny new functionalities like file change listeners, memory mapping, asynchronous I/O, etc.), and in Java 8 Oracle extended the API to use Streams.

I recently discovered that it’s great… until you need to delete a directory on Windows. The odd thing is that I really wasn’t using much NIO code.

I have integration tests where the code under test creates a temporary directory (not using NIO). The code under test test writes a file into that directory (not using NIO). The test used NIO (Files.lines()) to read the generated file and a static file for comparison. The post-test clean-up deletes the contents of the temporary directory and then the temporary directory itself (not using NIO).

Without using NIO to do the deletion, I had only a boolean (the result of File.delete()) to tell me that the directory was not actually deleted. Switching to NIO (Files.delete(), which throws an exception when it fails), I could see that the deletion failed because the directory was supposedly not empty. This might be caused by the never-to-be fixed bug 4715154 in the JDK.

I tried looping with delays and forced garbage collection (as suggested in this StackOverflow comment), but to no avail. When I got rid of NIO in the test code that reads the files for comparison, then the deletion worked. All’s well that ends well, I suppose, and the hour spent debugging that issue was educational.

I thought it was worth writing up a warning about this, though, because when you’re doing TDD you don’t expect to spend an hour debugging code in your @AfterClass method which does nothing but delete a directory and its contents.