Category: Tools

See which lines of code are really tested with PIT

PIT (a.k.a. pitest) is a mutation testing tool which is fairly easy to set up (in maven or gradle) for most java applications.

The basic idea with mutation testing is that the tool will iterate over each line of code, changing that 1 line, and then run all the tests against the mutated code base. If the changed line provokes an error in at least one test, it’s considered covered. If no test fails, then the line is not covered.

It’s possible to configure PIT to ignore your slow integration tests or end-to-end tests, which is a good idea, because PIT will run the same tests over and over again. With isolated unit tests, it’s pretty fast and a great coverage tool.

I tried it out on my interface-it project. Here’s a link to my pom.xml file which uses the pitest maven plugin. I struggled a little with the configuration of target classes, and their getting started document was slightly off, but it did not take long to set up.

Here are a few screen captures of the resulting report, which includes drill-down to see each line of code and an explanation of the mutation for the line and the result of the mutation :

Main page

pitestmethod

pitestexplanation

 

When an upgrade is a downgrade

I recently blogged about software factory tools, but now I have to rant about one of those tools.

Sometimes, it seems, for non-technological reasons, a software editor will, perhaps intentionally, make the new version of its software worse than the previous one. Windows 8 comes to mind.

I did not expect a minor release of SonarQube server to fall into this category, but release 6.2 seems like a man-made disaster to me. The creators of SonarQube decided that it would be a good idea to suddenly and irrevocably remove several of its central features. They removed the distinction between unit and integration tests because it’s too hard for them to deal with. That could really mess up the data in your customized reports, but don’t worry – they also deleted your custom reports! All of them!

Apparently they removed dashboards and widgets because they don’t want SonarQube to be “seen as a multi-purpose aggregation and reporting tool”.  It’s true that as of 6.1 the reports they generated automatically were pretty good, but the reports you meticulously set up that your boss has been using in executive progress reports ? Gone forever. The only alternative is restoring the SonarQube database and re-installing version 6.1 without ever having the possibility to upgrade again. How’s that for progress?

Key Software Factory Tools in the Java World

Here are some popular free and/or freemium tools for creating a software factory for a Java project :

I’ve written about these build tools before.

Here’s a chart showing the relative popularity of these 3 tools.

There are many more unit test facilitators available for free, and I’ve developed some lesser-known open-source unit test facilitators myself.

  • Integration test facilitators
    • Arquillian – allows you to run a test on a Java EE server (either a remote server or one configured and launched by your test code)
    • OpenEJB – a lightweight EJB framework implementation that can be launched in a test
    • DBUnit – set up and clean up after tests that use real database connections
    • Docker – actually, Docker is a facilitator for just about any sort of automation you want to do, but I put it in the integration test category because it allows you to spin up a database or other server from a test.

Note: do not mix the Arquillian and OpenEJB together in the same test suite – jar conflicts can ensue when tests are run manually in Eclipse

There are surely other tools I’ve left out.  Feel free to mention your favorites in the comments.

Software factories

In the real world, most of us are not quite there, but it’s now possible with only free tools to have a reliable product release process which consists of just developing and committing code. In the case of an editor releasing a versioned software product, or for a site where new features are released gradually or via AB testing, or in all cases where human testers and/or management approval are necessary, there would also have to be the click of a button to approve a release. I’m speaking in broad, hand-waving generality, but such a process, when achieved, can provide a competitive advantage to a product or site by reducing time-to-market for new features and reducing the risk of quality issues (translation: easier to develop faster with fewer bugs).

Of course setting up such a process requires a lot of up-front work, considerable expertise, and occasional maintenance down the road, and such a system will (by design, normally) come to a halt very quickly if it is fed anything other than clean, well-tested code.

This notion has been called a “software factory” or “continuous delivery” or “continuous deployment” system.

I’ll have more to say on this subject soon, but today I will just point you to a couple of interesting diagrams:

Dan North’s JGoTesting library

Dan North, who is a pioneer in the field of Behavior-Driven Development (or rather “Behaviour-Driven” – he’s British) and a very entertaining public speaker on agile/extreme development practices, has created a new Java library which brings some ideas from the Go language to JUnit 4. His library JGoTesting is in the early stages, and there are some aspects which only seem useful for retrofitting existing tests. There’s also some competition from JUnit 5 and AssertJ for accomplishing some of the same goals, but it’s worth trying out.

The major goal of JGoTesting is to allow a single test to log multiple failures. In other words, the test goes on after the first failure, and at the end the tests fails with a list of errors. This is also possible with AssertJ SoftAssertions, and it’s standard functionality in JUnit 5 with assertAll(). One nice bonus in JGoTesting is that you can also log messages which are ignored if the test succeeded. So for integration tests or tests involving random generated values, you can get extra info about the cause of a failure without log pollution in the case of tests which succeed.

JGoTesting offers a testing DSL which distinguishes between failed checks and more serious errors. Syntactically, validations and logging can be called from the unique rule object for the test class. Calls can be chained together, and the calls are basically lambda-friendly for tests in Java 8 (JGoTesting is compiled in Java 7 for wider compatibility). JGoTesting also offers static delegate calls to replace calls to JUnit 4 assertions. These only seem useful for converting existing tests. As someone who auto-generates non-static mixin delegate calls for such API’s (TODO : LINK to tdd-mixins-junit4), I can’t really be a fan of replacing one static call by another, especially when there’s already a ready-made object with all the necessary semantics in non-static methods.

I took JGoTesting for a test drive (code available on github) to see what it can do and to see if it plays well with other API’s – I tried it with Zohhak, AssertJ and Mockito (the latter two via my tdd-mixins-junit4 library). Because JGoTesting uses a rule instead of a runner, there’s no conflict between it and Zohhak’s runner.

I discovered that JGoTest can be used in conjunction with Mockito verify() and with AssertJ assumptions and SoftAssumptions. JGoTest failed checks and logs which are executed before the failing error (whether it’s a Mockito failure, an AssertJ failure or a JGoTest failure, do indeed appear in the failure logs whether the test is run from Maven or from Eclipse. Maven gets a bit confused about the number of actual tests when there are JGoTest check failures.

My Maven output is available online if you don’t have time to run the test yourself, and here are some screen captures of results when run from Eclipse:

jgomockitowithlog

jgosoft1

Conclusion:

I don’t write a lot of tests which need a tool like JGoTest. I prefer tests which are focused on one case. For tests like that, there’s no need to log extra information or multiple failures– knowing the name of the test and the fact that it failed is enough to detect and quickly resolve the problem. For tests which are more integrated (with longer scenarios), or which test a number of possible parameters in a loop (as opposed to Zohhak-style parameterized tests), JGoTest’s log feature could be helpful to indicate where the test went wrong. As far as checking multiple failures in a test goes, while I like the simple syntax of JgoTest, I prefer the output generated by AssertJ SoftAssumptions. These are very early days for JGoTest, so I will keep an eye on it to see how it improves over time.

Java build tools: Ant, Maven or Gradle?

When you create a java project (for a library or an application or a web site), while you could just rely on your IDE to build things for you, in most cases you really need a build script. For a build script, you basically have 3 choices: Ant, Maven or Gradle.

Now that I’ve used all three of these tools, thought I’d share some thoughts about how to choose between them. This is not an article about how to migrate from one of them to another – that’s usually not worth the pain if your build is complicated, and it’s fairly trivial if your build is very simple.

In the context of a new project, let’s take a look at each of the choices:

Ant

Ant is the old-school tool. Your build script is basically a list of configured commands in XML, organized as a set of interdependent tasks. It’s easy to learn, and relatively easy to debug. It’s pretty easy to extend with custom tags implemented in Java (I’ve created one myself). Ant does not naturally include dependency management (loading external jar files), bit that can be added with Apache Ivy.

Strong points:

  • easy to learn
  • easy to understand
  • excellent integration in IDE’s
  • lots of examples and help online

Weak points:

  • Procedural (build scripts can get very complicated)
  • inelegant
  • For dependency management, Ivy is less intuitive and requires extra configuration and tooling

Maven

Maven is the opinionated tool which forces a project structure and build life cycle on you, and it’s also the tool that introduced built-in dependency management and a central repository for dependencies. Like Ant, it’s based on XML. Some people hate Maven for making them do things Maven’s way, but the benefit is simplicity. Unlike with Ant and Gradle scripts, two Maven scripts which do roughly the same thing are very similar. I have not tried to extend its functionality, but Maven has a plugin architecture, and a lot of useful plugins are available. Maven is also well-integrated in a number of popular tools.

Strong points:

  • generally very little thought or debugging required to get a typical build working
  • strong integration with various CI and IDE tools
  • simple dependency management
  • lots of examples and explanations online

Weak points:

  • Lots of pain if you don’t organize your project the Maven way
  • inelegant

Gradle

A relative newcomer, Gradle offers an elegant groovy-based DSL to run your build. Like Maven, Gradle has built-in dependency-management and makes some assumptions about project structure, but like Ant its execution is based on tasks. Unlike both Ant and Maven, Gradle has an elegant syntax which allows you to essentially write your build code in Groovy. Unlike Maven, the assumptions about project structure are conventions rather than rules. I have to say, though, that I recently encountered an official Gradle plugin which takes some of those assumptions as hard rules, and I had to abandon the plugin and use another tool. Hopefully this situation will improve over time. One thing I really like about Gradle is its dependency management syntax, which allows much simpler find-replace for version upgrades than Maven’s XML tags. The library name and version are all together on one line.

Strong points:

  • Elegance (i.e. lack of XML tags)
  • flexibility
  • simple dependency management
  • concision

Weak points:

  • Immature tooling and immature integration with popular tools (CI, IDE)
  • limited examples and help online
  • steep learning curve (especially for Groovy noobs)
  • inflexible plugins.

Conclusion

All three of these tools are good tools which can get the job done if you know what you’re doing. Personally, for a new Java project, I’d chose Maven for basic building of jar and war files. It’s what I did choose, even for my open-source Ant plugin. I Ant also, for non-build tasks (mass file copying with text replacement, for example) and tasks around the builds (for example, to update versions of dependency libraries and to rebuild the delegate interfaces using interface-it ant). Gradle is a very elegant and promising tool which will surely resolve some of its weak points over time. If you’re comfortable with groovy, it might be the right choice for you, but only if you stick to their recommended multi-project structure and don’t lean too heavily on your IDE.