Author: redgreenwp

Ethics

I recommend the video on youtube of the excellent presentation on ethics in software development which was given at this year’s Devoxx UK by Steve Freeman (of GOOS fame) and Jonathan Rothwell. Ethics is an important element of software craftsmanship, which I have discussed previously here. We’re not necessarily in charge of what happens in the world, but nothing happens without software these days, so nothing happens without our involvement. Involvement implies moral choice, and moral responsibility for the choices we make.

Particularly troubling in the presentation is the mention of an article by a journalist who stumbled upon software developers at a Nazi rally in Seattle, WA. I have tried to steer clear of politics in this blog, but maybe we can no longer separate code from politics. Can I no longer assume that all who read this blog are “fine people”? I may have to take political positions on this blog now because evil has come to our digital world as our digital world exerts more and more influence over the physical world. Code is neither good nor bad. It’s a tool. But we have a responsibility to advocate that it, like other tools, be used for good. I like that the internet has made it easy to express our uniqueness and made it possible for people who feel different to find others who are like-minded. There’s no more “normal”, and that’s probably good, but hate should never be acceptable.

Freeman and Rothwell also cite several times the excellent writings/talks of Maciej Ceglowski, who I consider to be the Socrates of Silicon Valley. He is the cold shower that pulls us out of sci-fi fantasies and into the real world that we software developers are really creating. The reality of what we’ve created is not always as pretty as the story that was told to sell its creation, and behind the idealism of changing the world there is greed.

We might want to think about protecting privacy and human attention and even truth, which have been increasingly under attack by software developers, whether intentionally or as a side effect. Even the environment is under threat by software, as the ponzi scheme that is crypto-currency demands ever-increasing energy resources in order to insure scarcity.

There is hope. Personally, I’ve been fortunate to be able to work recently on software that facilitates compliance with Europe’s GDPR law, which is an important step forward for the protection of privacy.

Advertisements

Code is Forever

If you still need another source of motivation for taking care to write clean code, now there’s the Software Heritage Foundation. It’s a non-profit organization dedicated to preserving for the long haul all the world’s source code (or at least all the code found in public repositories like Github). It’s early days for them, but they have already archived and normalized a huge amount of code, which they make available for browsing and data mining. They’re looking for donations, as well as volunteers to help with documentation and coding adapters to repositories they haven’t yet tackled.

So now we can imagine that 100,000 years from now, there will be some archaeologists who, after browsing through the photos of you posted by your “friends” on Facebook (“Ah, look. Another one puking at a party. Why did they do that ?”), they will then run FindBugs or some more advanced analysis on your code (“How about that ? The puker’s lifetime code has 355,000 copy and pastes, a cyclomatic complexity per method of 18.7 and 23,000 abstraction leaks.”). And if you think the code you’re working on is secret, well in 50 years it will probably be in a public repository.

So take good care of your code – it will be around for a long time, and you never know who will be reading it.

Is StackWalker Useful?

In Java 9, Oracle plans to introduce the StackWalker class and API.

The old way (since Java 1.4) of getting stack trace info was to call:

StackTraceElement[] stackTrace = new Throwable().getStackTrace();

The new API provides more choices with a stream-based API. To do roughly the same thing as the old example, you’d call:

List<StackFrame> stack = StackWalker.getInstance().walk(s -> s.collect(Collectors.toList()));

It’s cleaner to be able to get stack trace information without having to create an otherwise superfluous Throwable object, and the new API facilitates recovering the Class object, rather than just the qualified class name.

However, there seems to have been no effort to retrofit Throwable for the new API. Most of the time, it’s when we’re dealing with exceptions that we’re interested in stack traces. It would be nice to have a slick stream-based or builder API which allows better control over which stack frames are printed in the logs. This is especially important for Java EE, where the code you wrote is at times a needle in the haystack of layers of proxy calls and reflection. It is possible to create such an API in Java 8, though its slickness would be limited by the fact that it would need to be applied on top of existing Throwable and Exception classes (you’d have to call it just before logging the exception, perhaps making a deep clone of the original exception with altered stack frame elements). It would be nicer to be able to add a filtering predicate or a more complete object we might call a StackTracePrinter to the Exception object before logging.

So I’m not sure in what cases I’d actually use StackWalker. It could be useful for one-off diagnostics that never get committed to production. Also, in the debugger you have the full stack trace in the UI, but maybe it could be faster to walk the stack in the debugger to find certain targeted elements. I’m stretching it a bit, because I don’t see a lot of use cases. Feel free to add a comment if you have a great use case for StackWalker which is not possible pre-Java 9.

A gentle introduction to Domain Driven Design

Domain Driven Design (DDD), first described in Eric Evans’s so-called “Blue Book” is a diverse set of practices. Because these practices are loosely associated and not tied to any specific tool, it can sometimes even be unclear whether you are in fact doing DDD or something else.

There are, however, some core practices in DDD which are worth understanding and doing. The goal of DDD is to put the focus of developers on the domain by removing so-called “accidental” complexity (a key goal for any software craftsperson). Some of the DDD ideas and practices can also be viewed as bringing some of the benefits of functional programming to the object-oriented world.

Key ideas:

Ubiquitous language

The idea of ubiquitous language is simple: The code is developed to solve a problem for some people, so developers and their code should use the same terms as the people for whom they are solving the problem.
This seems like common sense, except that it’s rare in practice for institutional/organizational reasons. Developers are off in their cloister building stuff using the language of developers and architects, and they end up having to translate between what they are building and what the domain experts and end users are expecting. This is accidental complexity which increases the likelihood of and the difficulty of correcting specification problems.

Object types:

DDD divides objects into several categories. Objects which fall clearly into one of these categories are easier to understand and work with in the context of a business domain…

  • Value objects

A value object represents a piece of information in the domain which does not change but which could be replaced. For example, a bank account could contain $55. The value object is a currency object with a numeric value of 55 and a currency type value of “USD”. It could be replaced by a value of $85 if a deposit of $30 is received.
A value object can normally be immutable (which makes it easier to do functional-style processing with no confusing side effects). If a value object is replaced by another value object of the same type with identical values, no one notices or cares.

  • Entities

An entity has identity and state. A customer is a good example of an entity. The customer might have a customer id. Unlike value objects, entities are not interchangeable. An entity will often have “invariants” which must be enforced (do not expose any public methods, including constructors, which could allow violation of an invariant, unless it’s absolutely required by a framework you need to use). An invariant is a logical condition which applies to an entity (for example, an order always has a product ID and a customer ID and an order date, or a Flight always has a departure airport code and an arrival airport code).

  • Aggregates

An aggregate is a cluster of associated objects with a “root”, which acts as a façade – objects outside the aggregate interact only with the root. The aggregate root is responsible for assuring in each of its methods that by the end of the method call all invariants are satisfied within the aggregate (within a single transaction, generally). An example of an aggregate root can be found in this article by Julie Lerman.

  • Domain Events

A domain event encapsulates an event which could potentially trigger actions in various components of your system.  The use of domain events is a kind of decoupled publish/subscribe design pattern (similar to Observer) which allows code to emit events which are handled without being responsible for how and by which components they are handled.

Modules

You already know what a module is in general. In DDD, according to Eric Evans, modules should “tell the story of the system and contain a cohesive set of concepts”, and their names should be part of the ubiquitous language and “reflect insight into the domain”.

Repositories

Repositories are objects which allow retrieval of entities, aggregate roots. In DDD a repository serves as an adapter between the domain object model and the persistence model.

Bounded contexts

A complex application may have to solve different problems for different people. Sometimes the problems are similar enough to each other that you’re tempted to solve both using the same objects and paradigms. Don’t. It’s been drilled into most of us that you need to reuse objects (and database tables) as much as possible to reduce the amount of code (DRY, which is a useful practice in general), but DDD optimizes above all to minimize confusion and cognitive load rather than minimizing lines of code. A bounded context means that you solve problem A with one set of classes using one set of terms and approaches and you solve problem B with a separate set of classes using distinct terms and approaches. Even if subsystems A and B have to communicate with each other, it’s generally easier in the long run to translate between A objects and B objects for communication than to mix A objects into the B implementation or vice-versa.

Shared Kernel

Speaking of our bounded contexts A and B, there can be a core domain which is used in both context A and context B. This is called a shared kernel.
In practice, a shared kernel is a set of interfaces and objects (typically in its own library, such as a jar file in Java) which allow code outside a bound context to interact with the context. This could be service interfaces, aggregate interfaces or DTO’s (data transfer objects – generally behavior-free objects used to pass entity or aggregate data to and from a service implementation). Since the shared kernel can be used in many different places by different teams and products, it should be as stable as possible.

Supple Design

Supple design is a set of patterns/practices, some borrowed from functional programming, which make it easier to reason about behavior of a part of the system without worrying about what the rest of the system is doing.

This includes:

  • Intention-revealing interfaces
    The idea is that an interface (via method names, signatures, etc.) reveals what it does, not how it’s done.
  • Side-effect-free functions
    If a method never changes the state of anything, other than perhaps creating one or more completely new objects, then it won’t cause deadlock or change the behavior of other methods being executed simultaneously. And if called again with the same inputs, you get the same result. It’s easy to verify with unit tests that there are no bugs.
  • Standalone classes
    Reduce coupling between different classes to reduce unnecessary complexity.
  • Assertions
    The idea is to enforce invariants and post-conditions of operations either in the implementation code or with unit tests (or both).

Associated design practices/architectures:

  • A Hexagonal Architecture makes DDD easier because it further separates the technical infrastructure from the domain.
  • CQRS with or without Event Sourcing is a complementary architecture/practice which simplifies the handling of data.  It’s a big topic beyond the scope of this article, but CQRS is a useful choice for reducing hidden complexity in the spirit of DDD by separating things that do not have to go together (queries and data modification).  Event sourcing is a useful technique for dealing with data that may change over time if you might ever need to know what the state was at a given moment in the past.
  • Behavior-driven development (BDD) is complementary to DDD because it encapsulates the domain in tests based on the ubiquitous language.
  • TDD is a good practice for pretty much any development project, and ensures that DDD assertions are validated early and often.
  • Functional programming (lambdas and streams in Java, for example) works well where supple design is applied.
  • Simple CRUD can coexist with DDD.  Some persisted objects are simple enough that there’s no need for assertions and other domain logic.

Getting started

Well, that’s a lot of interesting ideas, but how to get started using them in development projects? It’s not like a framework where there’s a tutorial that will give you a “Hello DDD” in half an hour.

Some early steps toward DDD are possible, though :

  • Start using value objects.  For example, you can replace numeric dollar amounts by immutable Money objects which have an amount and a currency code.
  • Start using a ubiquitous language.  Align the terminology used in the code and project structure with the domain language.
  • Identify and segregate modules and bounded contexts. You may already have some notion of modules, but maybe you can find that you’re sharing some code between them which shouldn’t be shared.
  • Get started using domain events.  There are some examples on the internet that help you get started, including this C#-oriented article by Udi Dahan.
  • Separate out shared kernels. The flip side of enforcing bounded contexts is sharing precisely what your module needs to share with other modules and the outside world.
  • Use repositories. You may already be doing this.  The key idea is to have domain code using domain storage logic (CRUD and search for entities and aggregates) rather than database logic.

 

Software Craftsmanship and Heroism

A certain restaurant in the Silicon Valley has a sign which says “This is a bad place for a diet. This is a good place for a diet.” There is no contradiction – just 2 different contexts.

So it is with Software Craftsmanship and heroism. The opposite of heroism in software development is a smooth operation high up the CMMI ladder, where everyone knows what to do and no one wastes time or stresses out because the process is moving forward. On the bottom of the CMMI ladder, on the other hand, it’s sometimes chaos, and a developer can be put under pressure to deliver in a hurry because of bad communication or poorly designed and/or poorly tested code encountering demanding clients.

Software craftsmanship is bad for heroism because it reduces the chaos that requires a heroic developer to step in and fix things in a hurry. There’s a software factory pipeline that doesn’t let logic failures reach the client.

Software craftsmanship is good for heroism because sometimes even with a solid agile process and a finely-tuned software factory pipeline and clean, unit-tested code, misconceptions can slip in and a client can do something that the unit tests did not foresee and (rightly) complain about it. And why is craftsmanship good for heroism in this case? Because the code with the error is clean, extensible and unit-testable. If you have an idea for a fix, you can try it out in minutes in a unit test instead of having to code blindly, build, run and test manually and then repeat the process until it’s debugged or you decide to take a different approach. Ultimately, the software craftsperson has to build and run and test manually, but only AFTER the key idea for a solution has been tested and debugged. So with craftsmanship, you may have some shining moments of glory, but the heroic struggle happens gradually, every day, maintaining the quality of the code and the software factory.

To mock or not to mock

That is the question… debated in a recent civilized exchange of comments on this blog.

Like Ringo Star, I’m not afraid to position myself as pro-mock.

Of course mocking is not the goal, just a means of quickly writing tests which are focused on a small piece of logic. The goal is maintainable, thoroughly-tested code which functions as expected.

I’m not religious about mocking. If the method you’re testing has a collaborating class which you can just instantiate without any hassles and it works perfectly in the test from the start, then go for it. No need to mock. But that’s not something I see very often, apart from data structures or purely mathematical functions with no side effects. Most collaborating classes will use the network or disk or exhibit some non-repeatable behavior. At a minimum, there may be behavior that’s dependent on its state and which could evolve as the product evolves. You will likely need an integration test to show that your classes work together, but when you’re unit testing, you should be focused on the method you’re testing and not its collaborators.

For me (and many others), an interface is like a contract. If an interface is injected into the class I’m testing, I assume the object that implements it will do what’s expected, and I don’t care how it’s done. I would rather mock (or stub) the interface than write a test which gets involved with the details of its implementation. The contract doesn’t say the implementation won’t change, it just promises a certain behavior (a result and/ or side effects). If requirements evolve and the contract changes (different behavior, different return type, and/or different arguments), there will be some pain whether or not you use mocks, but it’s often more painful with mocks because many tests have to change (though the changes tend to be simple to implement). On the flip side, if, with no contract change, a bug slips into the implementation of the collaborating class, none of the tests which mock it will fail. If the creator of the collaborating implemented good unit tests, the bug will be spotted instantly in that case. On the other hand, without mocks, all the tests using the real collaborating class could fail at once, in which case it can take a little digging to find the bug.

There’s also the perhaps more frequent case of implementation details changing without a change to the interface. In that case, if the tests use mocks, there’s no issue at all. If, on the other hand, the tests call a constructor of the collaborating class which changes to add a new dependency, or if suddenly the collaborating class requires database access, all the tests without mocks are broken, often with no easy fix other than mocking.

For me, mocks are part of a modular approach to development. You develop little components, and testing quickly and simply with mocks you are confident that each component works exactly as specified. A few integration tests prove that the components can work together. This is the same sort of philosophy used with physical products such as cars. If you change the tire of your car, you don’t lose sleep over the fact that the tire has never been tested on the wheel in question. You assume that the tire has been tested, and you know that the wheel worked with a different tire with the same standards, and that’s that.

See which lines of code are really tested with PIT

PIT (a.k.a. pitest) is a mutation testing tool which is fairly easy to set up (in maven or gradle) for most java applications.

The basic idea with mutation testing is that the tool will iterate over each line of code, changing that 1 line, and then run all the tests against the mutated code base. If the changed line provokes an error in at least one test, it’s considered covered. If no test fails, then the line is not covered.

It’s possible to configure PIT to ignore your slow integration tests or end-to-end tests, which is a good idea, because PIT will run the same tests over and over again. With isolated unit tests, it’s pretty fast and a great coverage tool.

I tried it out on my interface-it project. Here’s a link to my pom.xml file which uses the pitest maven plugin. I struggled a little with the configuration of target classes, and their getting started document was slightly off, but it did not take long to set up.

Here are a few screen captures of the resulting report, which includes drill-down to see each line of code and an explanation of the mutation for the line and the result of the mutation :

Main page

pitestmethod

pitestexplanation

 

When an upgrade is a downgrade

I recently blogged about software factory tools, but now I have to rant about one of those tools.

Sometimes, it seems, for non-technological reasons, a software editor will, perhaps intentionally, make the new version of its software worse than the previous one. Windows 8 comes to mind.

I did not expect a minor release of SonarQube server to fall into this category, but release 6.2 seems like a man-made disaster to me. The creators of SonarQube decided that it would be a good idea to suddenly and irrevocably remove several of its central features. They removed the distinction between unit and integration tests because it’s too hard for them to deal with. That could really mess up the data in your customized reports, but don’t worry – they also deleted your custom reports! All of them!

Apparently they removed dashboards and widgets because they don’t want SonarQube to be “seen as a multi-purpose aggregation and reporting tool”.  It’s true that as of 6.1 the reports they generated automatically were pretty good, but the reports you meticulously set up that your boss has been using in executive progress reports ? Gone forever. The only alternative is restoring the SonarQube database and re-installing version 6.1 without ever having the possibility to upgrade again. How’s that for progress?

Unit Test Culture

Nobody gets paid proportionally to the number of unit tests they produce. That’s a good thing.

But how, then, do we motivate the many developers who think that they should do unit tests and/or TDD but don’t? It’s not easy to do serious unit testing and TDD when there’s legacy code (i.e. code without unit tests), and it’s never the most important or urgent thing.

The short answer is “culture”. The whole team should understand that you write unit tests and do TDD because it’s what a professional developer does. A professional surgeon washes hands before surgery. A professional chef makes sure the kitchen is clean at closing time. You get the idea.

It also helps to have a software factory set up where the coverage rate is tracked. Requiring a specific coverage rate is a questionable practice because 100% is sometimes too much (for example, if you need to implement a 3rd-party interface with 50 methods just so you can use one of them, do you have to write bogus unit tests for the 49 empty method implementations which are required by the compiler?), and less than 100% can lead to developers avoiding testing the most difficult-to-test but most important code. However, it’s motivating to see the effect that one’s commit has on the coverage number, be it the overall coverage or the coverage for the specific classes or packages which were modified.