Month: December 2015

Uncle Bob’s Oath Proposal

Robert C. Martin recently proposed an oath for the software professional. I applaud Uncle Bob’s efforts to encourage higher standards of professionalism, which have inspired me personally, but his oath is based on premises which seem unrealistic to me. The oath has already received much intelligent criticism, from Ron Jeffries, for example. Uncle Bob seems to want us to have a “priesthood” aspect to our profession, but it’s based on questionable analogies.

We are not like doctors and lawyers, who receive quasi-governmental powers in exchange for a difficult certification process, a code of conduct and possibly governmental or quasi-governmental oversight (which, by the way, does not always eliminate abuses of the aforementioned powers and sometimes serves to cover them up). We are not like academics, either, where a “priesthood” is necessary to protect people who invest time and resources into a career path which in many cases has limited commercial value (For example, how many of us would pay big bucks to hire someone to read Sanskrit or analyze Tolstoy or survey the spectral emissions of a distant galaxy?).

Clearly, we have some important issues to address in software development, including:

  • technical debt
  • ethical implications of the products we develop (the potential for fraud or invasion of privacy, for example)
  • risk of project failure
  • handling and security of sensitive data
  • the rapid and continuous proliferation of new tools and approaches (and the need for training/education that this creates)
  • diversity issues (notably gender, race, and age)
  • repetitive strain injuries

Uncle Bob’s oath attempts to address a few of these issues, putting much of the responsibility on the individual developer, who is assumed to receive in exchange for this responsibility the additional autonomy and the time to do his or her job as he or she sees fit. That sounds nice, but software development is often a collaborative team effort, often serving a goal which is not chosen by the developers themselves, so responsibility should be shared.

I think a lot of companies could gain a competitive advantage by treating programmers as respected, autonomous professionals and to provide them with training or time to self-train (and some of them actually do it). However, I can’t support an oath which assumes that all employers do that to the extent assumed by Uncle Bob.

Here’s my humble (rough draft) revision (my additions in italic) of Uncle Bob’s oath, with commentary in blue:

I Promise that, to the best of my ability and judgement:

  1. I will not produce harmful code which is intended to defraud or to harm innocent victims.(Uncle Bob’s version would eliminate from the profession anyone programming weapons systems, for example. My version is more permissive but still covers cases like Volkswagen and Bernie Madoff )
  2. The code that I produce will always be my best work. I will not knowingly release code that is defective either in behavior or structure. conceal defects in the code’s behavior or structure from my employer or customer.(Living with defective software is what made certain software companies what they are today. Far be it from me to say they made the wrong choice, but an ethical developer will be transparent about this choice. Many developers have loyalty clauses in their contracts which would prevent them from announcing a defect publicly, but it’s always possible to inform the product owner.)
  3. I will produce, with each release, a quick, sure, and repeatable proof use the most appropriate techniques available to me, taking into account my employer’s or customer’s wishes, to ensure in repeatable fashion that every element of the code I produce or modify works as it should.(I think the employer/customer wishes and business needs can not be ignored just because there’s a “right way” to engineer, and a developer can’t take responsibility for the entire base of legacy code for a short-term add-on project.)
  4. I will make frequent, small, releases so that I do not impede the progress of others. There is no number 4. (This is micro-management by oath. If a company has a 6-month release cycle, I can encourage them to do 2-week sprints, but it’s not always my call, and they might not be ready to handle it.)
  5. I will fearlessly and relentlessly improve the code I work on at every opportunity. I will never make the code worse without a commitment to correct the worsening by an agreed-upon date.(This one is not bad, if you interpret it such that “every opportunity” gives some leeway to respect the wishes of an an employer who discourages certain code improvements. I added a fail-safe for cases where a short-term need requires a quick hack to be repaired later. I also added a specifier so that I’m not responsible for all the code in the world but only the code I touch. I removed the notion of fearlessness because I know of code that only a fool would change without fear.)
  6. I will do all that I can to keep the productivity of myself, and others, as high as possible. I will do nothing that decreases that productivity.(The first sentence is enough. The second is too inflexible.)
  7. I will continuously ensure share information to facilitate that others can cover for me, and be open to receiving information to facilitate that I can cover for them.(Staffing and how to cover for an employee’s absence are management’s call – a professional who is not a fully independent contractor should facilitate management in this effort but should not have responsibility for it. )
  8. I will produce estimates that are as honest as possible both in magnitude and precision. I will not make promises without certainty transparency about uncertainty.(Honesty and transparency are important for estimates, but no one can eliminate all uncertainty. Some slack should also be built into estimates to account for uncertainty and interruptions and tasks not directly related to the development itself.)
  9. I will never stop continue to learning and improveing my craft.(Let’s not judge the developer who chooses to attend their child’s piano recital instead of going to a coding dojo or a user’s group meeting. We have a long-term commitment to improve, subject to short-term interruptions.)
  10. I will keep informed about security issues and follow important security guidelines as appropriate in order to ensure the security of sensitive data.(This is my own addition. It could be covered under #2 and #9)
  11. I will not knowingly post a complete solution to what is obviously a computer science homework problem on StackOverflow or other public forums.(A recent pet peeve of mine. It’s good to help people learn, but helping someone cheat while spoiling a good homework question for posterity seems like a breach of ethics.)

 

 

Advertisements

Observations about “Clean” / “Hexagonal” architecture

Introduction:

After seeing Cyril Martraire’s online talk (in French) on Parleys about “Hexagonal” architecture and having read Robert C. Martin’s take on the “Clean Architecture”, I wanted to try it out for myself. It sounded great in theory, but reading a high-level description of an approach doesn’t teach as much as being confronted by numerous low-level implementation choices that result from the choice of high-level approach. I set up a little (but growing) personal project, and got going.

Approach:

Following Cyril Martraire’s simplified approach, packages are either domain or infrastructure. The dependencies go only from infrastructure to domain.

The domain contains client-facing service objects (ex: AccountsService) and interfaces for low-level (but domain-focused) services used by them (ex: IAccountsStorage, ISecurity, IEventListener, IEventPublisherClient). It also contains value objects (ex: UserAccount).

Infrastructure contains everything else (ex: concrete implementations of the low-level domain-focused services, HTTP front ends, command-line controllers). My little project, in Java, relied on PostgreSQL for persistent storage, Redis for short-term session storage, and JBoss Wildfly for websocket and REST endpoints, a JMS queue, etc.

hexArch

 

Useful Patterns:

 – CRUD happens with a call to a service object (ex: createUserAccount() in an AccountsService object), which then relies on the infrastructure implementation of the service’s storage interface (ex: PostgresAccountsStorage which implements IAccountsStorage) and other low-level interfaces (ex: BCryptSecurity implements ISecurity).

To unit-test the service, it’s best to stub and mock the low-level interfaces and verify validation of inputs, error handling and correctness of the call to the storage interface (for example, was the password which was passed to IAccountsStorage.createUserAccount() transformed by ISecurity? Note that you must not test the real encryption here because it’s a domain-level test – use stubbing, such as Mockito.when()).

In the infrastructure, it’s useful to refactor the storage tests to put all tests for the interface in an abstract class (in the domain, not in infrastructure), and then the concrete implementation of the tests just implements setUp() to provide the class under test. For example PostgresAccountsStorageTest just provides a real Postgres connection in setUp() – if I decide later to switch to MongoDB, I just implement a new setUp() with a real MongoDB connection in MongoAccountsStorageTest and I have all the failing tests already written in the superclass, AbstractAccountsStorageTest. Note that the tests in the AbstractXXXStorageTest classes are the only ones which actually access a real database.

HexUML

Addendum: The pattern I describe for CRUD here seems to fit nicely with J.B. Rainsberger’s notion of contract tests and collaboration tests. The service tests are what he calls collaboration tests (where services are mocked) and then the tests which actually touch the database are what he calls contract tests, because they prove that the service really does what is asked. I like this terminology better than “unit” and “integration” tests, because the names reflect their function in verifying the implementation rather than just a notion without nuance of their scope. 

 – I also found it useful to create a session controller class in the infrastructure to act as an intermediary between the server endpoints (websocket, REST, etc.) and the domain service classes (more on this later…).

Difficulties and limitations:

  • Logging – Is it a domain element or an infrastructure element? Risk of a multiplicity of logging solutions because some infrastructure elements will have their own logging dependencies (ex: log4j). I chose to have a domain-level logging interface implemented using slf4j-logback in the infrastructure. This may be over-engineering, since slf4j is already a facade.
  • Extracting a low-level component – Tricky to try and pull out an infrastructure unit developed to make it a general-purpose stand-alone library because the infrastructure depends on the domain (including the domain-level logging interface – difficult to extract).
  • Domain classes can’t contain Java-EE annotations and the like. Auto-wiring CDI dependency injection can get a bit complicated and may require wrapper classes or somewhat complex Producer implementations. Java EE was not designed with a clean/hexagonal architecture in mind.
  • There are some gray areas for design. In particular, the session controller is responsible for calling domain services in a certain way and enforcing a certain flow of control. Flow of control seems like infrastructure to me, whereas the “certain way” of calling domain services (ex: the interpretation of a text command) seems like domain code. When in doubt, it’s infrastructure (the domain stays pure), but the need for some refactoring could emerge from this ambiguity. I will probably end up refactoring out infrastructure dependencies to put the control code in the second of 3 concentric circles (dependent on the domain but not the infrastructure) – which is closer to what Robert C. Martin proposed (though his diagram would put my session controller in the same circle with my PostgreSQL storage implementations, for example). This additional “control” circle seems to correspond to the ports layer described in this InfoQ article.
  • Finally, it’s impossible to eliminate absolutely all infrastructure dependencies from the domain code, because even the choice of language has certain infrastructure implications. If you code in Java, the JVM is part of your infrastructure, and you can’t use Rails, Django, Node.js, Xamarin or Unity, for example. I suppose you could write the domain code in Ruby or Javascript, which are JVM-compatible languages that also work on other types of infrastructure.

Advantages:

  • Easy to swap out or offer alternative infrastructure elements (ex : change from Postgres to Mongo) without modifying the domain code at all and possibly without the need to write new tests.
  • Domain code is easier to write, read, update and test because it’s unpolluted by infrastructure. Coding the infrastructure is often easier, too, because the interface with the domain is clear and clean.
  • Can delay decisions about infrastructure and presentation, making progress on the domain logic.
  • Easier to port domain code to other projects and/or infrastructures.
  • Infrastructure development is driven by the domain use cases (YAGNI is easier to enforce)
  • Facilitates IoC and DI, which makes the code easier to test, which makes the code easier to refactor, which makes the code cleaner, which makes changes easier to implement.

Disadvantages:

  • Can’t lean heavily on a framework (ex: CDI, ORM, MVC, AOP, JMS/MDB), especially in the domain (where no framework dependencies are allowed). Leaning on frameworks can be useful for enforcing canonicality (having things defined in only one place – for example: strictly speaking, none of your domain classes should be generated by or have dependencies on an ORM – not even JPA annotations – so in a strict clean architecture approach you can’t use an ORM to generate your domain classes from database tables or vice-versa).
  • Can’t easily optimize the domain code for specific infrastructures.
  • Initial progress can be slow (as is the case with TDD in general).
  • Domain logic changes can ripple out into multiple infrastructure adapter implementations.
  • Complexity and redundancy in TDD, because use cases often need to be unit-tested and developed in multiple layers (ex: servlet, controller, service, and storage). If the end-to-end testing cycle is slow (heavy app server restart, WAR/EAR redeployment, etc.), forgetting to add a test at one of these levels can cost a lot of time.

Some thoughts about microservices:

The hexagonal/clean architecture is not the same as a microservices architecture, but the two are not completely incompatible. In a hexagonal/clean architecture, the goal is to separate the domain from the infrastructure. In a microservices architecture, the goal is to split the domain into small manageable pieces which communicate and inter-operate via a messaging infrastructure (REST, SOAP, JMS, ESB, whatever). According to Martin Fowler, you can’t start a development project using microservices. The domain has to be sufficiently complex (and useful/lucrative) to justify the effort and heavy tooling involved in managing a microservices approach, and the domain split has to be clearly defined. So a hexagonal/clean architecture might be the right place to start before evolving toward microservices. Because in the hexagonal/clean approach the domain is cleanly separated from infrastructure, splitting up the domain should be easier. Each microservice will start with a clean piece of the domain with interfaces to adapt to whatever infrastructure is needed for the specific microservice. It may even be possible to port much of the existing infrastructure code to the microservice, or to replace it with lighter infrastructure.

Conclusion:

The hexagonal/clean architecture makes it easy to have and maintain quality code in the domain layer. It makes it easier to port the domain code to different infrastructures. The model/CRUD side of infrastructure is clearly and cleanly driven by the needs of the domain. The view/presentation infrastructure relies on well-defined service API’s to interact with the domain. These clear divisions of responsibility remove some sources of accidental complexity.

The approach seems to be a good one for most kinds of projects. However, for very small projects, for throwaway prototypes and short-term projects, and for projects requiring painstaking optimization for a specific platform, it might not be the best choice, but for all others, it seems like a very good choice for developing well-crafted, maintainable software which can evolve as infrastructure needs and possibilities change.

Toward Mastery

I’m starting this blog to share my experience and ideas and to participate in the discussions which shape the profession of software development. As an engineering discipline, software engineering, which is only about 60-odd years old, is a relative newcomer when you think about the engineering that went into building the pyramids in Egypt, the Parthenon in Greece, etc. As a profession, we’re young, dumb, and full of… scrum.

The goal of this blog is to participate in a discussion which moves the profession toward maturity, and which helps you, the reader, and me as well, move from pedantry toward mastery.

What does that mean? In “How to Solve It”, mathematician George Polya described the difference between pedantry and mastery as follows:

Pedantry and Mastery are opposite attitudes to rules.

To apply a rule to the letter, rigidly, unquestioningly, in cases where it fits and in cases where it does not fit, is pedantry. Some pedants are poor fools; they never did understand the rule which they apply so conscientiously and so indiscriminately. Some pedants are quite successful; they understood the rule, at least in the beginning (before they became pedants), and chose a good one that fits in many cases and fails only occasionally.

To apply a rule with natural ease, with judgment, noticing the cases where it fits, and without ever letting the words of the rule obscure the purpose of the action or the opportunities of the situation, is mastery.

This idea of pedantry, even if you never heard the word before, should seem familiar to most software developers. We are told many rules and things that we should or should not do, until we find that we’re “should-ing all over ourselves”. Technology changes quickly, so we can’t always wait until we’ve mastered everything, but mastery is the goal, and mastery allows us to do our best work.