What is a Painless Pipeline?

A painless pipeline does not create any new pains. It delivers the promised benefits of CD pipelines without making additional cognitive load for the developers on the team.

Typical Pipeline Pains

Some of the common pains I have seen in pipelines are:

• Hard to debug: “Why did it fail?”
• Hard to make changes: you have to keep pushing fixes until it works
• Slow feedback: wait minutes or more to see if a config change worked

Often, these pains are created by situations where pipelines are developed either by focusing on only the happy path (with a “just get it done” mindset) or were created by different people (e.g. DevOps engineers, contractors, consultants).

Modes of Interaction

There are three different ways a software team interacts with a pipeline. These are:

• When it is green (all jobs have successfully passed)
• When it is red (a job has failed)
• When a pipeline change is required (e.g. adding a new job)

Let’s consider each in turn.

When it is green

When a pipeline is green, there is generally little interaction with it. It is usually a sign that everything is well and the team can continue their work.

The only problem that might occur with a green pipeline is when something is actually broken; a pipeline that tells you everything is okay when a failure has happened behind the scenes is creating pain, so your pipelines should always aim to fail if there is a problem.

When it is red

When a pipeline is red, then some action needs to be taken. If the pipeline clearly points to the problem and how to fix it, then it is helping us. However, if the failure sends us off on a journey of searching and confusion, it creates more pain.

There are different types of failure that can cause your pipelines to turn red. These include:

1. Genuine failures
2. Environment failures
3. Transient failures

In the case of a genuine failure, it helps if it clearly tells us what went wrong. The same applies to environmental failures; however, this may need to be supported with additional tooling outside of the pipeline (dashboards, alerts, etc.). Finally, we want to try and stamp out transient failures altogether; if we have any jobs that “just fail sometimes”, they need to be treated as broken and fixed.

When pipeline changes are being made

The final interaction we have with our pipelines is updating them — when we need to add to them, change them, or fix them. If making these changes is unpredictable and frustrating, it is another pain source. An example is when you need to make a change but don’t know if it has worked until you have committed it, pushed it, and waited for the pipeline to run it — only for it to fail with a trivial mistake.

For pipeline changes to be painless, the config should be easy to work with and quick to test.

Making Pipelines Painless

To make your pipelines painless, I have come up with four principles, and four practices to help you realise those principles. How you realise these might vary depending on your tools, but I hope the ideas apply to them all.

The Principles

• Treat your pipeline as a software product
• Design your pipelines to be easy to use
• Failure is a feature
• The developers are also the users

The Practices

• Be defensive
• Be atomic
• Be expressive
• Be reusable

Now, let’s look at each of these in turn.

The Principles

The principles are a mindset. They don’t aim to tell you how to do anything, but they guide the decisions you make.

Treat your pipeline as a software product

Too often, pipelines are developed with less care than the thing they are deploying. They are cobbled together until they work, then forgotten about until they break or need updating. There are a few reasons for this; for example, it could be that the application developers do not have expertise with these tools, or it could be that there is pressure to deliver the user-facing features faster. Often, a “DevOps” engineer or team is brought in to create them, but these creates silos and reduce team ownership.

This principle is about treating your pipeline projects like your software projects. Design them well, refactor them, test them, and ensure that they are implemented to a standard.

Design your pipelines to be easy to use

The people who interact with the pipeline (primarily the development team) are the pipeline users. Make the pipeline easy and informative to use; make it tell you what has failed, what is wrong, and how to fix it.

Failure is a feature

Pipelines often have been built with a focus on the happy path. They work well when green but don’t provide a great user experience when they fail. However, failure is an important feature of a pipeline; it is the one we interact with the most so we need it to help us. Make failures helpful and informative.

The developers are also the users

One unique thing about pipelines is that the developers who build them are likely also the users. That means that changing and developing them is part of the users' experience. Therefore, we should treat the ability to maintain the pipeline as one of its features. Making changes should be easy, predictable, and as safe as possible.

The Practices

The principles above provide an idea of how we should treat our pipelines. The following practices aim to turn them into concrete actions.

Be Defensive

When the pipeline performs a task, it should explicitly check for any problems and report clear errors if anything is incorrect.

This practice helps us when we want to make changes to the pipeline by quickly informing us what went wrong. A pipeline should give a descriptive error when the task fails. Here are some examples of why a pipeline might fail:

• An error in an artifact
  e.g. the code is broken
• An environment error
  e.g. the deployment environment is broken
• A pipeline config/scripting error
  e.g. you forgot to provide a required environment variable

When the pipeline fails for one of these reasons, the error should help you resolve the issue. It should do one of the following:

• Describe the problem
  e.g. “The required AWS_ACCESS_KEY environment variable is missing.”
• Redirect you to a place where you can get more information
  e.g. “The production environment is not reachable; please check http://monitoring-dashboard.com”

Example:

In this example, we have a bash script as part of the pipeline job which substitutes some values into a Google AppEngine app.yaml.

#!/usr/bin/env bash

set -euo pipefail
# …
sed "s/#DB_HOST#/${DB_HOST}/g" app.yaml.tpl | \
 sed "s/#DB_USER#/${DB_USER}/g" | \
 sed "s/#DB_PASSWORD#/${DB_PASSWORD}/g" | \
 sed "s/#DB_DATABASE#/${DB_DATABASE}/g" >app.yaml
# …

# gcloud command to deploy the app

When pushed, the pipeline reports the following error:

500: An unknown error occurred

Thanks Google!

Eventually, we discovered that the app.yaml.tpl file had an extra #PROJECT_ID# placeholder that had not been substituted. We could have saved time searching for this issue if the script had also checked for any placeholders that had not been replaced and reported an error like so:

Error:
app.yaml contains an unsubstituted template variable #PROJECT_ID#

Be Atomic

Each step should be easy to run in isolation when given a set of inputs and expected outputs.

• Each step should be a script, program, or configuration that can be run locally on a developer's machine.
• Each step should have stubs, mocks, or whatever is required to allow it to be run and verified locally (a “unit test”).
• All steps can be run as a part of a test suite.
• If a step cannot be run locally, it should be run as a stand-alone step in a separate test pipeline.

Example:

$ export DB_HOST=localhost
$ export DB_USER=appuser
$ export DB_PASSWORD=secretsecret
$ export DB_DATABASE=customers
$ ./substitute-templates — template app.yaml.tpl — output app.yaml — var-source environment

$ ./run-pipeline-unit-tests
Tag and release tests
✅ creating git tag succeeds
✅ creating git tag fails when tag already exists
✅ creating git tag fails when tag has incorrect format AppEngine Deploy tests
✅ substitute template vars in app.yaml
✅ substitute template vars in app.yaml fails when env var is missing
❌ substitute template vars in app.yaml fails when env var is not substituted

Be Expressive

Use languages and tools that make coding and error reporting easy and clear.

• Use an appropriate programming language.
  i.e. don’t resort to bash or PowerShell because it’s quick.
• Create/use tools that can be reused.
  e.g. a templating tool that performs validation.
• Practice Test-Driven Development (TDD).

Example:

step:
 name: prepare app.yaml
 type: shell
 command: |
 #!/usr/bin/env bash
   set -euo pipefail# Substitute env vars
   sed "s/#DB_HOST#/${DB_HOST}/g" app.yaml.tpl | \
     sed "s/#DB_USER#/${DB_USER}/g" | \
     sed "s/#DB_PASSWORD#/${DB_PASSWORD}/g" | \
     sed "s/#DB_DATABASE#/${DB_DATABASE}/g" >app.yaml
   # Check for unsubstituted env vars
   grep -i '#[A-Z]*#' app.yaml

step:
 name: prepare app.yaml
 type: shell
 command: python substitute-templates.py — template app.yaml.tpl — output app.yaml — var-source environment

Be Reusable

Invest in building tools you can use again.

• If you are investing more time in building tools right, build them to be reused.
• Reuse your tools in different steps, pipelines and projects.
• Make them available to others (via open source if possible).

Example:

step:
 name: prepare app.yaml
 type: substitute-templates
 options:
 template-file: app.yaml.tpl
 output: app.yaml
 var-source: environment

Conclusion

Your pipelines are a critical part of your development process, therefore, treat them as you would treat your software projects. They should not be a string of minimum-viable-scripts, but rather tools that support the people interacting with them.

The things I have presented in this article are not the complete picture, there are other important things in pipelines, such as ensuring that the sample artifacts are used throughout, making sure pipelines don’t take too long, and ensuring that the jobs are idempotent. However, the practices and principles I have presented should be added on top of these common practices to enhance the user experience.

Also, pipelines alone do not solve all the problems. Having good observability (including logs, dashboards, alerts, and traces where appropriate), will also contribute to the painless experience.

Introducing Painless Pipelines was originally published in Cloudnative.ly on Medium, where people are continuing the conversation by highlighting and responding to this story.

Basic Application

Before introducing the concepts, let’s consider a simple scenario to extend with these patterns. A user interacts with a basic application, which stores and retrieves data from a database. Admittedly, most applications are significantly more complicated than this in the modern age, but this is a familiar scenario and the perfect starting point. We program the business logic in the application and use the database to store the current state of the entities in the system.

Event-Driven Architecture

With Event-Driven Architectures, we use events to communicate change between system parts. When an event occurs in some part of the system, we publish it so the other components can choose to react. This pattern can be used within a monolithic application using the observer design pattern; however, in modern systems, this is often distributed between in-dependent services using message queues.

Event-Driven Architectures have to embrace asynchronicity and eventual consistency.

Event Sourcing

Event Sourcing is a persistence pattern. Instead of having a database that stores the current state of the entities, we have an immutable event log that stores the history of all the changes made. The event log is the single source of truth. The current state is obtained by replaying the events in the application.

The event log provides an audit trail and the ability to construct the state at any point in history.

For more details on Event Sourcing, see my previous article, What is Event Sourcing.

Using Event Sourcing with Event-Driven Architectures

Event-Driven Architectures and Event Sourcing are two different things; the former is a communication pattern while the latter is a persistent state pattern. The only thing these two things have in common is events. However, the fact that they are both centred around events makes them both appealing to people building systems based on events.

However, there is an important question…

Are event-sourced events and event-driven events the same?

The default answer here should be “No”; here’s why:

Considering the pure concept of Event Sourcing, your event log is your database; the only service that reads your database is the event-sourced service. Its implementation, structure, and existence are purely an implementation of your service. On the other hand, the events published by a service are part of its public interface; consuming services are coupled to the events and their structure. When you expose the event-sourced events, you make them part of your public interface. In addition, you may only want to publish some of your events or some part of their content; you want to provide a limited subset. Conversely, you may also want to publish additional events that are not part of the state.

The non-default answer is “It depends”

Sometimes, it is too convenient to publish the event-sourced events to other services; you might find that when keeping them separate, you have to create two event datatypes for every new event. When this is the case, there is probably a tight relationship between the services publishing the events and those consuming them. They are likely local components of the same service, developed within the same team. If they are not, then this decision should be made very cautiously.

Command-Query Responsibility Segregation Architecture (CQRS)

One pattern that is bound to come up when talking about distributed and event-sourced systems is Command-Query Responsibility Segregation (CQRS). CQRS is a detailed pattern, although I’m skipping a lot of that detail in this example; I’m simply separating the write side (where commands are sent) of the model from the read side (where queries are made). It also does not need to be used with events (although they are an excellent fit). This pattern optimises the read model for high-performance reads at the cost of eventual consistency. It is excellent for systems with a high ratio of reads to writes. For example, a user may make a lot of requests while browsing products on an online store and then make just one purchase request at the end. CQRS is not a good fit if you have a 1:1 reading and writing ratio; it has probably only been added to mask slow performance on the write side by acting as a read cache.

For this example, I have used an event-sourced write service and an event-driven read service. This whole thing could be one service in a larger system. A single team might be building it as a single service as one business component in the system. In this case, the read database could simply be a viewable projection of the event log; the events that drive the read service are the events in the event log.

However, this model could also be distributed further. The writing service is one application built by one team, and the read service is a consumer service created by another team. In this case, it makes sense for the events to be different, curated ones.

Writes might not require state

One of the key points in the previous example is that the write service fetches the state from the event log. This is very important in a system where state changes depend on the current state. However, you may have a system where the commands generate events without referencing the current state. In these cases, the write service never reads back from the event log. The write service generates and publishes the event, and then the read service consumes it. From that aspect, the read service is event-driven. However, the event log is still the single source of truth, so we could also argue that it is event-sourced.

This pattern is more straightforward with a tool like Kafka as the event queue, because it can also persist the streams and behave as the log.

Summary

Event-Driven Architectures are a communication pattern, whereas Event Sourcing is a persistent state pattern. When looked at independently, they are two very different things, but combining them in different ways can blur the distinctions. Also, both patterns bring advantages, but at the cost of being more complicated. Finally, the locality of the services that consume your events affects which events you should expose.

My final suggestions are:

• Make sure you understand each pattern in isolation before combining them.
• Consider carefully if they are suitable patterns for your problem.
• Only expose your event-sourced events if the consuming service is very local. Otherwise, publish specific events for external consumption.

Event-Driven Architectures (EDAs) vs Event Sourcing was originally published in Cloudnative.ly on Medium, where people are continuing the conversation by highlighting and responding to this story.

Suppose you’re coming to a programming language which allows you to program in multiple paradigms, like Python or TypeScript, but you have come from an object-oriented programming (OOP) background. In that case, you might choose to continue building your software in the OOP way you love. Writing code this way is perfectly reasonable, but you might create unnecessary boilerplate code if you insist on using the class keyword for everything. I want to show how you can rewrite a common type of class differently, reducing the amount of typing you have to do and making the code more straightforward to understand.

Single Method Classes

The type of class I’m talking about is a single-method class. These classes represent a single action, but they might wrap up some state or dependencies which the method will use when invoked.

Let’s look at some examples:

Simple Action

The simplest form is a function with no dependencies wrapped in a class. The only reason these classes exist is that the language does not support functions outside of classes. Java introduced the Runnable interface for this exact reason.

class Adder:
    def calculate(self, a: int, b: int) -> int:
        return a + b

Here we can move the class out of the function altogether.

def add(a: int, b: int) -> int:
    return a + b

You might think you have lost the ability to define an interface that your Adder class implemented so that you supply alternative implementations to consumers — for example, a Calculator interface. You may not be able to create an interface (or a Protocol in the Python world) when you only have a function; however, you can still define an abstraction by creating an alias to a callable type.

Calculator = Callable[[int, int], int]

Dependency Injection

Another example of a class with a single method is one where we use constructor injection to provide dependencies for the method. This enables us to pass only the instance to where we need it without having to pass the dependencies too. Here’s an example of such a class.

class OrderCreator:
    def __init__(self, order_repository: OrderRepository, email_sender: EmailSender):
        self._order_repository = order_repository,
        self._email_sender = email_sender

    def create_order(self, customer: Customer, items: list[Items]) -> None:
        order_id = OrderID.generate()
        order = Order(order_id, items)
        self._order_repository.save(order)
        confirmation_email = Email(to=customer, subject=f"Thank you for your order ${order_id}")
        self._email_sender.send(confirmation_email)

We would then use it like this:

order_creator = OrderCreator(some_order_repository, some_email_sender)

# at another code location
order_creator.create_order(logged_in_customer, cart.get_items())

We can create this pattern by creating a constructor function which returns a new closure function with the dependencies baked in. We can again use a type alias to define a meaningful named type.

CreateOrder = Callable[[Customer, list[Items]], None]

def new_order_creator(order_repository: OrderRepository, email_sender: EmailSender) -> CreateOrder:
    def create_order(customer: Customer, items: list[Items]) -> None:
        order_id = OrderID.generate()
        order = Order(order_id, items)
        order_repository.save(order)
        confirmation_email = Email(to=customer, subject=f"We have recieved your order {order_id}")
        email_sender.send(confirmation_email)
    return create_order

We would then use it like this:

create_order = new_order_creator(some_order_repository, some_email_sender)

# at another code location
create_order(logged_in_customer, cart.get_items())

Deferred Dependencies

This pattern is the same as the previous example but used the other way around. In this case, we inject the values in the constructor but provide some dependencies when we invoke the method. Let’s see an example:

class Email:
    def __init__(self, to: str, subject: str, body: str):
        self._to = to
        self._subject = subject
        self._body = body

    def send(self, sender: EmailSender) -> None:
        sender.send(self._to, self._subject, self._body)

We would then use it like this:

email = Email(
      to=customer.email_address,
      subject=f"We have received your order {order_id}",
      body=f"""
      Dear {customer.name}
      Thank you for your order!
      """
  )

# at another code location
email.send(some_email_sender)

Because the pattern is the same as the previous one, the classless version follows the same pattern:

SendEmail = Callable[[EmailSender], None]

def create_email(to: str, subject: str, body: str) -> SendEmail:
    def send(sender: EmailSender) -> None:
        sender.send(to, subject, body)
    return send

We would then use it like this:

send_email = create_email(
      to=customer.email_address,
      subject=f"We have received your order {order_id}",
      body=f"""
      Dear {customer.name},
      Thank you for your order!
      """
  )

# at another code location
send_email(some_email_sender)

Summary

If you create classes with a single method, you might be better off just creating a function. Using functions might seem like you are not doing OOP, but the model remains the same; an instance wraps a behaviour with a single way to invoke it. Also, you can use type aliases to give these functions named abstractions, which you can use for type signature, descriptiveness and communication. The command classes can be necessary for object-oriented languages because there is no better way to implement them.

Better Ways to Implement Single-Method Classes was originally published in Cloudnative.ly on Medium, where people are continuing the conversation by highlighting and responding to this story.

Five years ago I wrote the article Designing Microservice Architectures for People. Today I want to reflect on the content and see how it has aged. In the previous article, I put the emphasis on microservices. I also presented three ideas that would bring the focus to aligning your architecture with your organisation structure: The Single Responsibility Principle, Bounded Contexts and Conway’s Law. Let’s reflect on the ideas I presented.

The Focus on Microservices

In the previous article, I stated that microservice architectures were everywhere at the time. I don’t think that has changed, but we’ve stopped talking about them so much. Cloud platforms and serverless offerings seem to have detracted from the term microservices, even though the systems built using these technologies are often still microservices architectures, just implemented with different tools.

Recently, we’ve also seen talk from prominent companies about them migrating from distributed systems back to monoliths. As a result, monoliths are getting a little bit of a buzz again. I’m sure we’re not going to move to an era when everything is a monolith by default, but I do think we’re entering an era where the stigma of monoliths has gone, and they will more commonly be considered a viable option for parts of the systems we build.

In the previous article, I focussed on microservices because I was trying to address a problem which I was finding widespread — teams building “microservices” because of the hype, but ending up creating tightly-coupled architectures; or what we affectionately call distributed monoliths. However, everything I covered still applies in these other types of systems; in fact, most of the thinking around the ideas I presented pre-exists microservices altogether. From this perspective, I think that what I presented would still serve well with the term microservices wholly removed.

The Three Ideas

There is very little to say about the three ideas I presented. Their usefulness and importance today are no different from how it was five years ago. They should be applied when building software systems; whether monolithic, cloud-based or microservices.

That said, there has been some interesting additional thinking that has come out since my previous article…

Team Topologies

In 2019, Matthew Skelton and Manuel Pais published their book Team Topologies. Like my original article, Team Topologies emphasises a team-first approach to software architecture. This idea is not new; for example, Michael Nygard’s book Release It! was published in 2007 and says “Team assignments are the first draft of the architecture.” — Matthew and Manuel reference this quote in their book.

Your early decisions make the biggest impact on the eventual shape of your system. The earliest decisions you make can be the hardest ones to reverse later. These early decisions about the system boundary and decomposition into subsystems get crystallized into the team structure, funding allocation, program management structure, and even time-sheet codes. Team assignments are the first draft of the architecture. (See ​Conway’s Law​.)

Release It!
Michael Nygard 2007

However, Team Topologies explicitly calls out Fast Flow as the output you want from your optimised team and software architectures. The book is based on the four team types it presents (stream-aligned, platform, enabling and complicated subsystem). It suggests Conway's Law as one of the driving forces in designing teams, as does my original article and Michael’s book from back in 2007.

This set of four team types (and the interaction modes that go with them) is an incredibly useful tool for designing people-centric software. Team Topologies is a must-have when designing software architectures for people. However, the most interesting piece for me was that it brought the idea of Cognitive Load into the equation.

The Team Topologies book shows how we don’t want to just line software boundaries up with teams, but also we want to ensure that teams are designed and sized so that the cognitive load required to do work is comfortable. By managing the cognitive load in the team, not only does work flow through the value stream query, and individuals and the team also enter flow states more easily. Woody Zuill refers to the combination of these three types of flow as Flow++.

The rise of “Socio-technical”

This idea that building software is less dominated by the technical and heavily informed by the social is becoming more widespread. As a result, the term Socio-technical is becoming more prevalent.

Socio-technical theory has at its core the idea that the design and performance of any organisational system can only be understood and improved if both ‘social’ and ‘technical’ aspects are brought together and treated as interdependent parts of a complex system.

Source: https://business.leeds.ac.uk/research-stc/doc/socio-technical-systems-theory

The term is not new; it was coined in the World War II era, but it has only recently become more commonplace when discussing software. If we see the software architecture and the social constructs in the organisation as two parts of the same system, then work will flow more naturally.

As this thinking evolves, some exciting ideas are being developed; for example, Nick Tune has published some socio-technical architecture patterns and an upcoming book on the topic.

Conclusion

The ideas behind improving the ways that large organisations are building software are still evolving and improving. What I have taken away from this reflection, is that tools, patterns, architectures and methodologies come and go, but solid underlying principles persist. An idea Melvin Conway came up with in 1968 is more relevant now than it was then. How systems are implemented over the next decade will change (I’m sure Artificial Intelligence is going to have a huge impact), but the interplay between humans and technology will remain paramount. Ideas like Conway’s Law, The Single Responsibility Principle, Bounded Contexts, building stream-aligned teams, and optimising for flow and low cognitive load will outlive any technology or architectural pattern.

Designing Software Architectures For People in 2023 was originally published in Cloudnative.ly on Medium, where people are continuing the conversation by highlighting and responding to this story.

I’m a firm believer that Test-Driven Development (TDD) should be the default way that software teams create code. I believe that its methodical approach leads developers to design and discover better ways to architect their software. However, I also believe that the best tool that we have today is not guaranteed to be the best tool for tomorrow. In this article, I present some of the things that I’ve explored which could perhaps replace TDD (and maybe even tests in general) as the best way to build and test software. This article does not seek to provide any answers, but rather, present some exciting things to play with.

“Technologies, methodologies, and practices are constantly emerging and evolving. There is always a better way of doing things. Although we described a set of practices in this chapter, that neither means they are the only practices out there, nor that they are suitable in every single context. Things that are considered good today may not be considered as good tomorrow. Maybe in a few years time, Software Craftsmanship will be strongly encouraging the adoption of a completely different set of practices”

Software Craftsmanship by Sandro Mancuso

About TDD

What is it?

Test-Driven Development is a precise process. It is the quick iteration of the red-green-refactor cycle. In this cycle, we think about what we need to achieve, write a single failing test (red), write just enough code to make that test pass (green), and then refactor to improve the design. Each iteration of this cycle takes only a few minutes (or less).

There are many benefits to TDD:

• It gives the developer a rigorous process which makes it easy to know what to do next.
• It breaks the problem down into much smaller pieces; this reduces cognitive load and allows the practitioner to maintain the state of flow.
• It encourages decoupled and cohesive, modular design in your code.
• It results in near 100% code coverage without even making that the focus (it should be 100%, but that’s dependant on the developers not accidentally missing a step).

What is its current state?

The good news is that tests are much more common now than they used to be. Many projects have at least a few tests, and many developers are familiar with testing frameworks. Also, the words Test-Driven Development (and acronym TDD) feel like they are mainstream now, you can find them on many CVs and job descriptions. However, my experience tells me something a bit different about the state of TDD — having done countless interviews of people with TDD on their CV, I can count on one hand the number of people who I’ve seen do TDD by-the-book in an interview. What I see instead is people practising their own interpretation of the words “test-driven development”; this can range from processes very similar to TDD (perhaps writing a suite of tests first) to merely having some form of tests present at the end of the process.

My experience suggests that, while the term Test-Driven Development is well known, the practice still has a long way to go until it becomes mainstream. Whether TDD will ever become entirely ubiquitous for the industry or not, I do not know. However, I still think we should strive for this as a goal. TDD is, without a doubt in my mind, the most reliable way that we currently know of for writing software with the languages, tooling, processes and teams that are most common in the industry. I am, however, also a firm believer that this will not always be true!

When is TDD Appropriate?

I think it’s also worth pointing out that TDD isn’t a tool appropriate for every single programming task; there are times when it is not the best tool for the job. Experienced TDD practitioners have explored TDD to a level where they know it’s limitations. They will know when they need to either change the process or switch processes altogether — however, this will generally be for a small part of the codebase rather than the whole project.

For general programming, TDD is usually applicable in my experience. When someone tells me that a piece of code is not able to be written using TDD, then nine times out of ten, I’ll disagree — often it requires a small change in mindset or approach. That one time I will agree though, it will be completely valid, and TDD is not the right tool.

I think it’s also worth pointing out that there are niche alternatives to TDD. These might exist in specific programming paradigms, domains or communities. These alternatives are very valid in their context; they are also some of the sources that this article draws from. However, in business software development, I stand by my statement that TDD is currently the best tool for the job.

Now we know where we are with TDD, I want to explore some different ideas…

Evolutions of TDD

The most obvious processes that could succeed TDD are the direct descendants of it. Many of these are TDD with some restriction applied (e.g. time limitations within the cycle) or some adjustments/additions to the process (e.g. revert changes when the tests fail).

One of the most talked-about evolutions is Kent Beck’s test && commit || revert (or TCR). With this process, every time you run the tests, the changes are committed if the tests pass, or reverted if the tests fail. A typical first reaction to trying this is that it’s infuriating, but as you proceed, you see that there are lessons to be learned.

Personally, I’m not sold on this being a better process than TDD for the experienced practitioner, but I do see it as a fantastic training tool. The reason I believe that it is so valuable as a learning tool is that it forces the person using it to discover their own ways to take smaller steps — this self-discovery I find much more valuable and rewarding than just following instructions from a book or teacher.

Rachel M. Carmena has created some great diagrams of some of the different flows people have developed. You can find them in the article Test-driven programming workflows.

While all these altered workflows are interesting, I still see them as striving towards the same goal. I thoroughly recommend experimenting with these different workflows and seeing what you learn (you definitely will learn something). Still, I see them as the evolution of TDD rather than moving beyond TDD.

REPLs

Many languages now come with a REPL (a “read, evaluate, print loop” or merely an interactive terminal). Python, Ruby and Scala, for example, all have tools which allow you to run commands and evaluate expressions interactively. The developers in communities from some languages, such as Clojure and F#, commonly use interacting with the REPL as their primary development process; that is, they iterate the development of functions by interactive tweaking and trying until they reach the desired implementation. You can call this REPL-driven development.

An evolution of REPL usage is where IDEs have implemented features where you can evaluate lines and expression inline in the code editor. When it is as convenient as hitting something like Alt+Enter to get the result of evaluating the expression on the current line, you are getting some of the fastest feedback possible.

Another variation on this, which is often found in the academic, science and data-science domains, is the use of Jupyter Notebooks to create interactive and executable documentation. This documentation can even include things like generated graphs, tables and charts. Jupyter Notebooks combines the REPL like feedback in a Literate Programming context. Interactive Notebooks can provide a fantastic record of the process behind the development of the code.

The big problem I have with the use of the REPL as the primary iteration process is that you still have to write the tests. TDD gives a great way to know which tests to write, but with the REPL, you have to decide which tests to create after the fact. The caveat here is that this apparent limitation might be due to inexperience on my part.

Types

Let’s briefly talk about types. In languages where we can specify the types of values, we write fewer tests. For example, in Ruby we might write a test it "returns an integer", but in Java, we’d define public int f(). There is no need to write such as test in Java because we trust the type system (although tests around null are still very much needed in many languages).

One thing some functional programmers like to do in strongly typed languages is to start creating a model by defining the types. Once the types are specified, then the implementations can be written, along with tests, to complete the features. You can find a great example of this in Scott Wlaschin’s fantastic Domain Modelling Made Functional, which shows a practical way to apply this approach.

Some languages have even more powerful type systems that allow for type-level programming. As we’re now about to see, some languages have far more advanced type systems than our mainstream languages. These languages can reduce the amount of testing required to a much greater extent.

Proofs

Since learning Test-Driven Development (I won’t say TDD in this section for reasons which are about to become apparent), I’ve been convinced that you must always have tests — that is until I read Type-Driven Development with Idris.

Idris is a dependently-typed language; this means that you can be much more expressive and restrictive in the type signatures of functions. This expressiveness is taken to a level at which the types of parameters and return values of functions can change and depend based on each other. This may sound complicated (it kind of is) but let’s look at a simple example. The following is a type signature for a function that joins two vectors together:

join : (xs : Vect n elem) -> (ys : Vect m elem) -> Vect (n + m) elem

This type signature states that the function join does the following:

• It takes one vector (Vect) as the first parameter (named xs), of any length n, and containing elements of any element type elem.
• It takes another vector as the second parameter (named ys), of any length m, and containing elements of the same element type elem as in xs.
• It then returns a resulting vector containing elements of the same typeelem, and it has the length of exactlyn + m.

This example has taken one property of the algorithm to join two vectors together (i.e. “the length of the resulting vector must be equal to the sum of the lengths of the two inputs”) and specified that in the type. Now here’s the fun part, you don’t need to test this property of the algorithm because Idris’s compiler will not compile this code until you prove to it that this property will always hold for your implementation.

Obviously, in the example above, you still need tests as there are other properties of this algorithm that are not encoded in the type. These include:

• The resulting vector begins with the first vector
• The resulting vector ends with the second vector

It is possible with dependently typed languages to encode a lot more in the type. Some time ago, Johan Martinsson and I paired on a FizzBuzz implementation as an Idris kata; eventually, we came up with a solution, and the types looked like this:

IsFizz : (k : Nat) -> Type
IsFizz k = (modNatNZ k 3 SIsNotZ = 0)

IsBuzz : Nat -> Type
IsBuzz k = (modNatNZ k 5 SIsNotZ = 0)

data Fizzbuzz : (k: Nat) -> Type where
 Fizz : (k: Nat) -> IsFizz k -> Not (IsBuzz k) -> Fizzbuzz k
 Buzz : (k: Nat) -> Not (IsFizz k) -> IsBuzz k -> Fizzbuzz k
 FizzBuzz : (k: Nat) -> IsFizz k -> IsBuzz k -> Fizzbuzz k
 Normal : (k: Nat) -> Not (IsFizz k) -> Not (IsBuzz k) -> Fizzbuzz k

This then enabled the following implementation:

fizzbuzz : (k: Nat) -> Fizzbuzz k
fizzbuzz k =
  let isFizz = decEq (modNatNZ k 3 SIsNotZ) 0
      isBuzz = decEq (modNatNZ k 5 SIsNotZ) 0
  in case (isFizz, isBuzz) of
       (Yes isfizz, No notbuzz) => Fizz k isfizz notbuzz
       (No notfizz, Yes isbuzz) => Buzz k notfizz isbuzz
       (Yes isfizz, Yes isbuzz) => FizzBuzz k isfizz isbuzz
       (No notfizz, No notbuzz) => Normal k notfizz notbuzz

Without going into detail of how this code works, the interesting point is that it is impossible to implement a function that does anything different than the implementation above with the type signature fizzbuzz : (k : Nat) -> Fizzbuzz k. Therefore, there is no valid test that you can write for this function that would fail if the code compiled!

Here I’ve shown that it’s possible (but not always practical) to write programs with Idris where tests are not needed, but Test-Driven Development is not just about the tests — it’s the methodology and process. Idris’s Type-Driven Development has its own process; it’s methodical just like Test-Driven Development, and it makes heavy use of the advanced IDE integration. Not only can Idris’s IDE integration/REPL evaluate statements, but it can also generate code and suggestions from the types. With Type-Driven Development, you specify a type and then work with the compiler to define and refine the implementation.

Dependent types and Idris’s great tooling seem near to perfection (although there are some more challenging gymnastics you have to do to convince the compiler that things are true), so why is everyone not doing it? The answer is that it’s still pretty academic, at the time I was playing with it (version 2 has since been released), it wasn’t mature enough or performant enough to write real-world applications with. It also requires some pretty radical changes in thinking from traditional software development. While there is some support for dependant and type-level programming in other languages, it’s still pretty niche and unknown — maybe this will change in the future.

Property-Based Testing

In the last section, I talked about properties quite a bit. We’ve seen that dependently typed languages allow us to encode properties into types, but can we test properties in languages which don’t support dependant types? This is exactly what property-based testing does.

With TDD (I’ll revert to using TDD in place of Test-Driven Development from now on), we tend to use example-based tests. That is, we set up the environment, then execute the test with a specific set of inputs, and finally assert that we receive a specific result. Example-based tests are ideal for TDD because they are perfectly suited to small iterations. However, a single example rarely tells us much about the system under test on its own — instead you need a suite of examples to build up a model of what you think the system does (one of the skills of TDD is choosing the right examples).

With a properties, we don’t state that specific inputs give a specific result, rather we state that properties hold across sets of inputs and results.

• “when given 6 return ‘fizz’” is an example
• “when given an input divisible by 3 return a result containing ‘fizz’” describes a property

The problem with properties is that is is too expensive, impractical or even impossible to create tests which check every possible input. Instead, property-based testing frameworks use small random samples (say 100 or 1,000 sets of inputs).

Here’s an example of how you might test the above property might using Python’s Hypothesis framework.

@given(x=integers())
def test_input_divisible_by_3_returns_fizz(x):
    assert f(x * 3) == 'fizz'

Hypothesis will run this a specific number of times (e.g.100) with random integers provided for x. This means that when you implement f you will have to provide a solution which works for any integer value of x rather than just the specific one — otherwise, the test could fail.

Inspecting examples allow us to estimate the behaviour of an algorithm, whereas, with properties, we can specify the complete algorithm. A set of examples for Fizz Buzz might be:

• 1 == ‘1’
• 2 == ‘2’
• 4 == ‘4’
• 3 == ‘fizz’
• 6 == ‘fizz’
• 5 == ‘buzz’
• 6 == ‘buzz’
• 15 == ‘fizzbuzz’
• 30 == ‘fizzbuzz’

From these examples, we can deduce a good guess of the behaviour of the algorithm, but having these as tests would not stop the implementation from having an overriding case saying if x > 100: return 'flop'.

A set of properties for fizzbuzz might look like this:

• if input is divisible by 3, return a result containing ‘fizz’
• if input is divisible by 5, return a result containing ‘buzz’
• if input is not divisible by 3, return a result which does not contain ‘fizz’
• if input is not divisible by 5, return a result which does not contain ‘buzz’
• if input is not divisible by 3 or 5, return the input

Given this set of properties, there is no other implementation that holds true other than a correct one.

Property-based tests look much better than example-based tests, so why not always use them? When comparing property-based tests with example-based tests, property-based tests seem superior, but the problem with them is that the workflow is not as straightforward as when using example-based tests and TDD. For some tasks, doing TDD with property-based tests works perfectly (try the fizzbuzz example above), but for other tasks it’s not so obvious. You will often find that you end up either taking too big steps or using weak properties (rather than the ones which most characterise the algorithm) when trying to do TDD with property-based tests.

I find a combination of property-based and example-based tests works well. Sometimes it’s obvious when using a property makes sense, other times I’ll use example-based TDD and then refactor some of the example-based tests into property-based tests when I feel they are more appropriate.

Algebra-Driven Design

I recently discovered the book Algebra-Driven Designed by Sandy McGuire. What I found compelling about this approach is that it brings a methodology to using a collection of approaches (some of which I’ve mentioned already) together.

In the section about types, we talked about how some developers like to define the types as the first step when modelling in code. Sandy McGuire extends this a bit by creating the types and then determining the laws which must be true in the model (defining an algebra). In the book, Sandy builds a composable graphics library where a function called cw rotates an image tile 90 degrees clockwise. Its type signature takes a tile and returns a new tile:

cw :: Tile -> Tile

An example law for this is that “if you rotate the tile 4 times, then you get the original tile”. He represents this like so:

∀ (t :: Tile).
   cw (cw (cw (cw t))) = t

This is very much like defining a property which we talked about earlier.

The next step with Algebra-Driven Design is to implement a minimum viable version of the algebra (known as an initial encoding). This encoding implements all the functions in a way that obeys all the laws but isn’t necessarily useful.

Once this reference implementation is built, a tool called QuickSpec is employed to discover all the properties of the model! These properties can be converted to property-based tests, and you are then free to create the implementation you desire.

The process of Algebra-Driven Development feels very different from TDD. You can define your algebra and laws with pen and paper while discovering the domain. The act of implementing is then creating the appropriate code within the laws of that algebra.

In the minimal experiments I’ve tried with Algebra-Driven Design, I’ve noticed that I tend to hit a well designed and composable API first time (as opposed to multiple refactorings before I get there). I’ve yet to be convinced that this approach can be used in all aspects required to build large systems, but it is definitely a completely different approach to that of writing code with TDD. I’m pretty sure Algebra-Driven Development excels with certain types of problem (the examples in the book are very good).

Conclusion

This post has been quite a journey down the rabbit hole. It doesn’t contain any answers as to what will succeed TDD, but that was never my intent. In fact, I don’t believe that anything will succeed TDD; instead, we’ll discover new and exciting methodologies that can be used as complementary tools to support each other. Each of the things I’ve mentioned has their own strengths:

• TDD with examples excels in simplicity, methodology and test coverage.
• REPLs excel in the feedback cycle and discovery.
• Property-based testing excels in clarity, extended test coverage and edge-case detection.
• Type-level programming excels in correctness.

I don’t know how many of the things I’ve talked about will ever become mainstream programming; they are certainly not new (many of the ideas have been around more than a decade). I think that the reason they are becoming a bit more popular now is due to the rise in popularity of functional programming (where most of these ideas have come from). In the talk Polysemy: Chasing Performance in Free Monads, Sandy McGuire, says that the functional programming community is generally about 30 years ahead of the mainstream community — I have no idea if this is true, but perhaps some of these ideas will become mainstream one day.

In summary, learn TDD, practice TDD and teach TDD — I think we should all be trying to make TDD more common in our work. But also don’t be dogmatic — explore crazy ideas and alternatives and see what you learn or come up with. Finally, consider reading Domain Modelling Made Functional, Type-Driven Development with Idris or Algebra-Driven Design this holiday.

Beyond Test-Driven Development was originally published in Cloudnative.ly on Medium, where people are continuing the conversation by highlighting and responding to this story.

Execution Order is the lowest strength dynamic connascience. The characterisation of dynamic connasciences is that they cannot be understood without considering how the code runs. When we encounter any dynamic connasciences in legacy codebases, they can present quite a challenge to understand. In this post, I present an example of how removing this type of coupling from the code can quickly help efforts to improve the understanding of the system.

How I Got Here

For this article, I will be using the ugly trivia coding kata. This is a legacy code kata where the exercise is to refactor the code to a more understandable and maintainable state.

I was first introduced to this kata about two years ago during a remote pairing session with Adrian Bolboaca (with part 1 and part 2 posted on his YouTube channel). The approach we took was to create a Golden Master test and then start extracting methods from the tangled code to try and gain understanding. I was never particularly pleased with the result of the pairing session as we never got to the point of truly understanding the code.

I’ve since tried the same approach that I used with Adrian several times and it’s never gone particularly well. It gives the illusion that the code is becoming understandable, but it never escapes the fact that there’s a strange coupling between methods which is very hard to reason about.

At DDD Europe 2020 I took part in Johan Martinsson’s Bug Zero Kata workshop. Johan uses the same kata as the basis of this workshop. This time I took a very different approach with my pair. Rather than starting by trying to make the code more descriptive, we removed the source of the confusion early on — this source of confusion was the connascience of execution order.

At first, this step seemed to make the code messier (creating one big method), but the result was that all the tangled code was brought together into one place and the connascience of execution order was removed. From this point on, refactoring was easy.

Now I’ll show this in action.

The Initial Code

You can get the original, legacy Ugly Trivia code from GitHub. It comes in many different languages; I have chosen Java for this article. The code comprises of two classes, the Game class and the GameRunner class.

Below you can see some of the Game class, there’s more in it than what I have shown below, but this is the part we’re interested in. Please don’t try and understand the detail of this code, rather, notice that the methods roll(), wasAnsweredCorrectly() and wrongAnswer() are three different public methods that depend on the states of inPenaltyBox and isGettingOutOfPenaltyBox. Without knowing which order these are called in (or even when you do know) it’s very hard to understand what this code is doing.

public class Game {
    /* More fields are defined here */

    boolean[] inPenaltyBox  = new boolean[6];

    int currentPlayer = 0;
    boolean isGettingOutOfPenaltyBox;

    public void roll(int roll) {
        System.out.println(players.get(currentPlayer) +
            " is the current player");
        System.out.println("They have rolled a " + roll);

        if (inPenaltyBox[currentPlayer]) {
            if (roll % 2 != 0) {
                isGettingOutOfPenaltyBox = true;

                System.out.println(players.get(currentPlayer) +
                    " is getting out of the penalty box");
                places[currentPlayer] = places[currentPlayer] +
                    roll;
                if (places[currentPlayer] > 11)
                    places[currentPlayer] =
                        places[currentPlayer] - 12;

                System.out.println(players.get(currentPlayer)
                        + "'s new location is "
                        + places[currentPlayer]);
                System.out.println("The category is " +
                    currentCategory());
                askQuestion();
            } else {
                System.out.println(players.get(currentPlayer) +
                    " is not getting out of the penalty box");
                isGettingOutOfPenaltyBox = false;
            }
        } else {
            places[currentPlayer] = places[currentPlayer] + roll;
            if (places[currentPlayer] > 11)
                places[currentPlayer] = places[currentPlayer] - 12;

            System.out.println(players.get(currentPlayer)
                    + "'s new location is "
                    + places[currentPlayer]);
            System.out.println("The category is " +
                currentCategory());
            askQuestion();
        }
    }

    public boolean wasCorrectlyAnswered() {
        if (inPenaltyBox[currentPlayer]){
            if (isGettingOutOfPenaltyBox) {
                System.out.println("Answer was correct!!!!");
                purses[currentPlayer]++;
                System.out.println(players.get(currentPlayer)
                        + " now has "
                        + purses[currentPlayer]
                        + " Gold Coins.");

                boolean winner = didPlayerWin();
                currentPlayer++;
                if (currentPlayer == players.size())
                    currentPlayer = 0;

                return winner;
            } else {
                currentPlayer++;
                if (currentPlayer == players.size())
                    currentPlayer = 0;
                return true;
            }
        } else {
            System.out.println("Answer was corrent!!!!");
            purses[currentPlayer]++;
            System.out.println(players.get(currentPlayer)
                    + " now has "
                    + purses[currentPlayer]
                    + " Gold Coins.");

            boolean winner = didPlayerWin();
            currentPlayer++;
            if (currentPlayer == players.size()) currentPlayer = 0;

            return winner;
        }
    }

    public boolean wrongAnswer(){
        System.out.println("Question was incorrectly answered");
        System.out.println(players.get(currentPlayer) +
            " was sent to the penalty box");
        inPenaltyBox[currentPlayer] = true;

        currentPlayer++;
        if (currentPlayer == players.size()) currentPlayer = 0;
        return true;
    }

    /* More methods are defined here */
}

In the GameRunner class, we get to see how these methods are called. The two classes are coupled, such that the order in which the methods are called in GameRunner is critical to the behaviour inside the methods in Game. This is the connascience of execution order.

public class GameRunner {

    private static boolean notAWinner;

    public static void main(String[] args) {
        Game aGame = new Game();

        aGame.add("Chet");
        aGame.add("Pat");
        aGame.add("Sue");

        Random rand = new Random();

        do {
            aGame.roll(rand.nextInt(5) + 1);

            if (rand.nextInt(9) == 7) {
                notAWinner = aGame.wrongAnswer();
            } else {
                notAWinner = aGame.wasCorrectlyAnswered();
            }
        } while (notAWinner);
    }
}

This connascience of execution order is what makes the code in Game hard to understand.

Adding Tests

Before we make any changes, we need to add tests to give us the confidence that we don’t break anything. For legacy code like this, creating a golden master is a good approach. I’m not going to explain how to do this here, but you can find details in one of the following ways:

• Read Sandro’s post on golden master testing with the Approvals testing framework.
• Review the test from the Bug Zero kata codebase which uses the Approvals testing framework.
• Review the first video of the session I recorded with Adrian.

Since the GameRunner uses Random to generate its behaviour, we need to seed it to have a consistent test. To make this possible, I used IntelliJ’s Extract Method refactoring to separate the creation of the Random object from the rest of the code like so:

public class GameRunner {
    private static boolean notAWinner;

    public static void main(String[] args) {
        run(new Random());
    }

    public static void run(Random rand) {
        Game aGame = new Game();

        aGame.add("Chet");
        aGame.add("Pat");
        aGame.add("Sue");

        do {
            aGame.roll(rand.nextInt(5) + 1);

            if (rand.nextInt(9) == 7) {
                notAWinner = aGame.wrongAnswer();
            } else {
                notAWinner = aGame.wasCorrectlyAnswered();
            }
        } while (notAWinner);
    }
}

This allows the golden master test to create its own instance of Random and call the run() method with it, while still keeping the main() function behaving as it always has. Note, this is a temporary step and not something I’d leave in the codebase permanently.

For completeness, my test looks like this.

public class GameTest {
    @Test
    public void golden_master() {
        Approvals.verify(runWithStdout());
    }

    private String runWithStdout() {
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        PrintStream ps = new PrintStream(baos);
        PrintStream old = System.out;
        System.setOut(ps);

        for (int seed = 1; seed < 20; seed++) {
            Random random = new Random(seed);
            GameRunner.runMain(random);
        }

        System.out.flush();
        System.setOut(old);
        return baos.toString();
    }
}

Here you should use code coverage reports and mutation testing (see Billie Thompson’s video here if you don’t know what mutation testing is) to gain some confidence about how much you can rely on this test — finding ways to increase the confidence if it’s not good enough. I’m not going to go into any more detail on that now but it may be something I write more about in the future.

Removing Execution Order Connascience

Now we have a golden master test, we can confidently start refactoring. The first thing I want to try and do is remove the connascience of execution order. For this, I’m going to focus on the three methods being called inside the loop. Technically the calls to add() also form connascience of execution order, but they aren’t causing too much confusion right now, so I’ll leave that until later.

To remove connascience of execution order from the loop, I want GameRunner to make just one call to Game in each iteration of the loop. The first step I take is to separate the inputs from the logic inside of the loop in the run() method. Here the random numbers are the inputs, so I use the Extract Variable refactoring each time a random number is generated and move the declarations to the top of the loop (the important thing is to ensure the order of generating the numbers doesn’t change).

public static void run(Random rand) {
   Game aGame = new Game();

   aGame.add("Chet");
   aGame.add("Pat");
   aGame.add("Sue");

   do {
      int roll = rand.nextInt(5) + 1;
      boolean incorrectAnswer = rand.nextInt(9) == 7;
      
      aGame.roll(roll);

      if (incorrectAnswer) {
         notAWinner = aGame.wrongAnswer();
      } else {
         notAWinner = aGame.wasCorrectlyAnswered();
      }
   } while (notAWinner);
}

I also notice that notAWinner is defined as a static field on the class, but it is only ever used in local scope, so I make it a local variable.

public static void run(Random rand) {
   Game aGame = new Game();

   aGame.add("Chet");
   aGame.add("Pat");
   aGame.add("Sue");

   boolean notAWinner;
   
   do {
      int roll = rand.nextInt(5) + 1;
      boolean incorrectAnswer = rand.nextInt(9) == 7;

      aGame.roll(roll);

      if (incorrectAnswer) {
         notAWinner = aGame.wrongAnswer();
      } else {
         notAWinner = aGame.wasCorrectlyAnswered();
      }
   } while (notAWinner);
}

I now use Extract Method to move the logic to a new playTurn() method.

private static void run(Random rand) {
   Game aGame = new Game();

   aGame.add("Chet");
   aGame.add("Pat");
   aGame.add("Sue");

   boolean notAWinner;

   do {
      int roll = rand.nextInt(5) + 1;
      boolean incorrectAnswer = rand.nextInt(9) == 7;

      notAWinner = playTurn(aGame, roll, incorrectAnswer);
   } while (notAWinner);
}

private static boolean playTurn(
        Game aGame, int roll, boolean incorrectAnswer) {
   boolean notAWinner;
   aGame.roll(roll);

   if (incorrectAnswer) {
      notAWinner = aGame.wrongAnswer();
   } else {
      notAWinner = aGame.wasCorrectlyAnswered();
   }
   return notAWinner;
}

Finally, I can use IntelliJ’s Convert To Instance Method refactoring on playTurn() to move it into theGame class.

GameRunner:

public class GameRunner {
   public static void main(String[] args) {
      run(new Random());
   }

   private static void run(Random rand) {
      Game aGame = new Game();

      aGame.add("Chet");
      aGame.add("Pat");
      aGame.add("Sue");

      boolean notAWinner;

      do {
         int roll = rand.nextInt(5) + 1;
         boolean incorrectAnswer = rand.nextInt(9) == 7;

         notAWinner = aGame.playTurn(roll, incorrectAnswer);
      } while (notAWinner);
   }
}

Game:

public class Game {
    /* Other class content */

    public boolean playTurn(int roll, boolean incorrectAnswer) {
        boolean notAWinner;
        roll(roll);

        if (incorrectAnswer) {
            notAWinner = wrongAnswer();
        } else {
            notAWinner = wasCorrectlyAnswered();
        }
        return notAWinner;
    }
}

At this point, I have removed the connascience of execution order from the loop. But what has this achieved?

We still have all this complicated logic spread throughout the methods in Game, but now all the calls to these methods is from the same class (in the language of connascience, the coupling is more local). One benefit here is that the Game and GameRunner classes are less coupled, but we’ve only made a minor impact on the clarity of the code. Let’s see what can be done now.

Inlining the Methods

Because all these methods are now in the same class, I can inline them all into playTurn() and see if it presents any new ways to structure the code. Specifically, in this case, we can use the approach I described in Refactoring — Untangling Conditionals to localise the complexity and quickly refactor the logic to a more desirable state.

The inlining looks like this:

public boolean playTurn(int roll, boolean incorrectAnswer) {
    boolean notAWinner;
    System.out.println(players.get(currentPlayer) +
        " is the current player");
    System.out.println("They have rolled a " + roll);

    if (inPenaltyBox[currentPlayer]) {
        if (roll % 2 != 0) {
            isGettingOutOfPenaltyBox = true;

            System.out.println(players.get(currentPlayer) +
                " is getting out of the penalty box");
            places[currentPlayer] =
                places[currentPlayer] + roll;
            if (places[currentPlayer] > 11)
                places[currentPlayer] = places[currentPlayer] - 12;

            System.out.println(players.get(currentPlayer)
                    + "'s new location is "
                    + places[currentPlayer]);
            System.out.println("The category is " + 
                currentCategory());
            askQuestion();
        } else {
            System.out.println(players.get(currentPlayer) +
                " is not getting out of the penalty box");
            isGettingOutOfPenaltyBox = false;
        }
    } else {
        places[currentPlayer] = places[currentPlayer] + roll;
        if (places[currentPlayer] > 11)
            places[currentPlayer] = places[currentPlayer] - 12;

        System.out.println(players.get(currentPlayer)
                + "'s new location is "
                + places[currentPlayer]);
        System.out.println("The category is " + currentCategory());
        askQuestion();
    }

    if (incorrectAnswer) {
        System.out.println("Question was incorrectly answered");
        System.out.println(players.get(currentPlayer) +
            " was sent to the penalty box");
        inPenaltyBox[currentPlayer] = true;

        currentPlayer++;
        if (currentPlayer == players.size()) currentPlayer = 0;
        notAWinner = true;
    } else {
        boolean result;
        if (inPenaltyBox[currentPlayer]) {
            if (isGettingOutOfPenaltyBox) {
                System.out.println("Answer was correct!!!!");
                purses[currentPlayer]++;
                System.out.println(players.get(currentPlayer)
                        + " now has "
                        + purses[currentPlayer]
                        + " Gold Coins.");

                boolean winner = didPlayerWin();
                currentPlayer++;
                if (currentPlayer == players.size())
                    currentPlayer = 0;

                result = winner;
            } else {
                currentPlayer++;
                if (currentPlayer == players.size())
                    currentPlayer = 0;
                result = true;
            }
        } else {
            System.out.println("Answer was corrent!!!!");
            purses[currentPlayer]++;
            System.out.println(players.get(currentPlayer)
                    + " now has "
                    + purses[currentPlayer]
                    + " Gold Coins.");

            boolean winner = didPlayerWin();
            currentPlayer++;
            if (currentPlayer == players.size()) currentPlayer = 0;

            result = winner;
        }
        notAWinner = result;
    }
    return notAWinner;
}

This is a pretty horrible piece of code, but by applying the techniques from Refactoring — Untangling Conditionals, I’m able to start simplifying the code. This brings me to something like this:

public boolean playTurn(int roll, boolean incorrectAnswer) {
    System.out.println(players.get(currentPlayer) +
        " is the current player");
    System.out.println("They have rolled a " + roll);

    boolean notAWinner;
    notAWinner = moveAndAnswerQuestion(roll, incorrectAnswer);
    nextPlayer();
    return notAWinner;
}

private boolean moveAndAnswerQuestion(int roll,
                                      boolean incorrectAnswer) {
    if (inPenaltyBox[currentPlayer]) {
        if (failedToLeavePenaltyBox(roll, incorrectAnswer)) {
            return true;
        }
    }

    move(roll);
    askQuestion();

    if (incorrectAnswer) {
        answerIncorrectly();
        return true;
    }

    answerCorrectly();
    return didPlayerWin();
}

private boolean failedToLeavePenaltyBox(int roll,
                                        boolean incorrectAnswer) {
    boolean rollIsOdd = roll % 2 != 0;

    if (rollIsOdd) {
        System.out.println(players.get(currentPlayer) +
            " is getting out of the penalty box");
        return false;
    }

    System.out.println(players.get(currentPlayer) +
        " is not getting out of the penalty box");

    if (incorrectAnswer) {
        answerIncorrectly();
        return true;
    }

    return true;
}

private void answerCorrectly() {
    System.out.println("Answer was correct!!!!");
    purses[currentPlayer]++;
    System.out.println(players.get(currentPlayer)
        + " now has "
        + purses[currentPlayer]
        + " Gold Coins.");
}

private void answerIncorrectly() {
    System.out.println("Question was incorrectly answered");
    System.out.println(players.get(currentPlayer) +
        " was sent to the penalty box");
    inPenaltyBox[currentPlayer] = true;
}

private void move(int roll) {
    places[currentPlayer] = places[currentPlayer] + roll;
    if (places[currentPlayer] > 11)
        places[currentPlayer] = places[currentPlayer] - 12;

    System.out.println(players.get(currentPlayer)
        + "'s new location is "
        + places[currentPlayer]);
    System.out.println("The category is " + currentCategory());
}

private void nextPlayer() {
    currentPlayer++;
    if (currentPlayer == players.size()) currentPlayer = 0;
}

This last step took about 30 minutes of moving the structure of the code about and then extracting methods. The process was not mentally taxing and it didn’t require me to understand the code. It was just a process of repeatedly adjusting (improving) the structure and checking that the tests still passed.

During the process, I identified one typo bug and fixed that, and I completely removed the isGettingOutOfPenaltyBox field from the class. The result is not beautiful code, but it is rationalised enough that I can now start to understand the logic enough to make some more interesting architectural decisions (perhaps extracting out Roll and Player classes for example). I even started to understand enough to start questioning the logic and whether there might be a bug.

Conclusion

To refactor legacy code, we don’t necessarily need to understand the logic we are refactoring first, rather, we can use the process of refactoring to gain understanding. However, to do this we need tests that we trust! We can use techniques like golden master tests to when we don’t already have tests, and we can use code coverage and mutation testing to start to gain confidence in these tests. Note that the purpose of such a test is not to test the system behaviour is correct, but rather to test that we do not change the behaviour while we refactor.

Once we are ready to start refactoring, we might be tempted to start applying techniques which will make the code “more readable”. However, some times there is some complexity which gets in the way of this. Identifying and removing that first can be a more productive approach. In this case, the complexity came from the connascience of execution order. If you identify any of the dynamic types of connascience in your code then it’s a good indicator that it might be a candidate for something to try and remove first.

Refactoring — Removing Connascience of Execution Order was originally published in Cloudnative.ly on Medium, where people are continuing the conversation by highlighting and responding to this story.

As programmers, we love to minimise duplication and maximise reuse. We love the acronym DRY — Don’t Repeat Yourself, we hate deeply nested blocks of code, and we like small short methods and functions. However, sometimes duplicating code and embedding blocks inside other blocks can be the perfect enabler for refactoring. This is what I will demonstrate in this article.

Just to be clear, what I’m talking about are steps along the way in your refactoring journey. I’m not suggesting that you should leave long methods full of duplication and nested if statements in your code, rather, these are small steps you can take along the way. I previously recorded some videos about taking small steps when refactoring here and here, you may want to watch them first.

The Starting Code

I’ve created a small refactoring kata for this article (the repository is available here if you want to try it yourself). It starts with the following code. The goal is to untangle it into something easier to understand.

public void run(Project project) {
    boolean testsPassed;
    boolean buildSuccessful;

    if (project.hasTests()) {
        if ("success".equals(project.runTests())) {
            log.info("Tests passed");
            testsPassed = true;
        } else {
            log.severe("Tests failed");
            testsPassed = false;
        }
    } else {
        log.info("No tests");
        testsPassed = true;
    }

    if (testsPassed) {
        if ("success".equals(project.build())) {
            log.info("Deployment successful");
            buildSuccessful = true;
        } else {
            log.severe("Deployment failed");
            buildSuccessful = false;
        }
    } else {
        buildSuccessful = false;
    }

    if (config.sendEmailSummary()) {
        log.info("Sending email");
        if (testsPassed) {
            if (buildSuccessful) {
                emailer.send("Deployment completed successfully");
            } else {
                emailer.send("Deployment failed");
            }
        } else {
            emailer.send("Tests failed");
        }
    } else {
        log.info("Email disabled");
    }
}

The sign that the refactoring approach that I’m about to demonstrate is appropriate in this code is that there are values that are being set, checked, updated, and rechecked through multiple if statements. That is, there are multiple top-level if blocks that a coupled by local, mutable state.

All these embedded if statements make this code hard to follow. Maybe we can find a way to simplify this code by bringing the sets and checks closer together, let’s try it.

Adding Duplication

My initial goal here is to bring the checks of testsPassed and buildSuccessful closer to where they have their values set — maybe then we can even remove the mutable state altogether.

I’m going to start by moving the second top-level if block into both of the branches of the first top-level if block.

public void run(Project project) {
    boolean testsPassed;
    boolean deploySuccessful;

    if (project.hasTests()) {
        if ("success".equals(project.runTests())) {
            log.info("Tests passed");
            testsPassed = true;
        } else {
            log.severe("Tests failed");
            testsPassed = false;
        }

        if (testsPassed) {
            if ("success".equals(project.deploy())) {
                log.info("Deployment successful");
                deploySuccessful = true;
            } else {
                log.severe("Deployment failed");
                deploySuccessful = false;
            }
        } else {
            deploySuccessful = false;
        }
    } else {
        log.info("No tests");
        testsPassed = true;
        
        if (testsPassed) {
            if ("success".equals(project.deploy())) {
                log.info("Deployment successful");
                deploySuccessful = true;
            } else {
                log.severe("Deployment failed");
                deploySuccessful = false;
            }
        } else {
            deploySuccessful = false;
        }
    }

    if (config.sendEmailSummary()) {
        log.info("Sending email");
        if (testsPassed) {
            if (deploySuccessful) {
                emailer.send("Deployment completed successfully");
            } else {
                emailer.send("Deployment failed");
            }
        } else {
            emailer.send("Tests failed");
        }
    } else {
        log.info("Email disabled");
    }
}

In fact, I’m going to do the same thing again with the inner if statements in the first branch.

public void run(Project project) {
    boolean testsPassed;
    boolean deploySuccessful;

    if (project.hasTests()) {
        if ("success".equals(project.runTests())) {
            log.info("Tests passed");
            testsPassed = true;
            if (testsPassed) {
                if ("success".equals(project.deploy())) {
                    log.info("Deployment successful");
                    deploySuccessful = true;
                } else {
                    log.severe("Deployment failed");
                    deploySuccessful = false;
                }
            } else {
                deploySuccessful = false;
            }
        } else {
            log.severe("Tests failed");
            testsPassed = false;
            if (testsPassed) {
                if ("success".equals(project.deploy())) {
                    log.info("Deployment successful");
                    deploySuccessful = true;
                } else {
                    log.severe("Deployment failed");
                    deploySuccessful = false;
                }
            } else {
                deploySuccessful = false;
            }
        }
    } else {
        log.info("No tests");
        testsPassed = true;

        if (testsPassed) {
            if ("success".equals(project.deploy())) {
                log.info("Deployment successful");
                deploySuccessful = true;
            } else {
                log.severe("Deployment failed");
                deploySuccessful = false;
            }
        } else {
            deploySuccessful = false;
        }
    }

    if (config.sendEmailSummary()) {
        log.info("Sending email");
        if (testsPassed) {
            if (deploySuccessful) {
                emailer.send("Deployment completed successfully");
            } else {
                emailer.send("Deployment failed");
            }
        } else {
            emailer.send("Tests failed");
        }
    } else {
        log.info("Email disabled");
    }
}

The tests still pass but this definitely doesn’t feel like an improvement right?

However, now we have some of the if statements right after where testsPassed is being set. This makes some of these inner branches impossible to be reached, so let’s remove those.

Note, IntelliJ will help you locate and remove unreachable branches.

public void run(Project project) {
    boolean testsPassed;
    boolean deploySuccessful;

    if (project.hasTests()) {
        if ("success".equals(project.runTests())) {
            log.info("Tests passed");
            testsPassed = true;
            if ("success".equals(project.deploy())) {
                log.info("Deployment successful");
                deploySuccessful = true;
            } else {
                log.severe("Deployment failed");
                deploySuccessful = false;
            }
        } else {
            log.severe("Tests failed");
            testsPassed = false;
            deploySuccessful = false;
        }
    } else {
        log.info("No tests");
        testsPassed = true;

        if ("success".equals(project.deploy())) {
            log.info("Deployment successful");
            deploySuccessful = true;
        } else {
            log.severe("Deployment failed");
            deploySuccessful = false;
        }
    }

    if (config.sendEmailSummary()) {
        log.info("Sending email");
        if (testsPassed) {
            if (deploySuccessful) {
                emailer.send("Deployment completed successfully");
            } else {
                emailer.send("Deployment failed");
            }
        } else {
            emailer.send("Tests failed");
        }
    } else {
        log.info("Email disabled");
    }
}

That’s improved things a bit, but we are still left with some duplication. Before we address that, let’s repeat the process we just went through, this time for the bottom top-level if statement.

First duplicate:

public void run(Project project) {
    boolean testsPassed;
    boolean deploySuccessful;

    if (project.hasTests()) {
        if ("success".equals(project.runTests())) {
            log.info("Tests passed");
            testsPassed = true;
            if ("success".equals(project.deploy())) {
                log.info("Deployment successful");
                deploySuccessful = true;
                if (config.sendEmailSummary()) {
                    log.info("Sending email");
                    if (testsPassed) {
                        if (deploySuccessful) {
                            emailer.send("Deployment completed successfully");
                        } else {
                            emailer.send("Deployment failed");
                        }
                    } else {
                        emailer.send("Tests failed");
                    }
                } else {
                    log.info("Email disabled");
                }
            } else {
                log.severe("Deployment failed");
                deploySuccessful = false;
                if (config.sendEmailSummary()) {
                    log.info("Sending email");
                    if (testsPassed) {
                        if (deploySuccessful) {
                            emailer.send("Deployment completed successfully");
                        } else {
                            emailer.send("Deployment failed");
                        }
                    } else {
                        emailer.send("Tests failed");
                    }
                } else {
                    log.info("Email disabled");
                }
            }
        } else {
            log.severe("Tests failed");
            testsPassed = false;
            deploySuccessful = false;
            if (config.sendEmailSummary()) {
                log.info("Sending email");
                if (testsPassed) {
                    if (deploySuccessful) {
                        emailer.send("Deployment completed successfully");
                    } else {
                        emailer.send("Deployment failed");
                    }
                } else {
                    emailer.send("Tests failed");
                }
            } else {
                log.info("Email disabled");
            }
        }
    } else {
        log.info("No tests");
        testsPassed = true;

        if ("success".equals(project.deploy())) {
            log.info("Deployment successful");
            deploySuccessful = true;
            if (config.sendEmailSummary()) {
                log.info("Sending email");
                if (testsPassed) {
                    if (deploySuccessful) {
                        emailer.send("Deployment completed successfully");
                    } else {
                        emailer.send("Deployment failed");
                    }
                } else {
                    emailer.send("Tests failed");
                }
            } else {
                log.info("Email disabled");
            }
        } else {
            log.severe("Deployment failed");
            deploySuccessful = false;
            if (config.sendEmailSummary()) {
                log.info("Sending email");
                if (testsPassed) {
                    if (deploySuccessful) {
                        emailer.send("Deployment completed successfully");
                    } else {
                        emailer.send("Deployment failed");
                    }
                } else {
                    emailer.send("Tests failed");
                }
            } else {
                log.info("Email disabled");
            }
        }
    }
}

Then remove all unreachable branches:

public void run(Project project) {
    boolean testsPassed;
    boolean deploySuccessful;

    if (project.hasTests()) {
        if ("success".equals(project.runTests())) {
            log.info("Tests passed");
            testsPassed = true;
            if ("success".equals(project.deploy())) {
                log.info("Deployment successful");
                deploySuccessful = true;
                if (config.sendEmailSummary()) {
                    log.info("Sending email");
                    emailer.send("Deployment completed successfully");
                } else {
                    log.info("Email disabled");
                }
            } else {
                log.severe("Deployment failed");
                deploySuccessful = false;
                if (config.sendEmailSummary()) {
                    log.info("Sending email");
                    emailer.send("Deployment failed");
                } else {
                    log.info("Email disabled");
                }
            }
        } else {
            log.severe("Tests failed");
            testsPassed = false;
            deploySuccessful = false;
            if (config.sendEmailSummary()) {
                log.info("Sending email");
                emailer.send("Tests failed");
            } else {
                log.info("Email disabled");
            }
        }
    } else {
        log.info("No tests");
        testsPassed = true;

        if ("success".equals(project.deploy())) {
            log.info("Deployment successful");
            deploySuccessful = true;
            if (config.sendEmailSummary()) {
                log.info("Sending email");
                emailer.send("Deployment completed successfully");
            } else {
                log.info("Email disabled");
            }
        } else {
            log.severe("Deployment failed");
            deploySuccessful = false;
            if (config.sendEmailSummary()) {
                log.info("Sending email");
                emailer.send("Deployment failed");
            } else {
                log.info("Email disabled");
            }
        }
    }
}

Now we see that the intermediate variables are no longer used, let’s remove them too:

public void run(Project project) {
    if (project.hasTests()) {
        if ("success".equals(project.runTests())) {
            log.info("Tests passed");
            if ("success".equals(project.deploy())) {
                log.info("Deployment successful");
                if (config.sendEmailSummary()) {
                    log.info("Sending email");
                    emailer.send("Deployment completed successfully");
                } else {
                    log.info("Email disabled");
                }
            } else {
                log.severe("Deployment failed");
                if (config.sendEmailSummary()) {
                    log.info("Sending email");
                    emailer.send("Deployment failed");
                } else {
                    log.info("Email disabled");
                }
            }
        } else {
            log.severe("Tests failed");
            if (config.sendEmailSummary()) {
                log.info("Sending email");
                emailer.send("Tests failed");
            } else {
                log.info("Email disabled");
            }
        }
    } else {
        log.info("No tests");

        if ("success".equals(project.deploy())) {
            log.info("Deployment successful");
            if (config.sendEmailSummary()) {
                log.info("Sending email");
                emailer.send("Deployment completed successfully");
            } else {
                log.info("Email disabled");
            }
        } else {
            log.severe("Deployment failed");
            if (config.sendEmailSummary()) {
                log.info("Sending email");
                emailer.send("Deployment failed");
            } else {
                log.info("Email disabled");
            }
        }
    }
}

We are left with a single, large, multi-level conditional block. This block contains quite a lot of duplication, so now let’s change our focus to addressing that.

Removing Duplication

The most obviously repeated blocks are the if (config.sendEmailSummary()) ones. Let’s use extract method to remove this duplication.

public void run(Project project) {
    if (project.hasTests()) {
        if ("success".equals(project.runTests())) {
            log.info("Tests passed");
            if ("success".equals(project.deploy())) {
                log.info("Deployment successful");
                sendEmail("Deployment completed successfully");
            } else {
                log.severe("Deployment failed");
                sendEmail("Deployment failed");
            }
        } else {
            log.severe("Tests failed");
            sendEmail("Tests failed");
        }
    } else {
        log.info("No tests");

        if ("success".equals(project.deploy())) {
            log.info("Deployment successful");
            sendEmail("Deployment completed successfully");
        } else {
            log.severe("Deployment failed");
            sendEmail("Deployment failed");
        }
    }
}

private void sendEmail(String message) {
    if (config.sendEmailSummary()) {
        log.info("Sending email");
        emailer.send(message);
    } else {
        log.info("Email disabled");
    }
}

Now things are starting to look better!

Next, we can use the Invert 'if' condition and convert conditional to guard clause refactorings to make it possible to remove the other duplicated block.

public void run(Project project) {
    if (project.hasTests()) {
        if (!"success".equals(project.runTests())) {
            log.severe("Tests failed");
            sendEmail("Tests failed");
            return;
        }
        log.info("Tests passed");
    } else {
        log.info("No tests");
    }

    if ("success".equals(project.deploy())) {
        log.info("Deployment successful");
        sendEmail("Deployment completed successfully");
    } else {
        log.severe("Deployment failed");
        sendEmail("Deployment failed");
    }
}

private void sendEmail(String message) {
    if (config.sendEmailSummary()) {
        log.info("Sending email");
        emailer.send(message);
    } else {
        log.info("Email disabled");
    }
}

Finally, we can use all the basic refactoring techniques to extract methods and remove else statements. With a bit of work we can refactor the code to something much more understandable and manageable:

public void run(Project project) {
    if (testsFailed(project)) {
        sendEmail("Tests failed");
        return;
    }

    if (deployFailed(project)) {
        sendEmail("Deployment failed");
        return;
    }

    sendEmail("Deployment completed successfully");
}

private boolean testsFailed(Project project) {
    if (!project.hasTests()) {
        log.info("No tests");
        return false;
    }

    if ("success".equals(project.runTests())) {
        log.info("Tests passed");
        return false;
    }

    log.severe("Tests failed");
    return true;
}

private boolean deployFailed(Project project) {
    boolean deployFailed = !"success".equals(project.deploy());
    if (deployFailed) {
        log.severe("Deployment failed");
        return true;
    }

    log.info("Deployment successful");
    return false;
}

private void sendEmail(String s) {
    if (!config.sendEmailSummary()) {
        log.info("Email disabled");
        return;
    }

    log.info("Sending email");
    emailer.send(s);
}

Final Thoughts

Often when refactoring, it’s tempting to reach straight for techniques which break our code down into small, digestible pieces (e.g. extract method). If the bigger pieces (i.e. the three top-level if blocks in the example in the article) that we are breaking down are tightly coupled to each other, then breaking each of them down independently leave things even more tangled. A way to avoid this is to bring the coupled pieces of code together first — even if this introduces duplication — and then start to address the duplications once the coupling is localised.

I hope you found this useful and maybe have a go at the kata yourself ;-)

Resources

• The Untangled Conditionals Kata

Refactoring — Untangling Conditionals was originally published in Cloudnative.ly on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this article, I want to give a basic introduction to what Event Sourcing is, and share some thoughts from the community about deciding if it’s right for your project.

I’ve had a keen interest in event sourcing for several years, however, in the last two years I was not actively following the community. In that time I’ve seen some posts and articles popping up on social media with tales of event sourcing, but not in the way that I understood it. Often it would seem that I was reading about Event Driven Architectures with a Kafka log stored somewhere.

I felt like I had fallen behind the times and that maybe Event Sourcing had evolved. When DDD Europe 2020 announced that they were going to hold a one day, pre-conference conference on Event Sourcing, I jumped at the chance to get back up to speed. After attending this event (and the fabulous DDD Europe main conference) I was happy to learn that Event Sourcing is still what I knew it to be. More importantly, I also learned some valuable lessons from the advice and experience shared in the talks.

What Event Sourcing is

In the opening keynote at the event sourcing conference, Udi Dahan said:

Event sourcing refers to a collection of patterns based on persisting the full history of a domain as a sequence of “events”, rather than persisting just the current state.

Storing Events

With event sourcing, we store a record of any event that changes the state, rather than the resulting state after the event occurred. Events are stored as immutable pieces of data that record what event happened, along with any data that is relevant to that event. Events are immutable because they represent something that has already happened (you cannot change the past). As such, they are named as actions in the past tense, e.g. OrderPlaced, PaymentTaken, OrderShipped.

Events are stored in streams, with a stream of events representing the entire history of a particular entity (or Aggregate in DDD terminology). These event streams are the source of truth for the state of the aggregate.

Once actions have been performed on the aggregate, we need to be able to get the events out to persist them in the event store.

As an example, I’m going to use an Order aggregate from an e-commerce system. You can create an order, add items to it, and then place it. Below you can see the public methods which perform these actions. In addition, there is also retrieveNewEvents()which will return all new events that have been generated since the object instance was constructed. These events can then be persisted to the event store.

public class Order {
    public static Order create(OrderID orderID) {
        // Create a new Order (generates OrderCreated)
    }

    public void addItem(ItemCode item) {
        // Add an item to the order (generates ItemAdded)
    }

    public void place() {
        // Place the order (generates OrderPlaced)
    }

    public List<OrderEvent> retrieveNewEvents() {
        // Return all new events have have been generated
    }

    // ...
}

Replaying an Event Stream

To load an event sourced aggregate’s state into memory, the stored events need to be replayed in order. To be able to do this, we need need to have another constructors for the aggregate which takes the stream of events

public class Order {
    public static Order fromEventStream(ArrayList<OrderEvent> {
        // Create new Order and replay events to get the state
    }

    // ...
}

We now have two constructors:

1. create() creates a brand new order and adds an initial OrderCreated event to the list of new events.
2. fromEventStream() builds the current state from a stream of events. It also initialises the list of new events to an empty list.

Let’s now take a look at a simplistic way that we might implement this. It is possible to refactor this code into a better state, but I want to keep it as simple as possible to highlight the key concepts.

To start with, let’s look at the state in the Order class:

private ArrayList<OrderEvent> newEvents = new ArrayList<>();

private OrderID id;
private HashMap<ItemCode, Quantity> lines = new HashMap<>();
private boolean placed = false;

id, lines and placed are the current state of the aggregate, while newEvents is used to keep a track of events as they occur.

Let’s now see that how the create() constructor works:

public static Order create(OrderID orderID) {
    var orderCreated = new OrderCreated(orderID);
    var events = Collections.singletonList(orderCreated);
    var order = Order.fromEventStream(new ArrayList<>(events));
    order.newEvents.add(orderCreated);
    return order;
}

The code above creates a new stream, containing a single OrderCreated event, and then creates the Order from that stream. This shows that actually fromEventStream() is the important constructor. The only other thing that create() does is save the event to newEvents.

Now let’s look at fromEventStream():

public static Order fromEventStream(ArrayList<OrderEvent> events) {
    return new Order(events);
}

private Order(ArrayList<OrderEvent> events) {
    if (events.size() == 0) {
        throw new EmptyEventStream();
    }

    if (!(events.get(0) instanceof OrderCreated)) {
        throw new InvalidEvent(
                "The first event must be OrderCreated"
        );
    }

    for (OrderEvent event : events) {
        applyEvent(event);
    }
}

We see that fromEventStream() just delegates through to a private constructor. The private constructor does a few checks on the event stream and then iterates over it, calling applyEvent() with each event.

applyEvent() is a private method that takes all event types and mutates the state accordingly. Here I have implemented it with an overloaded method and a nasty switch statement to check the event type. There are better ways to do this but it keeps the example simple:

private void applyEvent(OrderEvent event) {
    if (event instanceof OrderCreated) {
        applyEvent((OrderCreated) event);
    } else if (event instanceof ItemAdded) {
        applyEvent((ItemAdded) event);
    } else if (event instanceof OrderPlaced) {
        applyEvent((OrderPlaced) event);
    }
}

private void applyEvent(OrderCreated event) {
    id = event.id;
}

private void applyEvent(ItemAdded event) {
    if (placed) {
        throw new OrderHasAlreadyBeenPlaced(id);
    }

    var quantity = lines.containsKey(event.itemCode)
            ? lines.get(event.itemCode).increment()
            : Quantity.of(1);

    lines.put(event.itemCode, quantity);
}

private void applyEvent(OrderPlaced event) {
    if (placed) {
        throw new OrderHasAlreadyBeenPlaced(id);
    }

    if (lines.isEmpty()) {
        throw new OrderHasNoItems(id);
    }

    placed = true;
}

We have now seen how our event sourced aggregate’s state can be created from a stream of events. The final piece of the puzzle is how we create new events. With all the internal event driven infrastructure in place, this is very easy:

public void addItem(ItemCode item) {
    OrderEvent event = new ItemAdded(item);
    applyEvent(event);
    newEvents.add(event);
}

public void place() {
    OrderEvent event = new OrderPlaced();
    applyEvent(event);
    newEvents.add(event);
}

Now we have the complete example, I hope that it’s clear what event sourcing is. This example has been highly trivialised though, the core concepts are clear but event sourcing brings a lot of new complexity; for example:

• How do you deal with conflicts when you try and save new events but someone else has saved some new ones before you?
• How do you version events? Since they are immutable, the old versions still need to be understood by the system.
• What about performance when you have large event streams?
• How do you delete sensitive data from the history (e.g. comply with GDPR)?

The answers to these questions are beyond this scope of this article, but it’s important to know that the simple concept of event sourcing brings a lot of new challenges.

You can find the complete code for the example here.

When to use Event Sourcing

Now that we know what event sourcing is, when should we use it?

Interestingly, a fairly consistent message at the talks in the conference was a strong warning against event sourcing all the things. Once you start thinking in domain events it seems like everything should be event sourced because it maps to the way we experience reality. However, the complexities of implementing this powerful technique make it a trade-off that should be carefully considered.

First and foremost, you should consider what event sourcing gives you:

• A full audit log (also useful for understanding and debugging issues)
• The ability to restore the system to any point in history
• The ability to gather information retrospectively
• Decoupling the state from what happened

These all sound great, but the question is - does the value event sourcing brings to your solution outweigh the challenges to implement it?

In addition to this, Udi also gave some guidance on where it’s appropriate to use event sourcing in his keynote. Firstly he defined event sourcing as “a collection of patterns based on persisting the full history of a domain”. He then talked about how event sourcing should be applied to just small, appropriate domains within your organisation. He used the following advice from the definition of a domain model in Patterns of Enterprise Application Architecture to suggest how you might identify somewhere where implementing event sourcing might make sense:

“Use where you have complicated and ever-changing business rules.”

One other piece of guidance that Udi gave was, that event sourcing can be a good fit if you have a bi-temporal domain — that is, “something happened, and, it is valid until…”

Event Source vs Event Driven Architecture

One common theme of the talks during the conference were people sharing the lessons they’d learned and the mistakes that were made along the way. Amongst this advice, there were some common themes.

The most common theme that I noticed throughout the talks was that often people confuse event sourcing with Event Driven Architectures. That is to say that the events were viewed as a means of communication between services. Event sourcing and Event Driven Architectures can be used together, but they are not the same thing.

The big problem with merging the two concepts comes when you start publishing your internal event sourced events as public messages to be consumed by other services. Doing so couples the internal domain to the external messaging. You may indeed want to store an OrderPlaced event and publish one to a message bus for other services to consume, but different information might need to be in these data structures. For example, the internal event stored event does not need to include many details because they are already in the event history. However, the OrderPlaced message might be the first public message to be published about this order and should contain all the relevant details (i.e. order ID, order line details, etc.)

In Conclusion

Having a dedicated event sourcing conference is a great sign that event sourcing is something you should know about. One thing that appeared on a few slides during the conference was the Gartner Hype Cycle graph, with a suggestion that many people are somewhere between the Peak of Inflated Expectations and the Trough of Disillusionment. Meanwhile, the community leaders are guiding us to the Plateau of Productivity.

I hope this article has served as a useful and digestible introduction.

References

• DDD Europe 2020
• Slides from Udi Dahan’s keynote
• The example code used in this article
• Greg Young’s CQRS/ES PDF — a great introduction to both CQRS & event sourcing
• Event Sourcery — a great event sourcing tutorial

What is Event Sourcing? was originally published in Cloudnative.ly on Medium, where people are continuing the conversation by highlighting and responding to this story.

A few days ago, I posted a simple tweet about how investing in design pays back very quickly. This led to a short conversation with Aamir Idrees, about how much engineering is enough, before it starts to become over-engineering.

In my response, I said a phrase which I don’t think I’ve used before: “be sure to have a clear conscience on what you deliver”. Thinking back on it, I realise that this is my perfect way to summarise how I decide if a piece of work is complete. Here, I’d like to explain what I mean in more detail.

Having a Clear Conscience

clear conscience noun

a knowledge or belief that one has done nothing bad or wrong

Definition from Merriam Webster

To commit code with a clear conscience, you must believe that you have not done anything bad or wrong while creating it. To quantify whether this is true, we must first understand who might judge our work as being bad or wrong. I’m going to generalise these people into three groups:

• People who benefit from the running software
• The people investing in your work
• Other developers

I now want to look at our obligations to each of these groups:

The People Who Benefit From the Running Application

I’ve intentionally made the description of this group a bit vague. It could refer to the end-users of the software, they obviously benefit from using the software (otherwise why would they use it?). However, there many other people in this category. For example:

• It could also be people downstream from the end-users (e.g. as a customer of a telephone network, I benefit from the CRM that their support staff are using when I call up to query about my account).
• It could be the people who are paying you to develop the software. The software may be the only reason they have a business.
• It could even be you and the other developers, your job may be based on the success of the software.

Regardless of who benefits from the running software, your obligation to them is to build software which performs at a satisfactory level and does not cause any harm (e.g. loss of money, physical or emotional harm, loss of productivity, etc.).

This gives us the first question to ask yourself when delivering a feature:

Could this feature have any impact on the software which might cause distress to the people who benefit from it?

Some things to consider when answering this question are:

• Are you confident that the users will be able to use it?
• Are you confident that the changes haven’t broken the system elsewhere?
• Do you have SLAs,2 SLOs3 and SLIs4 that you meet and track?

The People Investing in Your Work

Hopefully, you are being paid to write code. Hopefully, you are being paid well to write code. If this is the case, then the people paying you are paying for a reason — they want a return on their investment. The tricky part here is that not only should you deliver working features on time, but you also need to deliver them in a way which doesn’t hinder further development later on.

The question here is one you should be asking daily:

Am I doing a professional job while keeping a clear focus on what I’m trying to deliver?

The key factors to enabled you to consistently answer yes to this question, are:

• Reviewing performance retrospectively, and applying learnings from these experiences.
• Clearly communicating progress and challenges to manage expectations.
• Architecting software with a view to the future; enabling accurate understanding for future tasks.

Other developers

As part of a team, everyone should help each other succeed. By creating good quality as sustainable code, can help your team achieve their goals. Confusion or misleading code can cause other developers to loose hours trying to understand it. Tangled or tightly coupled code can be very hard to extend. Missing tests make it hard to be confident that your changes work. To have a clear conscience, we should deliver code that other team members will enjoy working with.

The question we should ask ourselves here is:

Is my code going to make the work on future features harder?

This time, I’m going to go a bit deeper into each of the factors which contribute to delivering high-quality code.

Testing

Tests enable other developers (or our future selves) to change and extend the system with confidence. Having a good quality test suite helps us avoid wasting unnecessary time checking that we haven’t inadvertently broken something.

To have a clear conscience, you should ask yourself “If something is changed that would alter the behaviour of the system, will a test fail?”

There are cases when it’s acceptable to have some code which is not tested. An example might be where the gain of testing the integration with an external system does not outweigh the effort. In such a case, we can still ask how we will know when it fails (monitoring may be) and have we sufficiently handled failure. But we should also make an effort to isolate such code so that it’s easy to identify should a failure occur.

Regardless of the specifics of testing, a clear conscience comes from knowing you have protected the code from inadvertent damage, and enabled developers to continue to develop the system.

Formatting

Having a consistent formatting style in a project reduces the cognitive load for the people reading it. When the layout is consistent, our eyes instinctively follow the code, and we don’t have to hunt so hard for the logic we are looking for.

Ask yourself “Have I laid out my code in a way which is consistent to the project?”

The important thing is that the team chooses a coding style and everyone follows it. Better yet, use a linter to check/fix your code for you. It’s important that you code to the project’s coding style, not your own. Having a clear conscience is not about you thinking it’s perfect, rather, it’s about ensuring that it doesn’t frustrate your colleagues.

Naming

The name of the variables, methods, classes, etc. is one of the best ways that we can ensure other developers can understand what the intent of the code is. Unfortunately, choosing the right name can be one of the hardest decisions a developer has to make.

Ask yourself “Have I named anything in a way which will cause confusion?”

If you can’t think of the perfect name then it’s not a problem (we have powerful refactoring tools at our fingertips which makes renaming things simple). It is a problem though if our names cause confusion or misunderstanding.

Well Architected

How we structure our programs has a huge impact on how future work is done. On one hand, we might be able to make something work as intended with a thousand lines of ifs, fors and gotos. On the other hand, we have a huge library of Design Patterns, SOLIDs, YAGNIs, Laws of Demeter, Object Calisthenics, the list goes on - and we know we should be using them all, but sometimes it seems like too much. So what’s the right amount?

The question is simple “Will other developers curse my name when working with my code?”

We’ve all thought “What was X thinking when they wrote this code!?”. We’ve all been X. And worse still, we’ve all thought that about code where X was ourselves.

By considering if there’s anything in our code which might cause someone to ask this question, and fixing it before we deliver it, we can improve others lives and have a clear conscience.

Interestingly, the culture of the team can make a big difference to this question. If you’re working in a team of people who have a strong crafting culture, then a YAGNI approach can be the perfect way to deliver something. If you know that when someone picks up the code for the next feature, they will start to factor it into a shape which fits the evolving architecture, then this is probably the most efficient way to work. On the other hand, if the culture in the team doesn’t work this way, then you probably need to engineer the solution a bit more to ensure it guides other team members. However, in the latter case, I think efforts should be investing in improving the culture (and possibly skill level in the team).

Finally, we should make efforts to conform to the architecture of the project; it’s never fun trying to navigate some code which solved a problem in three different ways at the same time. At the same time, we shouldn’t be afraid to change the original architecture if we collectively agree it is a hindrance.

Failure Cases

Things fail, that’s a fact. If that failure causes distress to the end-users, then that’s a problem. If reported failures from end-users cannot be easily tracked down and diagnosed, then everyone suffers.

The obvious question here is “Have I handled all failure cases sufficiently?”

This can involve many things, such as handling nulls, catching exceptions, raising exceptions, displaying meaningful error messages, etc.

Other Factors

There are also many other things which might be important to consider, depending on your specific circumstances:

• Security: “Is my code secure?”, “Can any personal details be leaked?”, “Are the dependencies up to date?”, etc.
• Logging: “Have I logged useful messages?”, “Have I logged at the right error level?”, “Am I logging any unnecessary noise?”, etc.
• Accessibility: Does the design of the feature exclude anyone from using it?

There are many more, but I’ll leave it up to you to decide what is appropriate for each situation.

Summary

To have a clear conscience you must deliver software which does not knowingly do bad or wrong things in the eyes of any of these groups of people. It is not possible to do this without considering all groups. If you over-engineer the perfect solution, then the developers might be very happy, but features might be very expensive to develop. Conversely, if you rush features out the door as quickly as possible, then developers are likely to get unhappy with working conditions, and end-users are likely to get unhappy with broken features.

With this in mind, the only way we can keep them all happy is by managing expectations; bending over backwards to make one person happy is likely to have negative effects on the other people.

It’s also worth noting that these choices are context-specific; many factors can determine what the minimum viable implementation looks like. Some of these include:

• Number of users using the system
• The scale of the system, teams or organisation
• Do the developers have the right tools (e.g. refactoring tools)

I hope this article has provided some guidance towards keeping your conscience clear, while keeping productivity high.

Writing Code With A Clear Conscience was originally published in Cloudnative.ly on Medium, where people are continuing the conversation by highlighting and responding to this story.

These days, the functions map, filter and fold/reduce are pretty well known by most programmers. Any pure data manipulation performed in a loop can be re-written with a fold, but it’s not always easy to understand. We use map and filter when we can to make things clearer, and fold is useful when folding a collection down to a single value. However, for more complex loops, most would argue that an imperative loop is easier to understand than trying to wrap up all the state in the accumulator of a fold. In this article, we look at a situation where using fold to map over a collection while maintaining state can be clarified by mapping over a State monad.

The Challenge

The map function lets us transform each element in a list, like this:

newList = map (+1) [1, 2, 3]
-- newList is [2, 3, 4]

But what if we want to do something like this (Javascript):

function addAnIncrementingAmount(numbers) {
  let result = [];
  let amountToAdd = 0;

  for (const number of numbers) {
    result.push(number + amountToAdd);
    amountToAdd++;
  }

  return result;
}

const newList = addAnIncrementingAmount([1, 2, 3]);
// newList is [1, 3, 5]

Even though the result is a list mapping each element from an element provided in the argument, we are unable to use map because the calculation for each element is dependant on the previous one. Therefore, we have to use foldl instead:

addAnIncrementingAmount :: [Int] -> [Int]
addAnIncrementingAmount numbers =
  fst $ foldl addAmount addAmount ([], 0) numbers

addAmount :: ([Int], Int) -> Int -> ([Int], Int)
addAmount (result, amountToAdd) number =
  (result ++ [number + amountToAdd], amountToAdd + 1)

While this code is fairly simple (because the example itself is very simple), if we had more stateful values used in the calculation then it would get more complex. Also, the addAmount is trying to do two things at once - i.e. build the resulting list and perform the calculations.

Generalising with a new higher-order function

Let’s start by separating the calculation from the stateful mapping. We can do this by creating a new mapWithState higher-order function.

addAnIncrementingAmount :: [Int] -> [Int]
addAnIncrementingAmount numbers = mapWithState addAmount 0 numbers

mapWithState :: (a -> state -> (b, state)) -> state -> [a] -> [b]
mapWithState f state xs = fst $ foldl f ([], state) xs

addAmount :: Int -> Int -> (Int, Int)
addAmount number amountToAdd = (number + amountToAdd, amountToAdd + 1)

This is a nice separation of concerns. The mapWithState function is concerned with building the resulting list and passing the state through, while the addAmount function is only concerned with our calculation. This is a good start, but it’s not clear in the type of addAmount which argument is the state and which is the element we are mapping - can we do something about that?

The type signature should give us a clue as to what we could try. The fact that it returns a tuple of (result, newState) should make us think about the State monad.

The State Monad

State is available via Control.Monad.State of the mtl1 package. It is defined like this:

newtype State s a = State { runState :: State s a -> s -> a }

And is used like this:

import Control.Monad.State (State, get, modify, put, evalState)

performCalculation :: Int -> Int
performCalculation n = evalState statefulCalculation n

statefulCalculation :: State Int Int
statefulCalculation = do
  initalValue <- get
  put (initialValue + 3)
  modify (*2)
  newValue <- get 
  return (newValue - initalValue)

Which would be be the equivilent to this in Javascript:

function performCalculation(initalValue) {
  let newValue = initalValue + 3
  newValue = newValue * 2
  return (newValue - initalValue)
}

While the example above isn’t particularly beautiful, I hope it provides enough information about using the State monad for our example. One additional point about State (or any other monad) is that its context is maintained through functional calls within the monad. Let’s see that:

import Control.Monad.State (State, get, modify, put, evalState)

performCalculation :: Int -> Int
performCalculation n = evalState statefulCalculation n

statefulCalculation :: State Int Int
statefulCalculation = do
  initalValue <- get
  add 3
  multiply 2
  newValue <- get 
  return (newValue - initalValue)

add :: Int -> State Int ()
add amount = modify (+amount)

multiply :: Int -> State Int ()
multiply amount = modify (*amount)

Back to our example

Now we know about the State monad, let’s try re-writing the addAmount function to use it:

addAmount :: Int -> State Int Int
addAmount number = do
  amountToAdd <- get
  modify (+1)
  return (number + amountToAdd)

Now the type tells us something interesting about the function — it takes an Int and returns an Int, but it also maintains some state (also of type Int). But how do we use it?

Traversable

We know that we want to our addAmount function map over our input list, so what do we get if we try that now?

Prelude Control.Monad.State> :t map addAmount [1, 2, 3]
map addAmount [1, 2, 3] :: [State Int Int]

We’ve individually added each element into a State and ended up with [State Int Int], but that’s not what we need, we want to compute the entire list inside State - i.e. State Int [Int]. Wouldn’t it be great if there was a simple function to convert from what we have into what we want - i.e. [State Int Int] -> State Int [Int].

It turns out that lists implement the typeclass Traversable. If we look at the definition of Traversable then we see this:

class (Functor t, Foldable t) => Traversable (t :: * -> *) where
  traverse :: Applicative f => (a -> f b) -> t a -> f (t b)
  sequenceA :: Applicative f => t (f a) -> f (t a)
  mapM :: Monad m => (a -> m b) -> t a -> m (t b)
  sequence :: Monad m => t (m a) -> m (t a)

If we represent our list with Traversable t and our State monad with Monad m, then the function we after would have the type t (m a) -> m (t a) - that’s exactly the type of sequence in the typeclass above! Let’s try using it:

addAnIncrementingAmount :: [Int] -> [Int]
addAnIncrementingAmount numbers =
  evalState (sequence (map addAmount numbers)) 0

addAmount :: Int -> State Int Int
addAmount number amountToAdd number = do
  amountToAdd <- get
  modify (+1)
  return (number + amountToAdd)

This works exactly as expected. In fact, this combination of map and sequence is so useful, that if we look at the type signatures long enough, we see that mapM in Traversable is the composition of these two functions:

mapM :: (a -> m b) -> t a -> m (t b)

Let’s try mapM:

addAnIncrementingAmount :: [Int] -> [Int]
addAnIncrementingAmount numbers =
  evalState (mapM addAmount numbers) 0

addAmount :: Int -> State Int Int
addAmount number amountToAdd number = do
  amountToAdd <- get
  modify (+1)
  return (number + amountToAdd)

At this point, I hope that I’ve managed to practically show how State and mapM can be used together without digging too deeply into the details (there’s plenty of excellent resources on this elsewhere 2). However, my example so far is not that useful, let’s look at something a bit more complex.

The Bowling Game Kata

The Bowling Game Kata is a well-know Test-Driven Development exercise where the goal is to calculate the score of a game of ten-pin bowling.

A typical Javascript solution might look something like this:

class Game {
  constructor() {
    this.rolls = [];
  }

  roll(pins) {
    this.rolls.push(pins);
  }

  score() {
    let score = 0;
    let framePointer = 0;

    for (const frame of Array(10).keys()) {
      if (this.isStrike(framePointer)) {
        score += this.scoreStrike(framePointer);
        framePointer += 1;
      } else if (this.isSpare(framePointer)) {
        score += this.scoreSpare(framePointer);
        framePointer += 2;
      } else {
        score += this.scoreNormalFrame(framePointer);
        framePointer += 2;
      }
    }

    return score;
  }

  isStrike(framePointer) {
    return this.rolls[framePointer] === 10;
  }

  isSpare(framePointer) {
    return this.rolls[framePointer] + this.rolls[framePointer + 1] === 10;
  }

  scoreStrike(framePointer) {
    return 10 + this.rolls[framePointer + 1] + this.rolls[framePointer + 2];
  }

  scoreSpare(framePointer) {
    return 10 + this.rolls[framePointer + 2];
  }

  scoreNormalFrame(framePointer) {
    return this.rolls[framePointer] + this.rolls[framePointer + 1];
  }
}

The score function has a for loop in it so this is the bit which will need to be re-written to convert it to Haskell. Let’s start by extracting the scoring of each frame and then add up the results at the end.

score() {
    let frameScores = [];
    let framePointer = 0;

    for (const frame of Array(10).keys()) {
      const [frameScore, rollsInFrame] = this.scoreFrame(framePointer);
      frameScores.push(frameScore);
      framePointer += rollsInFrame;
    }

    return frameScores.reduce((score, frameScore) => score + frameScore);
  }

  scoreFrame(framePointer) {
    if (this.isStrike(framePointer)) {
      return [this.scoreStrike(framePointer), 1];
    } else if (this.isSpare(framePointer)) {
      return [this.scoreSpare(framePointer), 2];
    } else {
      return [this.scoreNormalFrame(framePointer), 2];
    }
  }

Now we’ve extract scoreFrame we can convert it to Haskell by replacing the for loop with a foldl (the implementation below isn’t like for like, but it’s close enough):

score :: [Int] -> Int
score rolls = sum $ fst $ foldl scoreFrame ([], rolls) [1..10]

scoreFrame :: ([Int], [Int]) -> Int -> ([Int], [Int])
scoreFrame state@(frameScores, rolls) _frame
  | isStrike rolls = scoreStrike state
  | isSpare rolls  = scoreSpare state
  | otherwise      = scoreNormal state

isStrike :: [Int] -> Bool
isStrike rolls = allPins $ scoreRolls 1 rolls

isSpare :: [Int] -> Bool
isSpare rolls = allPins $ scoreRolls 2 rolls

scoreStrike :: ([Int], [Int]) -> ([Int], [Int])
scoreStrike (frameScores, rolls) =
  (frameScores ++ [scoreRolls 3 rolls], moveFrame 1 rolls)

scoreSpare :: ([Int], [Int]) -> ([Int], [Int])
scoreSpare (frameScores, rolls) =
  (frameScores ++ [scoreRolls 3 rolls], moveFrame 2 rolls)

scoreNormal :: ([Int], [Int]) -> ([Int], [Int])
scoreNormal (frameScores, rolls) =
  (frameScores ++ [scoreRolls 2 rolls], moveFrame 2 rolls)

scoreRolls :: Int -> [Int] -> Int
scoreRolls numRolls rolls = sum $ take numRolls rolls

moveFrame :: Int -> [Int] -> [Int]
moveFrame numRolls rolls = drop numRolls rolls

allPins :: Int -> Bool
allPins = (== 10)

We can see here that there’s a lot of confusing looking type signatures. Although this could be greatly improved by using explicit types rather than Int everywhere, there is still a lot of noise from having to manually thread the state through everywhere.

Refactoring to Use State

Now lets take a look at a version which makes use of State:

import Control.Monad (mapM)
import Control.Monad.State (State, get, put, modify, evalState)
import Control.Conditional (condM, otherwiseM)

score :: [Int] -> Int
score rolls = sum $ evalState (mapM scoreFrame [1..10]) rolls

scoreFrame :: Int -> State [Int] Int
scoreFrame _frame =
  condM [ (isStrike,   scoreStrike)
        , (isSpare,    scoreSpare)
        , (otherwiseM, scoreNormal)
        ]

isStrike :: State [Int] Bool
isStrike = allPins <$> scoreRolls 1

isSpare :: State [Int] Bool
isSpare = allPins <$> scoreRolls 2

scoreStrike :: State [Int] Int
scoreStrike = do
  score <- scoreRolls 3
  moveFrame 1
  return score

scoreSpare :: State [Int] Int
scoreSpare = do
  score <- scoreRolls 3
  moveFrame 2
  return score

scoreNormal :: State [Int] Int
scoreNormal = do
  score <- scoreRolls 2
  moveFrame 2
  return score

scoreRolls :: Int -> State [Int] Int
scoreRolls numRolls = sum . take numRolls <$> get

moveFrame :: Int -> State [Int] ()
moveFrame numRolls = modify (drop numRolls)

allPins :: Int -> Bool
allPins = (== 10)

While there are a few bits in this version which require some understanding, it does remove a lot of the clutter from the previous version. It also makes a clear separation between the state and the frame number itself (which incidentally is never actually used).

Final Worlds

I don’t claim that using mapM instead of foldl is always clearer, and I don’t claim that either of the implementations of the Bowling Game scoring exercise above is the best way to solve this in Haskell, but I do hope you have enjoyed this article and perhaps learned something new along the way.

Haskell — Mapping with State was originally published in Cloudnative.ly on Medium, where people are continuing the conversation by highlighting and responding to this story.