In this post, we will look at what application flow is and how it can be defined. We will then discuss where this logic could live by comparing choreography and orchestration.

What do we mean by application flow?

Application flow is the flow of execution through your system. It describes how some input events progress through the system, going through a number of microservices in the correct order.

Firstly we need a way to determine from an event where it is in the flow, essentially why it has been produced. Each event has a type, but this might not contain enough information. For example, if an event fails and we want to retry this event by firing it back through the system, but we only want to retry it 3 times. How can we determine from the type alone how many times it has been retried, as this type will be the same every retry?

In order to describe where an event is in the flow, a string can be added to an event’s data. We will call this string identifying an event’s position in the flow a ‘flow node’. Now accurately knowing where an event has come from allows us to determine where an event should go next. We can think of the flow through the system as a graph of these flow nodes.

Thinking of the system flow as a graph will help us think about the ways in which we can define it.

Where should the flow logic live?

Given the above, we can see that there is a reasonable amount of logic around handling application flow. There are two ways in which we can control the flow in the system, choreography and orchestration:

• Choreography spreads this flow logic throughout the system, giving control to each microservice
• Orchestration puts this logic in one place—centralising control of the flow logic

Below we will look at both solutions in more detail.

Choreography

This is likely the first one that came to mind if we think of this from a microservices point of view. We have microservices and they output their events to any microservice which subscribes to those events. Together, all the microservices naturally end up defining the flow throughout the system.

Using this solution (along with the event bus) gives us adequate decoupling between the microservices. As the publish/subscribe nature of an event bus provides a layer between microservices where they don’t directly know about each other. This results in a decentralised solution that fits nicely with microservices.

The only issue here is that if you have fairly complex flow logic that is used over a number of microservices, the microservices could end up duplicating some of this functionality between them. The logic is also spread throughout the system, which could make reading and understanding this logic difficult.

Orchestration

A centralised solution like orchestration doesn’t have the same issues as choreography. This is due to orchestration containing all flow logic in one place. Every time an output event is created, a centralized piece of logic is used to determine where it should go to next.

Orchestration allows easy readability of the flow of the whole system, as it is not spread throughout the system. And large changes to the flow would still only require one section of the code to be changed.

Using a shared library could work, it would be used by all microservices to determine which output events it should send. So when the output of a microservice is generated, it would pass through this library to determine the specific events which should be output. This library would be run on every microservice.

This does solve the problem of similar logic being spread throughout the system but means each microservice relies on a new library which:

• Causes rebuilds to every microservice when this library changes (any time the flow changes)
• Makes it difficult to be language-agnostic, as each language used would need to somehow use this library

How would this look if we contained all the logic in one microservice? Outputs from other microservices would always go to this orchestration microservice, which would then determine where to go next. This would help fix these negatives from using a shared library.

A downside to using an orchestration microservice is that each step of the flow would require an extra event to go through this microservice; potentially doubling the number of events. As this architecture can scale easily this slight performance hit may not be a worry, but it is something to keep in mind. This also adds a single point of failure because if the orchestration microservice goes down the flow of events throughout the system stops.

Pros and cons of each approach

Below is a table showing the advantages and disadvantages of each approach discussed:

With this being said, a hybrid approach of orchestration and choreography can be used as there is nothing forcing us to pick just one of the two. With this hybrid approach, you could have localised orchestration, where only subsections of the flow which are complex could be orchestrated. Other things in the system which don’t fit nicely into the application flow could be choreographed, for example, if we want to consume every event in the system for logging purposes. This gives us flexibility to choose the right solution for different scenarios.

TL;DR

Think of how the flow through your system should be defined; microservices can be choreographed together to naturally define the flow or they could be orchestrated by some microservice or library.

If you found this interesting you can read more on event-based microservices here!

Event-based Microservices: Application Flow was originally published in UserTesting Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you‘re not familiar with the basics of an event-based microservice architecture, you should check out this Event-based Microservices: Overview post.

What’s the problem?

When a system is distributed (like a microservice-based system) along with the advantages comes the potential for errors. Examples include: connectivity issues, serialization/deserialization issues, downstream system outages, peer system outages, bugs, and the list goes on. The handling of these errors in a simple and repeatable way is important in the success of a project.

This sounds like a daunting problem, but the key to the solution is to simplify and break down the complexity. We can break down the errors into two types, transient and non-transient errors. Examples of these are listed below.

Transient errors:

• Connectivity issues
• System outages

Non-transient errors:

• Serialization/deserialization issues
• Bugs

These two types of errors can be handled in repeatable ways. Transient errors can be handled by retrying, as these should fix themselves given time. Non-transient errors should be handled by alerting and notifying a responsible team to fix the issue, as these require human intervention. We will look at how to handle both types of errors below.

Dead Letter Events

We will start by looking at non-transient errors. What should the system do when something unexpected and catastrophic happens?

As distributed systems handle a number of events simultaneously, the last thing we want to do is block the pipeline of data coming through our system. But we also don’t want to lose this unprocessable data. So once this happens we mark the error event as processed, pull it out of the pipeline, and bump it to the side in the form of a dead letter event. The dead letter event contains all the information of the input event so that no data is lost, and it can be reprocessed at a later point.

Once we have done this, the disaster has been averted. Normal data is still flowing and we have not lost any information. From this point, our goal is to notify someone and point them towards the dead letter event. As we have these dead letters in the event bus, this can be done by using a monitoring tool relevant to the message bus technology being used. All the dead letter queues should alert whenever they receive events. From this point, the cause of the error can be fixed and the relevant dead letter events can then be manually flushed back into the system—picking up where they left off.

Retry Events

What should the system do if the cause of an error will fix itself given time? In the case of transient errors, we should handle them using retry events.

The process for retry events starts off the same as dead letter events. We have a flow of data and when an event errors, we again mark it as processed and bump it to the side, maintaining the flow of valid events.

This is where the difference comes in, the error data is put into retry events which will automatically be reprocessed after a given amount of time. This amount of time may vary depending on the type of error and the number of times it has failed before. As a safety precaution once a retry event has failed too many times, it should be turned into a dead letter event to notify someone, as this is not as transient as expected.

Expected Failures

Something to note—for simplicity—is to separate errors (as described above) from expected failures. Expected failures are failures that are allowed as part of the business logic or engineering logic. These are handled by the application-level logic and should have bespoke events which cause the correct downstream actions, no retries or dead letters should be created.

An example of an expected failure could be in a banking system. Like if a user wants to pay for something but doesn’t have enough funds in their account. This should result in a transaction failure, which should be handled by the application logic, potentially resulting in a payment declined notification to relevant users.

The similar terminology can easily become confusing and is why clearly separating expected errors and unexpected errors are so important and should be called out.

Implementation with Kafka

If we look at Kafka as a way to implement handling retries or dead letters there are two simple solutions. For both of these approaches the dead letter events and retry events will be stored in Kafka topics. A topic for dead letters, and multiple topics for each retry delay duration. The first solution is having the error logic in all microservices and the second is handling the errors in new specific retry and dead letter microservices. We will look at the advantages and disadvantages of both these approaches.

The first approach works great, although it comes with a complex implementation. If we can have multiple retries with varying delays, we will need multiple queues; one queue for each retry length. This is due to Kafka queues being ordered—forcing events to be processed in order. Along with the retry logic, we need additional logic to be able to flush dead letter queues on demand. Managing these queues requires logic in each individual microservice, increasing the complexity throughout the system.

The second approach works in the same way, although it removes the retry complexity from all microservices, and places it in a single retry microservice. The retry microservice’s job is to track and action all retries. This microservice receives an event, writing it to its own topics with both the event to retry and the timestamp to retry that event. It then pushes out these retry events once their timestamp has been reached. We can also have a single dead letter service that handles flushing dead letter queues on request.

On top of these two approaches, we also have the option of using a database, instead of Kafka topics, to store the retries. This simplifies the retry implementation (as we don’t have duplicated topics for different retry durations) and allows for more complex retry duration logic (we aren’t bound by creating a topic for each delay duration). This comes with two limitations. If the retry database is down, retries cannot be actioned for that time. And it does not support exactly once delivery of events, as there is potential event duplication through the interface with the database not being transactional.

Here is a table of the advantages and disadvantages of each listed out:

TL;DR

Distributed systems require seamless error handling to work well; a simple solution is to treat transient errors by retrying and non-transient errors by alerting.

If you found this interesting you can read more on event-based microservices here!

Event-based Microservices: Error Handling was originally published in UserTesting Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

A message bus (or event bus) solves the problem of how microservices communicate with each other. This sounds like a problem that could be solved using simple direct connections, but there are challenges that would come along with that solution. Some examples of these challenges include handling horizontal scaling, network disconnects, and service discovery.

To learn more about where a message bus fits into a microservice architecture, you can view the Event-based Microservices: Overview post.

In short, the event bus is the hub that holds the system together.

Event Structure

First of all, we need a standard way to define microservice events. Events are blobs of data that can be structured in any serialisable data format. The event needs to contain some information on what type of event it is, and the data required to be processed. Below is an example event using JSON.

https://gist.github.com/GVaughanUT/03b9d75fa19e442ec059de12be84858e#file-example_event-json

Message Bus Connection

The event bus uses a publish-subscribe pattern. It allows consumers to subscribe to messages of certain types and allows producers to create messages of a certain type. The bus then connects these producers and consumers together.

The following diagram shows an example of a message bus connecting two services. With red events being consumed by the left microservice and green events being produced. These green events are in turn consumed by the right microservice which then produces yellow events.

One thing you might have noticed is where the initial red events come from, how do they get into the system? Microservices don’t need to always consume input events from the message bus. In fact, input events can be consumed from external sources (database or web app, etc.). And in a similar way, not all microservices need to produce an output to the message bus.

Event Routing

A key feature of the message bus is that it handles message routing. This could be as simple as the top example where the green output is produced by one microservice and only consumed by one other microservice. But we can have as many microservices as we want consuming the same green message. A microservice can also consume or produce as many different types of events as it wants. Below is an example of more complex event routing.

This concept of event routing is where the message bus shows how powerful it can really be. As it puts the responsibility of subscribing to data on the service requesting it, meaning that the overall system can remain decoupled and modular. New sections of the system can be added, moved around, and removed relatively easily; resulting in a flexible solution.

Kafka

The event bus needs to handle more scenarios like service discovery, network disconnects, and multiple instances of the same microservice—quickly becoming a complex problem. Kafka is a technology that solves these problems, allowing a reliable message bus implementation.

Kafka is a distributed queuing system, which allows publishing and subscribing to queues. In a microservice architecture, microservices would subscribe to queues for their input events, and publish to queues with their output events.

You can read more about Kafka from its documentation here.

TL;DR

A message bus facilitates events being routed between microservices; without having tight coupling between microservices.

If you found this interesting you can read more on event-based microservices here!

Event-based Microservices: Message Bus was originally published in UserTesting Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

What are microservices?

A microservice is something that takes input and gives output. It can almost be thought of as a function. The only difference here is that this function can be run anywhere you want and can be replicated as many times as you want. It can be run on multiple machines, being horizontally scaled to handle more calls and so more data.

Why use them?

The reason a microservice architecture is valued is not solely because of this ability to handle vast amounts of data. It is also this concept that each service is strongly decoupled from each other. They are just black boxes from each other’s eyes.

The decoupled nature comes with a few benefits. Firstly it helps promote a simpler engineering environment. They can be independently edited, as long as the interface (the input and output) remains intact. If changes are made to the outputs a developer, additionally, just needs to understand how changes made affect areas downstream in the application flow. It also results in a more explicit flow throughout the system reducing the likelihood of spaghetti junctions in code.

It allows for focused testing, as each microservice can be tested in isolation. Ensuring that a microservice acts as expected given some inputs. You can then also contract test against the microservices it interacts with, to ensure that the inputs it expects are the inputs it will receive from another service. Beyond this you can test the system as a whole, but with more confidence in the building blocks which make up the system.

A microservice architecture is also un-opinionated. Because they are so decoupled, each service doesn’t even need to use the same language or libraries as each other. If a specific circumstance means another language or library would be more productive, you can use it. You are minimally tied into past technical decisions. One thing to note is that some code may be shared between services in shared libraries, which may affect language decisions.

In summary, the benefits of microservices are:

• Scalable performance
• Simple development
• Robust testing
• Flexible & Language agnostic

There are some complexities to be mindful of when using microservices, for example:

• Complex DevOps setup: There are a lot of services being deployed and run
• Distribution complexities: Service discovery & network disconnects between different services need to be handled

How do they communicate?

We have an understanding of what a microservice is, but how do they work together? If microservices can be run on different machines, how do they communicate?

A simple example of a communication framework is microservices communicating directly via REST endpoints. This allows microservices to communicate over a network, although it has a couple of drawbacks, one of which is slight coupling between services. The coupling comes from microservices needing to know about the microservice they are outputting to. Other drawbacks include; difficulty handling network disconnects and performing service discovery.

We will look at the alternative approach of using event-based communication, which keeps the benefit of loose coupling. In this approach, a microservice receives an input event, processes it, and returns any number of output events. An event bus is then used to transport these output events between microservices.

There is no direct line of communication between microservices to send events. Instead, the event bus takes in output events and provides them to any microservice which takes those events as inputs. Each service knows about the event bus and can subscribe to a selection of events, and produce new events back to the event bus.

Can they persist data?

The event bus comes with another benefit of persisting events, which ensures that even if a microservice is down or disconnected, it will still eventually receive its events. This frees us from having to worry about persisting events. But what if data needs to be persisted outside of an event’s lifetime?

Like normal systems, microservices can also have access to a database to allow saving or loading of data that you want to persist. The main thing to remember when using databases in a microservice architecture is that a database should serve just one microservice. As a microservice touching another microservice’s data breaks proper encapsulation. In the case where data is needed in another microservice, the data should be accessed by the owning microservice, and then sent via an event.

Microservice Guidelines

These are some guidelines on how to view microservices:

• Decoupled: A microservice should not know about other microservices, any shared code should be provided via a shared library.
• Encapsulated: Do not directly access another microservice’s database, think of this in the same way as why a class’s member variables should be private.
• Single Responsibility: Each microservice should do conceptually one job, and one job only. There is a skill in deciding what a service should own and when to break out a new service.
• SSOT: The event bus is the single source of truth. If data is contained in an event it should be the most up to date information, this event is the source of truth.

TL;DR

Microservices are like functions, which can run on a cluster of machines; and in event-based systems they communicate by sending messages through a message bus.

If you found this interesting you can read more on event-based microservices here!

Event-based Microservices: Overview was originally published in UserTesting Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.