If deploying a new version of a service requires deploying new versions of other services at the same time, you’re probably doing microservices wrong.

Let's say that to ship a feature, we need to deploy service A from v1.0 to v1.1, which requires updating service B from v1.5 to v1.6, which requires service C to be updated from v2.1 to v2.2. In scenarios like these, you’re forced to perform lock-step deployments.

A lock-step deployment is a deployment of multiple service versions at once, with careful coordination, usually due to breaking changes between their interactions, or due to functionalities scattered throughout many services. You might’ve heard of cases where teams assemble in a “war room” once a week to collectively ship their changes to production; that’s lock-step deployments gone really bad.

Lock-step deployments completely mess up:

• Independent deployability. Services should be loosely coupled as such that you can deploy them independently, without having to deploy other services along with them.
• Deployment frequency. Healthy agile teams ship frequently, have shorter feedback loops, and iterate fast.

Here are some common causes of lock-step deployments.

Backward compatibility: Services you depend on

Here’s a scenario: at a Food delivery company, your team manages the Delivery Routing service and plans to deploy version 1.1, which includes an optimization of delivery assignments based on driver availability and location.

However, this improvement relies on a new upcoming feature promised by the Delivery Driver service in version v1.8, which adds the new availability and location fields to the driver API response.

Should you wait for the Delivery Driver team to deploy their new version before you deploy yours? Perhaps, but it’s definitely not ideal, for a number of reasons:

• You’ll have to block your workflow and refrain from merging your work if the other team lacks the capacity to deploy at the moment. This slows you down.
• What if they deploy, but then roll back their version due to an unexpected failure? Now your service will also break unless you roll back too.

The solution here is to deploy a backward-compatible version of your service that works with both the current and upcoming versions of the dependency service. Going back to the example: you can implement your Delivery Routing service in such a way that makes the optimized assignment algorithm come into play only if it notices that the Delivery Driver response includes the new fields, implying that its new version made it to prod. If it doesn’t, it just falls back to the unoptimized version.

Feature flags are another effective approach, enabling you to control the release of this feature. Wrap the code that depends on the new version of the dependency service in a feature flag, and activate it when you confirm the successful rollout of the required version. Keep in mind that if the dependency service rolls back to an incompatible version, you’ll need to deactivate the flag accordingly.

Backward compatibility: Services that depend on you

When deploying a new version of your service, always keep in mind that consumer services already depend on your exposed API contract, so you shouldn’t make breaking changes to the API.

Keep your changes additive. Think twice before removing or even modifying an API response, because you’ll likely be breaking something downstream.

Deprecate features properly if you really want to phase out a resource that you’ve already exposed. Allow teams to gradually move over to the new preferred way, and have patience. A REST API deprecation example: exposing your new API with breaking changes under a new version URL, e.g. /v3/.

Always think of your service from the perspective of the consumer, even if your team is responsible for both the producer and the consumer. This way of thinking really gets you into the proper microservices mindset, allowing you to reap its benefits and avoid distributed monoliths. This has worked really well for me.

Functionality scattered throughout services

Arguably the most common reason: you have to deploy in lock-step because your feature has its functionality scattered throughout many services. This indicates that your service responsibilities are not properly delineated.

I’ve found that a common symptom of this is a lack of clear service ownership. Teams tend to be naturally separated by business domain, therefore services that reflect these boundaries tend to work in harmony. You’ve heard the saying: microservices don’t solve technical problems, they solve organizational problems. They work best when they have proper API & data boundaries, and that’s more easily achievable when they’re owned by teams. That’s because mere ownership of a service introduces a healthy amount of friction because teams must coordinate with each other to achieve their goals, and that slows them down. In time, these pain points will reveal themselves, and with it, a proper separation of services that you can plan and work towards.

I wrote this post after recently reading a Reddit thread that brought up this topic and having run into this problem myself. Hit me up with any other topics you think I should write about!

When I started working with microservices, I took the common rule of “two services must not share a data source” a bit too literally.

I saw it stapled everywhere on the internet: “thou shalt not share a DB between two services”, and it definitely made sense. A service must own its data and retain the freedom to change its schema as it pleases, without changing its external-facing API.

But there’s an important subtlety here that I didn’t understand until much later. To apply this rule properly, we have to distinguish between sharing a data source and sharing data.

Why sharing a data source is bad

An example: the Products service should own the products table, and all the records in it. They expose this data to other teams via an API, a products GraphQL query, and the creation of these records via a createProduct mutation.

The Products service has ownership over the products' source of truth, and no other team should reach into this directly, ever. If they want data out of it, they should ask the Products service for it via the contract (API) they adhere to. Under no circumstance should you allow direct access to the database, or you’ll lose the freedom to make changes to your schema. I learned this the hard way.

Sharing data is OK

The fact is, services need data that belong to other services.

For example, a Trip service will need access to passengers (from Passenger service) and drivers (from Driver service) to serve trip overviews.

Trip service asks each respective service for its data, synchronously, to fulfill the original request (getTrips). We can rest assured that the data is fresh, and the requesting client will get a strongly consistent view of the data (some of you might see where I’m heading at this point ;).

This synchronous request/response model to transmit data between microservices is a very natural mental model for teams that start off in microservices, at least in my experience. You need some data, you know where to get it, you ask the owning service for it, and it gives the data to you, on demand.

On top of that, serving fresh, strongly consistent data was a no-brainer for the teams I was in. Strongly consistent data means up-to-date data, the absolute “freshest” piece of data, straight from the source (of truth). To me, back then, serving anything other than consistent data was unacceptable. How could you serve anything other than up-to-date data? Anything else would be a lie!

We applied these patterns as dogma because we saw no other way, and above all, it felt natural.

Synchronicity and strong consistency don’t scale

Architectures that rely heavily on synchronous requests and strong consistency do not scale well. Sometimes it’s just not feasible, or strictly necessary, to always go straight to the source for your data needs.

The Trips service example above looks neat at first, but it’s rarely the case that systems remain that simple. New services are born, and they’ll require data from existing services. Sticking to the synchronous request pattern will, in time, have you end up with a tangled web of requests between services. Here’s a scenario:

1. A user completed a challenge, runs a completeChallenge mutation to Challenge service
2. After storing the completion, the Challenge service lets the Leaderboard service know, so it can update the leaderboard
3. The Leaderboard service asks User service for user display names and avatars to build the new leaderboard state
4. The Leaderboard service sees that there’s a new leader in the new leaderboard state, and lets the Notification service know so it can notify participants that there’s a new leader!
5. Notification service asks User service for the up-to-date email addresses of the users in that particular leaderboard, so it can send emails out

The User service is clearly a point of contention here: everyone is in one way or another depending on it. Imagine this service being down: it’ll render most of the other services down as well. Not only that, but you’re gonna have to make sure to keep this server buffed up at all times with more replicas and a high-performant DB to keep up with the demand.

On top of that, each hop in this chain of requests adds latency to the entire request. Each hop has the potential to add an exponential amount of latency because every service in the dependency chain could fire off more than one request to its own dependencies. Before you know it, you’ve hit unbearable levels of latency.

Finally, every additional dependency in the request chain increases the likelihood of the entire request chain failing. In a request chain involving five services with an SLA of 99.9% (~9h of yearly downtime), the composite SLA becomes 99.5%. That’s almost 2 days of downtime per year!

We can avoid all of these downsides by asking one question: do services really need up-to-date data?

The Notification service (step 5) arguably does. If a user changed their address and the Notification service doesn’t know about it, it’ll risk sending an email to the wrong address and not getting the notification to the intended user.

The Leaderboard service, on the other hand, probably doesn’t need the up-to-date display names and avatars to build the leaderboard — it’s not that big of a deal if users see stale avatars or display names.

As you can see, services have different data consistency needs. There are trade-offs that we can use as leverage to apply different data-sharing methods and build a more robust distributed system.

Enter Eventual Consistency

It’s at this point in my career that I discovered that services can maintain a copy of other services’ data, locally in their own DB tables. It comes with the responsibility to retain that data via events or polling.

Included in this package is the fact that the data may be stale for some time, but that it’ll eventually be updated, meaning the data is eventually consistent. We can’t guarantee that the data isn’t stale, but we can guarantee that we’ll eventually catch up.

The moment it “clicked” for me was when I thought about it from the perspective of a backend service depending on a public weather API for weather data. Instead of retrieving weather data for Prishtina or Berlin every time a user from those respective cities needs weather data, I cache it (maybe multiple times a day) by materializing it in a local table and serving cached data to those users. I’ve made the trade-off in favor of eventual consistency because it’s not crucial for my users to see the freshest data, it’s OK if it’s a few hours stale.

Going back to the Challenge example: we can cut off many synchronous dependencies to the User service by just maintaining a local copy of users in the services:

• Leaderboard service can maintain a local copy of users and avoid having to make requests to User service. No one really minds if the data is a bit old, it’s not a showstopper if someone sees a slightly old avatar.
• Challenge service can do the same; say if it exposed a getChallengeDetails query and needed user display names and avatars to show the current challenge participants — it can also serve this eventually consistent data from its own materialized users table.
• Notification service, although a bit more sensitive, can also utilize data sharing to remove its dependency on the Users service. It can materialize users locally and maintain a best-effort updated state by listening to User updated events to ensure it has the most up-to-date emails.

Although we didn’t talk much about how services share this data (a topic for another time), a final example architecture would make use of a combination of event sourcing and caching. Here’s a sneak peek into what this kind of architecture would look like:

If you want more examples, take a look at How to Share Data Between Microservices on High Scale by Shiran Metsuyanim, an engineer at Fiverr. It’s a great post that shows how to maintain robustness when adding a new service. It starts out by laying down the constraints, and then discussing the trade-offs between synchronous, asynchronous, and hybrid solutions.

Conclusion

I wanted to get this point across to developers that, like me a few years ago, are stuck in the literal sense of “do not share data” but must realize that this only applies to not sharing the source of truth. Maintaining a copy of a service’s data in another service’s domain is perfectly fine, and embraces the spirit of eventual consistency.

Thanks for reading. What other microservices topics would you like to read about? What isn’t discussed enough? Feel free to reply!

How do you handle an unexpected failure or system state?

Do you handle it gracefully and do a best-effort attempt at returning a response? (fail-safe)

Or, do you stop the failure in its tracks, let it blow up, and sound the alarms? (fail fast)

The general approach that has worked best for me is to always fail fast unless you’re close to the end-user.

Tip 32, The Pragmatic Programmer: Crash Early. A dead program normally does a lot less damage than a crippled one.

The user interface (frontend) is the closest you can be to the end-user. When running into a failure, fatally crashing the UI is the absolute worst response. Any type of response would be better than a fatal crash: serving cached (but stale) data, partial responses, or a retry button. As a last resort, you should be showing a general “Whoops, something broke!” message.

It’s worth putting in the effort in determining the failure state combinations of our frontend to gracefully handle failures and ultimately deliver a good experience. A bad user experience causes churn and uninstalls.

However, in any other situation, failing fast is a vastly superior strategy to make your systems more robust.

Failing visibly makes bugs easier to find

When your service encounters an invalid state, failing fast means failing visibly. After halting execution, make it clearly visible that your system has encountered an invalid state.

Failing visibly makes defects much harder to miss. Even if you have logging in place (which you should have anyway), errors are much more likely to be noticed if all involved parties have it “blow up in their face”.

Easier debugging

Continuing execution after an invalid state makes your system much harder to debug. Instead of knowing exactly where execution stopped, you’ll now have to deal with stepping into your code, reproducing the scenario, and figuring out at what point your program diverged into the state.

Avoid cascading failures

Unless you’ve carefully controlled all possible continuation scenarios, allowing execution by failing safe means you’re effectively allowing your system to enter unknown territory. Unexpected invalid system states lead to more invalid states, and before you know it, they cascade into a much larger failure than if you had stopped it in time.

Less cognitive load and simple mental models

Failing safe means adding branches of possible code paths, resulting in more things to juggle around in your head and, consequently, a greater likelihood of mistakes.

Failing fast means predictable and deliberate programming. You write code confidently when you can rest assured that the system is in the state you expect it to be. By failing fast, you’ve effectively ruled out the possibility of unexpected states, allowing you to work with a simple mental model of your system.

Assertive Programming

There’s a whole software development methodology for fail-fast enthusiasts, called assertive programming, which I first read about in The Pragmatic Programmer.

Assertive programming follows the principle of failing fast by using assertions in the code to continuously validate the system’s state, throwing (crashing) if an assertion’s criteria have not been met.

Here’s an example:

const adult = generateAdult();
assert(adult.age >= 18);
sellItemTo(adult);

We add an assertion after generating an adult as a defensive check to ensure that the system will only ever get to line 3 if the statement adult.age >= 18 holds true. You might think, “but that won’t ever happen”. Well, although this example is an oversimplification, it’s always worth adding assertions to ensure that something that can’t happen, won’t.

Assertive programming is a good practice to validate your assumptions as you write code, especially if it’s rather clever and prone to mistakes:

function someCrazyCalculationThatReturnsAPositiveNumber(num) {
  // do something with num
  ...

  return result;
}

for (let i = 0; i < total; i++) {
  const a = someCrazyCalculationThatReturnsAPositiveNumber(i);
  assert(a > 0); // I don't entirely trust my crazy calc code yet
  doSomethingWith(a);
}

Ultimately, it’s intended to be a tool to increase your confidence as you code and help you build robust, fail-fast systems.

I hope this post shed a bit more light on the many benefits of failing fast, and that you’ll be using this type of approach more often in your solutions. Thanks for reading, and I’d love to hear your thoughts. Feel free to connect with me on Twitter or LinkedIn!

I’ve had my fair share of projects based on microservices architecture. As fun as they are to work on, we fell into some nasty traps, some of which should’ve been painfully obvious. Here are four of them that I could gather from memory.

Services should hide their internal details

A trap I fell into pretty early in my microservices journey was allowing another team’s microservice to access our microservice’s database directly.

Their team was tasked with building a back-office internal tool capable of reading and writing data in all of the other microservices, including ours. Our public GraphQL API wouldn’t suffice, it had very conservative queries and mutations intended only for our public clients. For what this team needed, we would need to add a bunch of administrational operations to our API.

Amid a very tight project schedule and with us thinking it would save us a lot of time, we proceeded with having their service access our service’s DB directly. Yikes.

Not long after that, we realized that we had effectively given up our freedom to make changes to our database’s schema because the back-office tool had now been hard-coupled to it.

Getting things right was painful, but eventually, we got their microservice’s read paths to go through our API and cut off their dependency on our database.

What we learned

Don’t break a microservice’s boundary by allowing another service to directly access one of its components (e.g. the DB). This covertly introduces a new contract that it’ll have to adhere to. A microservice should (aim to) uphold one contract only: its API.

A microservice should hide as many details as possible. You can only easily change components that are hidden from the outside.

Invest early in the infrastructure required for service-to-service communication, i.e. service discovery and machine-to-machine tokens. You’ll need it eventually, and this was a major factor that drove us away from the correct solution because the amount of work seemed significant.

Chatty microservices likely belong together

On what should’ve been a boring Tuesday, some QA reports began storming in that the application was hanging on certain screens. We confirmed that on our observability tool, Instana, where we saw a couple of GraphQL queries taking >30 seconds then ultimately failing.

The distributed trace in Instana revealed an infinite loop of requests as a result of a cyclic dependency between two of our services. We located the bug which was introduced in a recent commit: a new piece of middleware in service A was making a request to service B, which then would make a request to service A, triggering service A’s middleware again to fire off a request to service B, and so on.

We realized we had been lucky to not run into this issue for as long as we did, considering that these two services were already very chatty and had pretty much always been dependent on each other.

As an urgent fix, we reverted the commit and decided to merge the two services into one.

What we learned

Observability tools and distributed tracing save the day. The long alternating sequence of requests in the request trace let us instantly recognize there was a cyclic dependency issue.

Two services being very chatty is a good indicator that they belong together. Most observability tools provide visuals of intra-service communication in your ecosystem that can help you spot which services are communicating excessively.

The best way to avoid cyclic dependency issues is to not have cyclic dependencies. Shallow cyclic dependencies like in our case (A → B → A) are manageable to a certain degree, but a poor design could end up introducing infinite request loops from transitive circular dependencies like A → B → C → A. Aim for no cyclic dependencies and a unidirectional request flow.

Minimize alerting noise. Ideally, an issue like this should have been noticed by us via our monitoring alerts, long before anyone had reported it. We had a Slack channel set up for automated monitoring alerts, but there was too much noise going on in there that we had gotten used to ignoring it. This incident made us take the time to revamp the alerts and be more attentive to the alert channel.

Maintain independent deployability

One of the core principles of microservices is independent deployability. You should be able to deploy a microservice at any time, without having to deploy anything else. Deviating from this rule brings trouble.

In one of my past teams, we had a tradition to gather once a week in a “war room” for mass deployment of a usual set of 6–7 microservices to launch new features. To minimize downtime, these services had to be released in a strictly ordered sequence.

The biggest reason for these coordinated deployments was that teams were allowed to break their API contracts. There were no rules in place to require backward compatibility for every service. We assumed this just came with the territory, accepting it as a normal part of practicing microservices. However, over time the major downsides became very apparent.

If we had messed up one of the services and couldn’t push a hotfix quick, we had to revert all the deployments. This was frustrating.

Allowing microservices to disregard backward compatibility effectively lowered our confidence in the stability of the ecosystem.

Getting many people to sync for the coordinated deployment was difficult since microservice teams usually had their own plan schedules. Also, any hiccup or failure during the process meant a lot of wasted time and productivity.

And, ultimately, it never felt right that teams had to concede their autonomy of deploying their microservices to production.

One of many things we did to address these was to add a rule: microservices must maintain backward compatibility and not break their API contracts. If breaking changes are inevitable, teams must be notified ahead of time and API features must go through a deprecation phase first.

What else we learned

Integrate automated checks in the CI/CD pipeline to report and prohibit breaking changes. We added a step to our pipeline that used static analysis to compare the new API contract to the previous one. Additive changes are OK, but any removal or modification of an API method failed the CI/CD job.

A culture of maintaining backward compatibility did wonders for our sanity and confidence in the system, knowing that you won’t wake up one morning to find out a microservice’s endpoint is now returning a completely new format or even disappeared completely.

Feature flags are a very useful tool to separate code deployment from feature deployment. If you want to deploy a breaking change, you can first hide it behind a feature flag, deploy the code, then switch the feature flag on at your convenience. More on feature flags below.

Adding end-to-end tests has a point of diminishing returns

The above statement admittedly applies to most systems regardless of architecture, but it was especially evident in a microservices project I took part in that had very healthy test coverage. The React frontend was full of integration and Cypress end-to-end tests, whereas the backend services had their own unit and end-to-end tests.

After a certain point, the number of bugs that made it through to staging and production began growing linearly with the number of new features added, even though we had strict measures in place that no code was to get merged without its accompanying tests.

We needed a strategy to complement the tests.

What we did (and learned)

The answer was testing in production, as so nicely explained in Cindy Sridharan’s “Testing in Production, the safe way”. This strategy involves applying post-release techniques to tackle encountered issues, rather than trying to prevent them all pre-release.

What this meant for us, was to crank up our monitoring and work on our knowledge sharing regarding our distributed tracing tools, making sure every developer can find their way around it.

But one of the techniques that proved most effective for us was the usage of feature flags. New feature code was expected to be wrapped in a feature flag, which we would then use to roll the feature out gradually, starting from our small team to the whole company, and finally to the entire userbase (we’ve had great success with LaunchDarkly). We piggybacked on this mechanism for error resolution as well: when our monitoring tools reported critical errors that originated from a particular feature, we could easily switch the feature off to buy us some time to understand the problem and apply a hotfix. This is a great example of how we did testing in production.

Endnotes

Some of the principles and rules I’ve mentioned above are brilliantly explained in the book Building Microservices by Sam Newman, which I’ve had the chance to read recently. It’s a great resource with principles and best practices that will save you from the many pitfalls of microservices. The second edition has been released and I highly recommend it!