<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Matthew Lothian on Medium]]></title>
        <description><![CDATA[Stories by Matthew Lothian on Medium]]></description>
        <link>https://medium.com/@matthewlothian?source=rss-9c6c8c03794c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*N58wXnTngoJOHEnkCgd7OQ.jpeg</url>
            <title>Stories by Matthew Lothian on Medium</title>
            <link>https://medium.com/@matthewlothian?source=rss-9c6c8c03794c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 14:29:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@matthewlothian/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Microservice Problem]]></title>
            <link>https://medium.com/@matthewlothian/the-microservice-problem-d3d21dd10ec8?source=rss-9c6c8c03794c------2</link>
            <guid isPermaLink="false">https://medium.com/p/d3d21dd10ec8</guid>
            <category><![CDATA[microservice-architecture]]></category>
            <category><![CDATA[delivery-management]]></category>
            <category><![CDATA[scalability]]></category>
            <category><![CDATA[monolith]]></category>
            <category><![CDATA[microservices]]></category>
            <dc:creator><![CDATA[Matthew Lothian]]></dc:creator>
            <pubDate>Sun, 06 Jun 2021 16:15:15 GMT</pubDate>
            <atom:updated>2021-06-06T16:15:34.168Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JfLHGm0Nmjl8hecnjM_9-A.png" /><figcaption>Aligning boundaries matters (Bad Example)</figcaption></figure><blockquote>Microservices are a great problem to have. You must be a really successful organisation to need the extreme investment and scale. right?</blockquote><p>These days we all know the term “Microservices” and we could probably confidently tell someone what it means, but I&#39;m not so convinced we get it right, and often we misunderstand and misinform others unintentionally.</p><p>As a pessimistic and inquisitive software engineer, I often like to question why we do things and how we got to where we are in an effort to better understand the tools and trends of the trade. So let&#39;s apply this to “Microservices”.</p><h3>What’s a Microservice?</h3><p>Let us start with a definition of what Microservices are, and from there explore why each of these principles exists and where they came from. These principles can seem simple enough in theory, but in practice can introduce significant accidental complexity.</p><p><strong>Decompose a solution into <em>multiple</em> processes where each is:</strong></p><ol><li><strong>Small</strong></li><li><strong>A single responsibility</strong></li><li><strong>Independently deployable</strong></li><li><strong>Autonomous</strong></li><li><strong>Loosely coupled</strong></li><li><strong>Aligned to a bounded context</strong></li></ol><p>While microservices are a distributed service orientated architecture the principles apply to the singular decomposed components of a solution. This is because:</p><blockquote>Microservices are about<strong> enabling</strong> and <strong>isolating change </strong>by decomposing a solution around existing context boundaries within a business domain, and are a function of <strong>Domain Complexity</strong>, <strong>Concurrent Delivery</strong>, and lastly <strong>Targeted Horisontal Scaling</strong>.</blockquote><p>You might even argue that microservices are a delivery pattern rather than an architecture pattern, as the only real design guidance on how to decompose a solution is to align to a bounded context, and I am not going to try to cover Domain Driven Design in this article at the same time, but you should definitely do this yourself.</p><p>Let&#39;s take a trip back to a time before microservices.</p><h3>In defence of the monolith</h3><p>In the beginning… single process applications were very practical and server infrastructure was a constraining factor, having <strong><em>one</em></strong> thing to build, deploy and execute was, and still is nice. Naturally, there was also one source repository, which is not really a problem in itself but <em>can</em> get out of hand as a project scales in complexity and delivery. This is what has become known as the <strong><em>monolithic</em></strong> application architecture.</p><p>The omniscient and omnipresent simplicity of a monolith is very appealing for good reason, even in the age of infrastructure on demand.</p><blockquote>The primary goal of any architecture should be to facilitate and enable teams to quickly deliver value and adapt to change.</blockquote><p>With a monolith, you can focus your initial energy on understanding the solution space of a domain, and less on the technical complexity of a distributed system, and infrastructure requirements.</p><p>The failure of a monolith is human, It&#39;s just too convenient to break modular boundaries and write coupled code when the method you need is just there in that file in the same project. This is not hard to tame initially but as a solution scales, you will inevitably lose this battle without introducing bespoke structural linting or structural boundaries that intentionally isolate modules and contexts within the same code base.</p><p>In fact, you should definitely have isolated modules and contexts within your codebase, monolith or not. The <a href="http://www.kamilgrzybek.com/design/modular-monolith-primer/"><strong>Modular Monolith</strong></a><strong> </strong>pattern<strong> </strong>describes this quite well, but in a lot of ways is just a rebrand of <a href="https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html">OOP / Port and Adapters / Clean / Onion application architecture</a> at the Monolith scale.</p><h4>When a Monolith is not enough</h4><p>You can get very far with a good Monolith and you can be very happy, I always recommend starting with a monolithic architecture, because it allows an organisation to focus on solving business problems rather than technical problems. If this is a struggle with a monolith it will be exponentially more difficult if you attempt to distribute it, and you will probably get the design wrong initially anyway, software modelling is hard.</p><p>When you have a working monolith with all your production operation bells and whistles, sit back and observe. Watch for what parts of the applications change together, looks for hot paths through the application, looks for pain points and conflicts in the delivery of new features. Those are areas that are candidates for decomposition, and more often than not will naturally align to the domain boundaries of your organisation and can be isolated into their own context. Just remember to decompose behaviour over data.</p><p>Alternatively, you may find that your single application is actually 2 or more internal products with their own stakeholder&#39;s goals and delivery timelines, and you’re all getting in each other way. That a good sign that you have discovered subdomains and can begin to carve out context boundaries between them.</p><p>You might be thinking why can’t I just start with microservices… well mostly because:</p><blockquote>it’s <strong><em>really hard</em></strong> to discover subdomains, context boundaries and get the decomposition of microservices right, and getting it wrong will ultimately stifle your delivery and you’ll end up fighting your own architecture.</blockquote><h3>The Service Oriented Architecture</h3><p>With the prevalence of on-demand server infrastructure and the perceived failure and frustration felt for monolithic application architecture SOA gained a lot of popularity. So how it different?</p><blockquote>Applying service-orientation results in units of software partitioned into discrete, autonomous, and network-accessible units, each designed to solve an individual concern. These units qualify as services</blockquote><p>The <a href="http://www.soa-manifesto.org/">SOA manifesto</a> gives us the following definition:</p><blockquote><strong>Business value</strong> over technical strategy<br><strong>Strategic goals</strong> over project-specific benefits<br><strong>Intrinsic interoperability</strong> over custom integration<br><strong>Shared services</strong> over specific-purpose implementations<br><strong>Flexibility</strong> over optimization<br><strong>Evolutionary refinement</strong> over pursuit of initial perfection</blockquote><p>Remember that human flaw of convenience that affected our monolithic code base? Well, we are putting in some hard boundaries in the form of moving all that code to a different repo and running it as its own process in a different service behind a firewall, try coupling that!</p><p>Only we still need to trigger some behaviour and access data that is hidden behind the interface of that new service. So we use some out of process protocol to connect them through a network.</p><p>Great, we now have controlled coupling via a service interface, but we also have 10x processes to implement with their own network interfaces, build, deployment and execution processes. That’s a lot of additional infrastructure and process.</p><p>And while these services can now change their implementation independently they can also be deployed independently and must aim to not break their public interface. You can solve this with service versioning and running multiple versions to avoid forcing breaking changes into consumers, but this is just more complexity. This is lots of fun to maintain in production without downtime or data consistency issues.</p><p>The SOA movement has led to many tools, practices, and platforms we use today with distributed architectures (ESB, AMQP, RabbitMQ, HATEOAS, Sagas…) that tend to favour dumb services connected by smart pipelines, Which is the exact opposite goal of microservices.</p><h3>Decoding the Microservice Pattern</h3><p>Microservices are a SOA but they differ in their intention in a few ways.</p><p>Martin Fowler does a great job of explaining the <a href="https://martinfowler.com/articles/microservices.html">characteristics of microservices </a>but I will summarise them here for you, and combine it with our principle from above to come up with our own manifesto.</p><h4>Microservice Manifesto</h4><p><strong><em>Left</em></strong><em> over </em>right</p><p>While the right is valuable and often useful, the left optimises for the successful decomposition of a solution and the delivery of a microservice architecture.</p><ol><li><strong>Decompose business capability</strong> <em>over</em> business function</li><li><strong>Gather product</strong> <strong>teams</strong> <em>over</em> project teams</li><li><strong>Enable delivery</strong> <em>over</em> enforcing standardisation</li><li><strong>Independence </strong><em>over</em> dependency</li><li><strong>Services</strong> <strong>own</strong> <strong>data</strong> <em>over</em> services<strong> </strong>use data</li><li><strong>Automation</strong> <em>over</em> manual process</li><li><strong>Availability</strong> <strong>and</strong> p<strong>artitioning</strong> <em>over</em> consistency</li><li><strong>Flexibility</strong> <em>over</em> optimization</li><li><strong>Evolutionary refinement</strong> <em>over</em> pursuit of initial perfection</li></ol><p>If we’re honest most “Microservices” are not this and are barely even an SOA, but rather the dreaded <em>Distributed Monolith</em>.</p><p>Let’s explore each of these about deeper.</p><h4><strong>Decompose business capability</strong> <em>over</em> business function</h4><p>Decomposition is at the core of microservices, you will want to decompose your solution vertically by capability. Aim to understand the journey that users of your product(s) will take and identify the intentions, business rules, data, and events involved. <a href="https://eventmodeling.org/">Event Modeling</a> can be a great tool for helping discover these.</p><p>Once you understand the domain(s) identify the context boundaries. A good rule for doing this is that a single action taken by a user will try not to cross a context boundary, but rather will contain some representation of data from another context that is native to the current context. Doing so will isolate the implementation of a context from changes in other contexts.</p><p>Avoid grouping similar functionality into a generic shared context, this will become a hot spot for conflicting requirements ownership and implementation.</p><p>A microservice should always have its own requirements and delivery schedule, and not inherit them from one or more consumers, this would be ok in an SOA but tends to lead to a distributed monolith, as the consuming components are likely not independent or autonomous</p><h4>Gather product teams over project teams</h4><p>Align your teams to business products and build a cross-functional team that has all the abilities to implement and deliver a solution end to end, reduce inter-team dependency as much as possible. This will generally vertically align your teams to business capability over business function.</p><blockquote>Conway’s law: Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure.</blockquote><p>This might involve reorganising your business internally. If you have horizontally aligned teams (UI devs, Backend devs, Ops, DBAs, BAs) that work independently, you will not be able to succeed with microservices.</p><h4><strong>Enable delivery</strong> <em>over</em> enforcing standardisation</h4><p>Embrace enabling your teams to deliver services using technology that is fit for the problem. This does not mean free reign to use whatever, but don’t try to enforce a standard set of tools and process onto every solution, it may actively hinder the delivery.</p><p>Instead, make it easier for your teams to build and reuse standard tools by automating and improving the developer experience with the tech and tools you want them to use.</p><h4><strong>Independence <em>over</em> dependency</strong></h4><p>Aim to design a service that is entirely independent and autonomous at build and run time. Reduce run time dependency on other contexts and services by embracing eventual consistency and projecting any data from external context into the services own private data source.</p><p>If you find that to deliver a new feature you have to change multiple repos and deploy multiple services, you may have got step 1 wrong. Things that change together should probably be in the same context or microservice.</p><p>Avoid the shared kernel, prefer to implement explicit and purposeful code try not to depend on a common lib of functions or common models. This creates a build time dependency and exposes shared implementation details breaking encapsulation and making it harder to adapt to change. Duplicate code if you need to.</p><h4><strong>Services</strong> <strong>own</strong> <strong>data</strong> <em>over</em> services<strong> </strong>use data</h4><p>Covered in part above, a service should own all the data it needs to fulfil its purpose, which should be aligned to a business capability. A microservice almost always has its own private data store that encapsulates its state.</p><p>A microservice is responsible for everything in the datastore and the only way to access or update that data is via the public interface of the service. This includes projected state from other contexts, a public consumer interface will listen for changes in other contexts via some mechanism or protocol.</p><p>That makes microservices independently testable without runtime dependency as all state is captured in its private data store.</p><h4><strong>Automation</strong> <em>over</em> manual process</h4><p>The first thing I recommend the team do when starting work on a new product is: deploy a blank page or a service that has only a version endpoint. Obviously, this is simple to implement from a coding perspective, but it forces your team to tackle the often overlooked steps on the path to production. If your organisation does not have a mature dev ops practice you will find that doing anything with microservices is hard.</p><p>Distributed systems are full of hidden complexity, every new service needs its own build and deployment pipeline, and you need to be able to efficiently stitch them all together once deployed to an environment, with TLS/mTLS, clustering and autoscaling, rolling deployment, multiple versions of the same service, network routing and firewalls, secret management, distributed authentication and authorisation, distributed tracing, logging and monitoring, automated testing, infrastructure audit logging.</p><p>And don’t forget data recovery failover.</p><p>If any of these take more than a day or two you’ll be betting on a lame horse.</p><ul><li>Getting a domain name</li><li>Acquiring a TLS certificate</li><li>Provisioning a database or schema</li><li>Changing a database schema</li><li>Setting up a new CI/CD pipeline</li><li>Network changes</li><li>Setting up new environments</li></ul><p>Microservices embrace and encourage purpose-built services and change, and you are going to be repeating things a lot, invest heavily in automating these processes and get process funnels and blockers out of the way ASAP.</p><h4><strong>Availability</strong> <strong>and</strong> p<strong>artitioning</strong> <em>over</em> consistency</h4><p>This is a byproduct of the independence and autonomous nature of a microservice, every context that is projected into a microservice is going to be eventually consistent, and every context that is a run time dependency might fail.</p><p>Failure is something that you must handle gracefully where you need to, with alternate or partial responses, or outright errors depending on your business continuality plan.</p><p>If your service can’t run without one of it dependencies then go back to step 1.</p><h4><strong>Flexibility</strong> <em>over</em> optimization</h4><p>It is very tempting to optimize while your building, but resist the urge and aim for something that is easy to change, adapt and maintain. The cost to refactor premature optimisation can often be much higher than the cost to run unoptimised code. Defer optimization as much as possible.</p><blockquote>Premature optimisation is the root of all evil.</blockquote><p>This often manifests as A and B sharing logic or a common service that initially worked for both; as A and B were similar. But A and B are likely not the same for a reason, they can change independently and suddenly that shared logic or service has branching (or worse conflicting) logic for “if A do X and if B do Y”.</p><p>Only optimise your code or your services after you have hard evidence that:</p><ul><li>It&#39;s not meeting some target SLA or performance goal</li><li>It&#39;s more effort to maintain <em>n</em> things over one</li></ul><h4><strong>Evolutionary refinement</strong> <em>over</em> pursuit of initial perfection</h4><p>You won’t get it right, change will be required, that’s ok, it’s part of building better software. Prefer measurable action over calculated decisions, small increments are your friend. CI, Agile, and Scrum won’t help you if you don’t know how to decompose a problem into small incremental improvements. Long-running branches or changes that don’t get deployed accrue <strong><em>feedback dept</em></strong>, which can be costly to address once finally reviewed.</p><ol><li>Decompose your problem into small incremental solutions</li><li>Implement increment</li><li>Review</li><li>Test</li><li>Deploy</li><li>Gather feedback</li><li>Integrate feedback into step 1</li></ol><h3>Closing thoughts</h3><p>Microservices are an opinionated pattern, an extreme perspective on SOA focused on isolating and enabling change, and the enable horizontal scaling of delivery. The cost of such an architecture is very high and requires a great deal of organisational skill and discipline to achieve the benefits.</p><p>As far as I know, no organisation that has successfully implemented microservices started with microservices as the goal. They build a product or two, often as a monolith. They understood their users and their needs. They scaled with the success of their product and eventually go to a point where decomposing their solution into multiple services made sense. They already had all the evidence and understood their context boundaries.</p><p>For a great example of this see the talk from GitHub below.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FSr0OE-AzAJU&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DSr0OE-AzAJU&amp;image=http%3A%2F%2Fi.ytimg.com%2Fvi%2FSr0OE-AzAJU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/9e6a56a22c9a2b142cb1eb706629ff96/href">https://medium.com/media/9e6a56a22c9a2b142cb1eb706629ff96/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d3d21dd10ec8" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>