<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Joan Zapata on Medium]]></title>
        <description><![CDATA[Stories by Joan Zapata on Medium]]></description>
        <link>https://medium.com/@joanzapata?source=rss-65e0c05c17d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*p418IggKc-QZTfTvGlmYBQ.jpeg</url>
            <title>Stories by Joan Zapata on Medium</title>
            <link>https://medium.com/@joanzapata?source=rss-65e0c05c17d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 01:28:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@joanzapata/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How we implemented a state machine framework]]></title>
            <link>https://medium.com/memobank/how-we-implemented-a-state-machine-framework-d9bd12200f57?source=rss-65e0c05c17d------2</link>
            <guid isPermaLink="false">https://medium.com/p/d9bd12200f57</guid>
            <category><![CDATA[kotlin]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[state-machine]]></category>
            <category><![CDATA[backend]]></category>
            <dc:creator><![CDATA[Joan Zapata]]></dc:creator>
            <pubDate>Tue, 30 Aug 2022 15:41:43 GMT</pubDate>
            <atom:updated>2024-08-19T00:31:05.902Z</atom:updated>
            <content:encoded><![CDATA[<p>As explained in most of our articles: building a banking system requires two properties more than anything, <strong>traceability</strong> and <strong>consistency</strong>. That’s especially true when it comes to the financial part, i.e. anything that makes money move.</p><p>In our core banking system, we mainly have three services involved:</p><ol><li>The <strong>core</strong>, for tracking accounts, balances, debits and credits.</li><li>The <strong>payment-processor</strong>, for connecting ourselves to other banks and actually performing and receiving transfers.</li><li>The <strong>transaction-engine</strong>, which coordinates both, amongst others.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3QYrwmX50TxN0up00g40HQ.jpeg" /></figure><p>The core being the source of truth for the entire system, it requires extreme traceability and consistency, so we used an architecture called “event sourcing”, implemented on the Elixir/Erlang stack, described in a <a href="https://medium.com/memobank/choosing-an-architecture-85750e1e5a03">dedicated article</a>.</p><p>For the other two, we were looking for something more lightweight, that would take into account the high number of interactions with other services, while still maintaining 100% traceability and consistency. To do that, we used a state machine model. For example, in the transaction-engine, the state machine of a transaction, including its interactions with other services, looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uH4ioDsZvBC5_4z9VPGgsA.jpeg" /></figure><p>With regard to our requirements, we paid attention to:</p><ul><li>Having a high level of formalization, in Kotlin;</li><li>Resiliency to all kind of crashes and errors over long periods — transaction can span over weeks — , by having states stored in database;</li><li>Side effects being executed with strong consistency guarantees;</li><li>History data being complete, and easily browsable for traceability.</li></ul><p>The formalization is mostly done through the State interface.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2429dc92fc434c67c07b2feea9255ad2/href">https://medium.com/media/2429dc92fc434c67c07b2feea9255ad2/href</a></iframe><p>It forces developers to declare all states, inputs, and entities as <a href="https://kotlinlang.org/docs/data-classes.html">pure data</a>, and implement onInput as a <a href="https://en.wikipedia.org/wiki/Pure_function">pure function</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/20cd356dd436391293f64921f1c7cf2c/href">https://medium.com/media/20cd356dd436391293f64921f1c7cf2c/href</a></iframe><p>For the same parameters, onInput will always return the same output, and it does not have any side effects. As such, it has two interesting properties: it’s easily testable, and by storing states and inputs we have perfect traceability over what happened.</p><p>To make those state machines long-lived — some transactions can last for months if we take into account all return scenarios — , we store them into a database. The minimum required is to store each state change with a timestamp, associated with the Entity&#39;s reference. As you probably noticed, states are not simple values but data objects, the framework delegates the serialization to the user code, and stores the result as a blob. Then, when a new input comes in, it uses the reference of the Input to rebuild the last state of the associated Entity, and calls the onInput function on it.</p><p>Even though storing states is enough for it to work, we also store all inputs for extra traceability.</p><p>Now, a big part of the job of these state machines is to interact with other services, by sending commands or events. It’s very important that these “side effects” are eventually consistent with transitions, i.e. that we aren’t able to send a command and then fail to commit the state change, or to commit the state change without eventually sending the command, because both these scenarios would leave the overall system in an inconsistent state.</p><p>To do that, we first had to make side effects declarative. For a transition to have a side effect, for example sending a command to another service on a particular transition, we declare it as data on the transition.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0f34a7747b4e52a3e48b445bb6e78569/href">https://medium.com/media/0f34a7747b4e52a3e48b445bb6e78569/href</a></iframe><p>The side effect EmitCommand is then run separately, only when we are sure that this transition was successful — fully committed to the database —, using the mechanism we already described <a href="https://medium.com/memobank/how-we-ensure-eventual-consistency-in-a-distributed-system-932ab2493b39">in this article</a>. This gives us a 100% consistency guarantee.</p><p>And of course, these side effects data is also stored on each state change, for full traceability.</p><p>One of the big advantages of all the formalization is the ability to build smart tooling. We do it once, and benefit from it for all further development. In our case, we added a user interface in our back-office that displays all state changes, inputs and side effects across both services, on the same timeline.</p><p>Here is an example for an outgoing transfer (simplified). The blue rounded badges are the side effects which are emitted when arriving in the state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_k9i9QiNiPItH0VNyeu1Ww.png" /></figure><p>All entities, inputs, states and side effects are clickable and show their data. Even better, in batch_creation for example, the batch itself is a state machine, and the reference can be clicked to see the batch’s timeline in the same UI.</p><p>This is invaluable for the developer team during maintenance and new developments, as well as the customer support team to answer precisely to clients.</p><p>We’ve been using and improving our state machine framework at scale for the past 3 years. The best advice we can give is to make sure all business decisions are made in the states. That’s very important to keep the state history valuable. To do that, we have three rules:</p><ol><li><strong>Documentation first. </strong>State machines are easy to represent with <a href="https://en.wikipedia.org/wiki/State_diagram">state diagrams</a>, it’s a great tool for quick brainstorming and later documentation. Before working on the code, we have a strict rule to first have a peer review on diagram updates. The code review is usually a no-brainer after that.</li><li><strong>Low abstraction level on inputs.</strong> Don’t create inputs that already say what needs to be done. For example when reacting to events, we first made the mistake to convert each incoming event into a dedicated input. Instead, we now have a single EventInput containing the raw event, and it’s up to the state machine itself to give it a meaning, or ignore it.</li><li><strong>Low abstraction level on side effects.</strong> Side effects should be dumb, they shouldn’t make business decisions. Even an if in a side effect executor is suspicious. Also, if on a transition from A to B you need to do 2 things, make 2 side effects with explicit names, don&#39;t try to abstract them into one or you&#39;ll end up calling it DoSideEffectsOfTransitionFromAtoB and have a meaningless state machine history.</li></ol><p>Building our own state machine framework was not the fastest choice but it paid off a thousand times. It does not have the exact same guarantees as event sourcing — all transition data are not guaranteed to be in immutable timestamped events — , but offers 100% consistency, and a fair amount of traceability, while being faster to develop with. It allowed us to gather data effectively in a single view, allowing us to track a financial transaction across the whole system, which proves super helpful for development, maintenance and customer support.</p><p>If it’s the kind of system you’d like to work on, we’re always looking for new talents to join the team, <a href="https://memo.bank/en/careers">get in touch</a>.</p><p><em>Originally published at </em><a href="https://memo.bank/en/magazine/memo-open/"><em>https://memo.bank/en/magazine</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d9bd12200f57" width="1" height="1" alt=""><hr><p><a href="https://medium.com/memobank/how-we-implemented-a-state-machine-framework-d9bd12200f57">How we implemented a state machine framework</a> was originally published in <a href="https://medium.com/memobank">Memo Bank</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we ensure eventual consistency in a distributed system]]></title>
            <link>https://medium.com/memobank/how-we-ensure-eventual-consistency-in-a-distributed-system-932ab2493b39?source=rss-65e0c05c17d------2</link>
            <guid isPermaLink="false">https://medium.com/p/932ab2493b39</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[tech-architecture]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Joan Zapata]]></dc:creator>
            <pubDate>Tue, 28 Jun 2022 14:03:32 GMT</pubDate>
            <atom:updated>2022-06-28T14:03:32.649Z</atom:updated>
            <content:encoded><![CDATA[<p>Previously, we shared <a href="https://memo.bank/en/magazine/messaging-in-a-service-oriented-architecture">our take</a> on communication inside a Service-Oriented Architecture. Today, we’d like to talk about a common pitfall in this kind of system, and the way we solved it, adding a new tool in our toolbox.</p><p>Consider the following code. We’re asked to invite a user, so:</p><ol><li>we open a transaction;</li><li>we save an invitation in the database;</li><li>we send an email to the invitee to complete their invitation.</li></ol><pre>fun inviteUser(request: InvitationRequest) {<br>   transaction { // (1)<br>      val invitation = db.save(buildInvitation(request)) // (2)<br>      emailQueue.enqueue(buildEmail(invitation)) // (3)<br>   }<br>}</pre><p>This code seems perfectly fine: should the queue be unavailable for some reason, (3) will fail, rollbacking the whole transaction, and we’ll be able to try again later. No harm done, right?</p><p>Well, not exactly. There are multiple ways this code can go wrong. We might lose the database connection and be unable to commit. Or the machine could simply crash right after the 3rd step.</p><p>Even if the 3rd step fails, rollbacking might not be the most suitable solution. There’s a well known problem in distributed systems called the <a href="https://en.wikipedia.org/wiki/Two_Generals%27_Problem">Two Generals Problem</a>. Simply said, due to the unreliability of the network, if the 3rd step throws there’s <em>no way</em> to know for sure if the message was enqueued or not. The network might have failed while delivering the final — positive — answer from the queue. We’ll never know.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eliHnxqhS5y4VELIM5R7DQ.jpeg" /><figcaption>In case of network error, we never knows for sure if the call succeeded or not.</figcaption></figure><p>So in the end, we might have rolled back an invitation, leaving no trace in our database, while the invitee actually received an invitation email. In this case the harm would be minor, we wouldn’t have caused major troubles to our client.</p><p><strong>It would be a much bigger problem if we had enqueued a money transfer.</strong></p><p>So let’s just move the enqueuing out of the transaction, right?</p><pre>fun inviteUser(request: InvitationRequest) {<br>   val invitation = transaction { // (1)<br>      db.save(buildInvitation(request)) // (2)<br>   }<br>   emailQueue.enqueue(buildEmail(invitation)) // (3)<br>}</pre><p>Nice try, but now if (3) fails, we might have the opposite problem: an invitation in database which will never be used, because the email was never sent to the invitee. So that’s not really better.</p><p>To solve this common problem, we built a tool based on <strong>persisted post-commit tasks</strong>.</p><pre>fun inviteUser(request: InvitationRequest) {<br>   transaction { // (1)<br>      // (2)<br>      val invitation = db.save(buildInvitation(request)) <br>      // (3)<br>      postCommit.addTask(SendEmailTask(buildEmail(invitation))) <br>   }<br>}</pre><p>This tool does 3 things.</p><ol><li><strong>Do not run the task inside the transaction.</strong> Instead, serialize and persist the task in the database <em>as part of the transaction</em>. This way, either everything goes well and we have both the invitation and the task in the database, or nothing.</li><li><strong>Register a post-commit hook on the transaction.</strong> A post-commit hook is simply some code that will run only <em>after</em> the transaction has successfully been commited. This hook retrieves the task from the database, deserializes it, and executes it. If it succeeds, simply delete it.</li><li><strong>Should it fail, have a job that regularly retries persisted post-commit tasks. </strong>This is very important to prevent the pitfall of the previous solution. The queue might be unavailable during the post-commit hook, or the machine could simply crash in between. So be prepared to retry.</li></ol><p>And that’s it. 🎉</p><p>After a few years of usage, here are some tips based on errors we made.</p><ul><li>You may have seen that the Two Generals problem is still there: if the “enqueue” in the post-commit throws, but the queue actually received the message, the job will retry it, sending the same message twice. Indeed, and it should be stressed that <em>this problem cannot be solved</em>. But thanks to this tool, it can easily be mitigated: <strong>make sure to save the ID of the message to be sent in the task</strong>. This way, if the task is executed twice, the 2 messages will have the same ID, making it easy to deduplicate them on the consumer side. If you’re lucky, the deduplication might even be supported by the middleware itself. And if this task is a direct call to another service, this ID will help make it idempotent as well.</li><li>Even if you can’t really prevent duplicated messages — and it’s OK — , you can try to reduce the occurrences. In the job, <strong>don’t fetch tasks that are too recent</strong>. Indeed, if the task was created less than a few milliseconds or seconds ago, the post-commit hook might be currently executing it, and you’ll increase drastically the chance of executing tasks twice.</li><li>Some tasks will fail forever, this is unavoidable. If you don’t want an infinite stream of error logs, save a count of retries on the persisted task and <strong>stop retrying at some point</strong>. In our case, just like <a href="https://memobank.atlassian.net/wiki/spaces/~jeremie.martinez/pages/3081703">for handling DLQs</a>, we developed a similar UI to deal with those abandoned tasks. We can ignore them, or retry them with a different content for example.</li><li>The framework you use to register post-commit hooks is likely to execute them <em>synchronously</em> right after committing the transaction. If you’re handling an HTTPS request, it means you won’t answer the request before the post-commit hook completes, and that’s probably not what you want. It’s a good idea to <strong>dispatch the post-commit hook to another thread</strong>.</li><li>This tool doesn’t work well if you need the result of the task synchronously inside the transaction, since the task is actually executed after the transaction. The only answer we have here is to <strong>design the system so that you never need to rely on the result of tasks</strong>. For example in the case above, if we needed the result of sending the email, we would instead listen for events from the mailing service, and react to these events instead.</li><li><strong>Apply this tool consistently</strong>. For example, in our case we already had common libraries providing clients for everything that needs to be sent to the network: emitting an event on Kafka, adding a message to ActiveMQ or doing an HTTPS call. We used this tool in each of them, enabled <em>by default</em>. Bypassing it is actually voluntarily hard, as we rarely see a good reason to do so.</li></ul><p>With this tool applied all across our core banking system, we never worry about inconsistencies after crashes or rollbacks. The fact it works transparently and by default brings a real peace of mind, and even newcomers can produce rock-solid consistent code on their first day.</p><p>Thank you for reading this far, I hope this will be as useful to you as it has been to us, and if it sounds like the kind of system you want to work on, <a href="https://memo.bank/en/careers">come say hi</a>.</p><p><em>Originally published at </em><a href="https://memo.bank/en/magazine/memo-open/"><em>https://memo.bank/en/magazine</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=932ab2493b39" width="1" height="1" alt=""><hr><p><a href="https://medium.com/memobank/how-we-ensure-eventual-consistency-in-a-distributed-system-932ab2493b39">How we ensure eventual consistency in a distributed system</a> was originally published in <a href="https://medium.com/memobank">Memo Bank</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Messaging in a Service-Oriented Architecture]]></title>
            <link>https://medium.com/memobank/messaging-in-a-service-oriented-architecture-ee7e7dba0cde?source=rss-65e0c05c17d------2</link>
            <guid isPermaLink="false">https://medium.com/p/ee7e7dba0cde</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[messaging-system]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[software-architecture]]></category>
            <dc:creator><![CDATA[Joan Zapata]]></dc:creator>
            <pubDate>Tue, 08 Dec 2020 14:38:51 GMT</pubDate>
            <atom:updated>2023-08-29T06:28:13.351Z</atom:updated>
            <content:encoded><![CDATA[<p>Service-Oriented Architecture (SOA) has proved its strengths in virtually all industries to produce scalable and evolutive systems. SOA has many meanings, but its core idea of breaking a complex application into smaller, reusable and loosely coupled services has come a long way since the late 1990s. We’ve embraced the concept from day one and after three years, we now have about twenty domain-driven decoupled services structuring our backend applications. It allowed us to use the languages we liked — like <a href="https://memo.bank/en/magazine/cudling-with-kotlin/">Kotlin</a>, <a href="https://memo.bank/en/magazine/why-elixir/">Elixir</a> — and the tools we wanted — like <a href="https://memo.bank/en/magazine/choosing-an-architecture/">Event Sourcing</a> — for the task at hand.</p><p>But having multiple services comes with its own challenges, the first one being <strong>how to make services communicate with each other</strong>? There are a lot of strategies out there, and this article is a return on experience on the ones we use at Memo Bank.</p><p>Let’s begin with two services and this simple use case as an example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VrjttoAn9vEYdoxXp4UQYQ.jpeg" /></figure><h3>REST over HTTPS</h3><p>Like many backend applications we started with <strong>REST </strong>calls over <strong>HTTPS.</strong></p><p><a href="https://en.wikipedia.org/wiki/Circular_dependency">Cyclic dependencies</a> between services need to be avoided at all cost, so <em>contracts</em> can’t directly tell <em>customers</em> to activate the new customer as soon as the contract is signed. Instead, <em>customers</em> has to send requests to <em>contracts</em> regularly to know when the task is finished.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lGNp5SRtJ6r38ARH5adxew.jpeg" /></figure><p>HTTPS is a great start, and definitely something to keep in the toolbox: it’s simple to set up, widely supported, and also used to expose APIs to frontend applications. But there is one big issue with this scheme, it’s <strong>synchronous </strong>(although not necessarily blocking). It means that while <em>contracts</em> is working, <em>customers</em> is waiting with an active connection. When used at scale, it means that the slowest service can slow down the entire system. It also means that when the <em>contracts</em> service is unavailable, the <em>customers </em>service isn’t either, which is called cascading failure.</p><p>How did we tackle this issue? We used two different patterns: discrete and continuous messaging.</p><h3>Discrete messaging</h3><p>Let’s take the first use case : <em>customers</em> wants <em>contracts</em> to send a contract.</p><p>For this, we can solve the problem using a discrete messaging mechanism:<strong> queues</strong>. You can see a queue as a high-availability third-party mailbox. Even when the main service is unavailable, the mailbox accepts messages. Multiple instances of the same service can consume the same queue, in which case the queue acts as a load balancer and ensures every message is handled at least once. Once a message has been handled, the mailbox can forget about it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g8DpYWa0f40zk0E8XRavBQ.jpeg" /></figure><p>We use queues to send <strong>commands</strong> <em>(e.g. “SendContract”)</em> from one service to another. It has several advantages:</p><ul><li>The caller service <strong>doesn’t rely on the availability</strong> of the receiving service, and can resume its work as soon as the command is in the queue;</li><li>The receiver can handle the commands <strong>at its own pace</strong>, and we can easily scale the receiver service up or down depending on the load of its queues;</li><li>As a bonus, <strong>failure</strong> can easily be isolated and handled manually (we hope to cover the <em>dead letter queues</em> topic in another article).</li></ul><h3>Continuous messaging</h3><p>Now let’s see the second use case, when the contract is signed, <em>contracts</em> needs to let <em>customers</em> know it happened so it can activate the customer.</p><p>It’s tempting to use another queue here, but as we said before we don’t want a cyclic dependency, so <em>contracts</em> doesn’t know anything about <em>customers</em>. Thus, it can’t send an “ActivateCustomer” command, and it can’t know when or where to send that command.</p><p>We solved this problem using a continuous messaging mechanism: <strong>streams</strong>. You can see a stream as an ordered sequence of events that can be consumed in real time but are also made available over time. Contrary to queues that are unrelated ephemeral commands, streams tell a <strong>persisting story</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pbqo6B9lupb-G0IKlPo2nA.jpeg" /></figure><p>Each service at Memo Bank broadcasts a stream of <strong>events</strong> describing the lifecycle of its resources. Building and maintaining these events is <strong>an integral part of any development</strong>, whether or not this event is immediately needed. It’s part of the routine, just like automated tests and metrics.</p><p>These events thus become a reliable <strong>timestamped log of</strong> <strong>immutable</strong> <strong>facts</strong>. And since they are part of the API of the emitting service, they can be consumed by any other service, both in real time (one by one) and in the future. These events have an immense value for <strong>traceability</strong> and <strong>data analysis</strong>.</p><p>To sum up:</p><ul><li>Queues are used to <strong>send commands to</strong> a specific service;</li><li>Streams are used to <strong>expose facts from</strong> a specific service.</li></ul><h3>All about dependencies</h3><p>The diagram above might make you wonder though looking at the arrows, doesn’t it introduce a <strong>cyclic dependency</strong> between <em>customers</em> and <em>contracts</em>?</p><p>No, it does not. A dependency is not defined by the direction of the data, but by the knowledge services have of one another. Here, <em>customers</em> knows about <em>contracts, </em>it tells it what to do and it listens to its story. But <em>contracts</em> knows nothing about <em>customers, </em>it doesn’t need to know who is sending the commands, nor who is listening to its story.</p><p>Both the queue and the stream are part of <em>contracts</em> API, and <em>customers</em> depend on this API.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8PyWNpahN76DcjtluTrHrQ.jpeg" /></figure><p>Having a <strong>naming</strong> <strong>convention</strong> for commands and facts is very important to convey this idea. We always use a base form for commands, like “SendContract”, and a past participle form for facts, like “ContractSent”.</p><p>Note that it conveniently matches with our Core Banking System architecture <a href="https://memo.bank/en/magazine/choosing-an-architecture/">based on CQRS/ES</a>. In this terminology, <em>commands</em> are the same, and <em>events</em> are facts.</p><h3>How to choose the direction of the dependency</h3><p>Given the principles explained before, this solution would be just as valid:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TJX-bpsnbThTY8ccqxtr_w.jpeg" /></figure><p>But if both solutions are valid, how to choose one over the other? Well, it’s up to you.</p><p>It all comes down to <strong>the direction of the dependency</strong> you want to set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nNEMPnEVulgcShO1pK_s3A.jpeg" /><figcaption>A: <strong><em>customers</em></strong> depends on <strong>contracts</strong>. B: <strong>contracts</strong> depends on <strong>customers</strong>.</figcaption></figure><p>Here are some questions we usually ask ourselves :</p><ul><li><strong>Can one service easily be agnostic of others?<br></strong>Here for example, as long as we give the content to <em>contracts</em>, <em>contracts</em> can be totally agnostic of which service is using it. It’s harder to imagine customers being agnostic about the fact it requires a contract. That’s in favor of A.</li><li><strong>What if one service was a third party?<br></strong>For instance, it would not make sense for Memo Bank to outsource <em>customers</em> but it could for <em>contracts</em>. Thus, this is in favor of A as well.</li><li><strong>Is one service orchestrating other services?<br></strong>Implicit orchestration is bad, you can find more about it in <a href="https://www.infoq.com/presentations/event-flow-systems/">this talk</a> by Bernd Ruecker. Creating a customer is a <em>complex workflow</em> which involves many services (sending emails, notifications, creating a bank account, etc.), so <em>customers</em> is probably an orchestrator here. Making the orchestrator depend on other services — and not the other way around — makes the code a lot easier to understand, because the full workflow can be found in a single place. That’s also in favor of A.</li><li><strong>Does it create a cycle in the overall architecture?<br></strong>Even if there’s no link between the two services, they both depend on other services. Let’s say <em>customers</em> depends on <em>users</em>, and <em>users</em> depends on <em>contracts</em> already. If we chose solution B, it would create a cycle with the three services. That’s also in favor of A.</li></ul><h3>Conclusion</h3><p>Messaging is one of the first questions we need to answer when creating a Service-Oriented Architecture. Using HTTPS and REST seems like the most straight-forward solution at first, but it has its limitations. We completed our arsenal with <strong>queues</strong> and <strong>streams</strong>, and we set mainly two guidelines.</p><p>First, every service should stream<strong> </strong>events<strong> </strong>when facts happen within this service, even if we don’t need these events yet. These events should tell a <strong>story</strong> of what happens in the service, like “ContractSent”, “ContractSigned”. This is great for traceability — which is required as a bank — , but also to consolidate each service’s API, and to make the system easier to work with for all teams.</p><p>Second, it’s all about the <strong>dependencies</strong>. Dependencies shape the system, and <strong>cyclic dependencies</strong> are the number one enemy. Once the dependencies are properly set, the messaging tools are just here to let the data flow in any direction.</p><p><em>Originally published at </em><a href="https://memo.bank/en/magazine/memo-open/"><em>https://memo.bank/en/magazine</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ee7e7dba0cde" width="1" height="1" alt=""><hr><p><a href="https://medium.com/memobank/messaging-in-a-service-oriented-architecture-ee7e7dba0cde">Messaging in a Service-Oriented Architecture</a> was originally published in <a href="https://medium.com/memobank">Memo Bank</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Choosing an architecture]]></title>
            <link>https://medium.com/memobank/choosing-an-architecture-85750e1e5a03?source=rss-65e0c05c17d------2</link>
            <guid isPermaLink="false">https://medium.com/p/85750e1e5a03</guid>
            <category><![CDATA[event-sourcing]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[elixir]]></category>
            <category><![CDATA[cqrs]]></category>
            <dc:creator><![CDATA[Joan Zapata]]></dc:creator>
            <pubDate>Wed, 18 Oct 2017 08:13:55 GMT</pubDate>
            <atom:updated>2023-08-29T06:36:56.166Z</atom:updated>
            <content:encoded><![CDATA[<h4>It’s all about the dependencies</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3imntXAHqKFqV6FYZTy1sw.jpeg" /></figure><p>Creating a bank from scratch is nothing like your usual Sunday stroll. Picture it more like going on a trek in a far and unknown jungle. As for all tough and long-running activities, it requires a good preparation and carefully chosen tools which allow to move fast and won’t break half way.</p><p>In software, the decisive tool isn’t the database or the framework we use, it’s not even the programming language, it’s the <strong>architecture</strong>. Simply put, it is how we organize our code such that it meets the technical and operational requirements of our business domain.</p><p>We defined at least five key requirements for our <em>Core Banking System</em>.</p><ul><li><strong>Traceability</strong>, in case of an audit we must be able to justify each data in the system with the series of operations that lead to it;</li><li><strong>Performance</strong>, our customers see bank accounts and operations, but our accountants see the transactions in a very different way, yet both should benefit from a very fast interface;</li><li><strong>Availability</strong>, we have the responsibility to keep and improve people’s faith in payment systems, refuting a card payment because we couldn’t answer an authorization request in time would be unacceptable;</li><li><strong>Consistency</strong>, we must be confident over an account balance before we charge it;</li><li><strong>Maintainability</strong>, we must be able to quickly diagnose problems and fix them. You don’t mess with people’s money. Also, our code should be fully tested, and the whole team should be able to work without stepping on each others toes.</li></ul><p>The choice we’re making now will be decisive for the pace at which we’ll be able to adapt to our customers, and provide them with the best experience.</p><h3>Traceability</h3><p>Our business lies in our data. Money itself is only data, an account balance is a number somewhere in a database that lets you pay your rent, your bills and your food. But that’s not enough, we want to be fully transparent on our data, be able to justify each and every operation that happened on an account, leading to its current balance. Our architecture must allow that by design.</p><p>There’s an emerging pattern in software architecture called <strong><em>Event Sourcing (ES)</em></strong>. The term first appeared in 2005 in <a href="https://martinfowler.com/eaaDev/EventSourcing.html">an article</a> by Martin Fowler, but only started to raise public interest in 2011, pushed by Greg Young. His <a href="https://www.youtube.com/watch?v=JHGkaShoyNs">2014 conference talk</a> is a reference on the topic. The mantra of <em>ES</em> is that the state of a system is given by <strong>all the events that lead to that state</strong>. Those events <strong>can never be modified</strong> once emitted, and are stored in an <strong>append-only storage</strong> called the <em>event store</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s3LbOhxnYFC4cKPshlAWcA.jpeg" /><figcaption>The event store is an append-only storage of immutable events.</figcaption></figure><p>This is <strong>very good for traceability</strong>, everything that ever happened to the system is in our database. But now even the <strong>simplest</strong> <strong>query becomes very complex</strong>. Getting an account balance, for example, would force us to iterate through the event store to sum deposits and withdrawals. Not only this is not trivial to implement, it will also get slower and slower as events pile up.</p><h3>Performance</h3><p>That’s the reason why <em>ES</em> is used in combination with another pattern called <em>Command Query Responsibility Segregation</em>, or <strong><em>CQRS</em></strong>. This pattern handles reads and writes in the system with two very distinct entities. In this configuration, <em>ES </em>offers a very natural boundary between them. This architecture is usually referred to as <em>CQRS/ES.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2IUbZocoe_zT-mSec-crmg.jpeg" /><figcaption>Overview of a CQRS/ES architecture.</figcaption></figure><p>The read side waits for events from the event store, and has its own database. After each event, it updates his database accordingly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1oUxSMaXHVuETYiWAG4rcQ.jpeg" /><figcaption>The read side updates its own representation of the system after each event.</figcaption></figure><p><em>BankAccount</em> is called a <strong>projection </strong>of the event stream. This approach has tremendous advantages when it comes to performance and maintainability.</p><ul><li>We can have <strong>as many projections</strong> as we want. For example, one for the client API containing users, accounts and operations. Another for the accountant API containing the general ledgers accounts, accounting movements, assets and liabilities. Those projections are entirely decoupled since the only source of truth are the events.</li><li>We can <strong>easily add or fix existing projections</strong>. All we need to do is to replay all the events using the new projection, then switch to the new database.</li><li>We don’t have to use SQL everywhere, we can choose a <strong>different storage for each projection</strong>, depending on the kind of queries we’ll make. For example, if we need to make a graph-related query we can project the events in a Neo4J database. We could also project events into Elastic Search for very fast search.</li></ul><p>Another advantage of those projections is that they can live on their own, which brings me to another of our requirements.</p><h3>Availability</h3><p>Our system, like most systems, will need to handle <strong>a lot</strong> <strong>more reads than writes</strong>. Yet, we need to keep short response time on every request, especially on the write side. If we’re a little too long to respond to a card transaction authorization, the transaction is rejected, causing frustration to our customer.</p><p>That’s why the event store must be <strong>asynchronous</strong>. This way the write side can simply post events to it and move on, not waiting for all the projections to handle them. Moving projections to <strong>different machines</strong>, we can absorb a <strong>huge traffic</strong> with zero impact on the write side. If the read side becomes overwhelmed, we can even <strong>replicate</strong> the same projection on multiple machines, all connected to the event store.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NCP3YktA4Oo86tHncrOXPw.jpeg" /><figcaption>The architecture can potentially handle a huge traffic using an asynchronous event store and multiple projections.</figcaption></figure><p>While this is very good for performance, there’s now one problem.</p><h3>Write consistency</h3><p>We make our <strong>business decisions</strong> on the write side, based on a given command and the <strong>current state</strong> <strong>of the system</strong>. If this state is in the projections, we’re likely to do a dirty read, thus making a decision based on an outdated state.</p><p>To solve this problem, we divide the write part into small stateful components called<strong> aggregates</strong>. One of them would be “a bank account”. An aggregate handles commands and produces events based on its internal state. In the following example, the state is the bank account balance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BHM0ljTjg2uCKsf8omVcyw.jpeg" /><figcaption>An aggregate receives <strong>commands</strong> and produce <strong>events</strong>.</figcaption></figure><p>The aggregate uses its state to make <strong>business decision</strong>. For example, it can reject a withdraw command due to insufficient funds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vBvbCa9qT_Ttl2ToQkvD1w.jpeg" /><figcaption>An aggregate can reject commands.</figcaption></figure><p>Since there’s only <strong>one instance of an aggregate per bank account</strong>, it ensures consistency. You’ll also note that we don’t need to rely on transactions and database locks.</p><p>Great news! The <strong>actor model</strong> of <a href="https://memo.bank/en/magazine/why-elixir/">Elixir</a> is a perfect fit for these aggregates. An actor, called a <em>GenServer</em> in Elixir, has a <strong>state</strong> and receives <strong>messages in its mailbox</strong>. It guarantees by design that two messages can’t be processed at the same time. It’s also very cheap to start and stop, and <strong>hundreds of thousands</strong> of them can easily run on a single machine.</p><h3>Maintainability</h3><p>Our final criterion was maintainability. The <em>CQRS/ES</em> architecture is <strong>harder to setup</strong> than traditional ones, but <strong>it improves the maintainability over time</strong>.</p><p>Most parts of the system are <strong>pure functions</strong>, which means they are very <strong>straightforward to test</strong>. An aggregate, which is where most of our business logic resides, takes commands and produces events with no side effects. Most tests will inject some commands and simply check the produced events.</p><p>Since most parts of the system are small and decoupled from each other, multiple developers can easily work on different parts <strong>without conflicts</strong>.</p><p>Projections are very <strong>tolerant to errors</strong>, since we can retroactively fix them. If we made the wrong rounding in the accounting projection for example, we don’t need to migrate the data after the fix, we only need to replay the events.</p><p><em>CQRS/ES</em> is not perfect though, so we carefully considered all aspects.</p><h3>Drawbacks</h3><p><em>CQRS/ES</em> is a very opinionated pattern and some tasks may become more complicated than in traditional systems.</p><ul><li><strong>Unicity checks</strong> is a good example. Since aggregates can’t communicate with each other, how do we check that a user email is not already taken?</li></ul><p>It should be made really clear that <strong><em>CQRS/ES</em></strong> <strong>rarely fits a whole system</strong>, and should be used sparingly. Using it for user management and roles for example wouldn’t make much sense, and would better fit a more traditional <em>CRUD</em> system.</p><ul><li>How do we handle an incoming request that needs to <strong>write then read</strong>? For example, a POST request to register a user should return the created user, but dispatching the command then querying a projection wouldn’t work since projections are updated asynchronously.</li></ul><p>Those cases rarely happen in practice, and if they do, there are some <a href="https://10consulting.com/2017/10/06/dealing-with-eventual-consistency/">workarounds</a>.</p><ul><li>If events are immutable in the event store and must replayable, how do we deal with <strong>breaking changes </strong>in the events?</li></ul><p>Short answer, we don’t make breaking changes. Migrating previous events is dangerous, a bit like traveling in time and irremediably changing the course of events. Instead, we add a <strong>version</strong> to each event and handle the different versions in aggregates and projections. That means we must be very careful when designing those events in the first place. However, it must be noted that it isn’t more difficult than handling the versioning of a standard HTTP API for instance.</p><ul><li>How do we deal with <strong>side effects</strong>? For example, how do we schedule jobs in the future, send a confirmation email to a user or interact with external systems?</li></ul><p>There’s an additional concept coming from <em>DDD/CQRS</em> called a “saga” or a “process manager”. It reacts to domain events and produces side effects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zIlUt0KHhLrSzR0IpuQZkQ.jpeg" /><figcaption>A <em>Saga</em> handle events and produce side effects.</figcaption></figure><h3>Conclusion</h3><p>Hopefully you now have a good overview of <em>CQRS/ES</em> and how it suits our needs for the foundation of our IT. At the price of an extra initial complexity, we believe it will help us scale in the right direction and achieve our goals. Being based on pure functions, and especially well-suited for actors, this choice is consistent with <a href="https://memo.bank/en/magazine/why-elixir/">our choice of using Elixir</a> as a language and platform.</p><p>If you happen to be in Paris (or happy to relocate) with a crazy and irresistible urge to learn, write and ship code in a <em>CQRS/ES</em> architecture: <a href="https://work.memo.bank/careers/">let’s talk!</a></p><p><em>Originally published at </em><a href="https://memo.bank/en/magazine/memo-open/"><em>https://memo.bank/en/magazine</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85750e1e5a03" width="1" height="1" alt=""><hr><p><a href="https://medium.com/memobank/choosing-an-architecture-85750e1e5a03">Choosing an architecture</a> was originally published in <a href="https://medium.com/memobank">Memo Bank</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>