<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dean Bachar on Medium]]></title>
        <description><![CDATA[Stories by Dean Bachar on Medium]]></description>
        <link>https://medium.com/@dean15198?source=rss-4d62080aae31------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*2r5ZmbveGZ_AZle7yK32fA@2x.jpeg</url>
            <title>Stories by Dean Bachar on Medium</title>
            <link>https://medium.com/@dean15198?source=rss-4d62080aae31------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 08:02:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dean15198/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Don’t Get Caught by Surprise: Understanding the READ AFTER WRITE Problem]]></title>
            <link>https://medium.com/wix-engineering/dont-get-caught-by-surprise-understanding-the-read-after-write-problem-caf155ca83ab?source=rss-4d62080aae31------2</link>
            <guid isPermaLink="false">https://medium.com/p/caf155ca83ab</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-architecture]]></category>
            <dc:creator><![CDATA[Dean Bachar]]></dc:creator>
            <pubDate>Sun, 11 Aug 2024 10:45:50 GMT</pubDate>
            <atom:updated>2024-08-11T10:45:50.253Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yP7-n8FByVH-rOggEHLfyA.jpeg" /></figure><p><em>So, what’s the problem with reading our own writings? After all, we’re using an ACID-compliant database, and once we commit, everything should be fine, right? Right??</em></p><p><strong>Well, not exactly!</strong> In many microservices architectures, server-to-server communication is inevitable, and Read After Write issues can easily arise. These issues can lead to flaky bugs that are particularly dangerous because they are often difficult to handle and may not even appear in integration or end-to-end tests.</p><p>I’m a backend engineer at Wix Restaurants, and recently we have encountered several Read After Write dilemmas. To illustrate these challenges, let’s consider a simplified and hypothetical case:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kvPN_pe0t0-Jaw4WzmcItg.jpeg" /></figure><p>Imagine you are working at a company developing an e-commerce platform with an event-driven distributed system. Your task is to add two new services: a Carts Service and an Items Service. Each service is responsible for a single entity, with its data stored in a SQL-like database, such as MySQL. Each service has access only to its own database.</p><p>Communication between these services can occur asynchronously through domain events or synchronously via a protocol like REST or gRPC.</p><p>The Items Service manages the items within the platform’s carts. Each item is associated with a specific cart stored in the Carts Service, identified by a cartId reference field. Both carts and items have unique ID fields. Besides cartId, each cart has other attributes like price and type.</p><p>The Carts Service handles the carts, which include a list of discount objects and the cart’s total price. To calculate the total price, we need to consider the discount list and all items associated with the cart, as each discount applies differently based on the item’s attributes.</p><p>Now, let’s say you are implementing the <strong>flow of adding a new item to a cart</strong>. After the initial step of adding the item to the database, the total price of the cart must be recalculated to reflect the real total price after the new item is added.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CYGiBVCbADzmggfOimDbVg.png" /></figure><ol><li>The client side calls an endpoint in the Items Service to create a new item in the cart.</li><li>After validating the request, the Items Service attempts to insert a new record into its database.</li><li>Once the insertion is committed, the Items Service sends an acknowledgment back to the client confirming the successful creation of the item.</li><li>Immediately after the commit, a domain event is released to the event broker. For simplicity, let’s assume this is done using a <a href="https://en.wikipedia.org/wiki/Change_data_capture">CDC connector</a> and the <a href="https://microservices.io/patterns/data/transactional-outbox.html">Transactional Outbox Pattern</a>.</li><li>The Carts Service consumes the “item created” event and begins recalculating the cart’s total price.</li><li>Using the cart ID from the event data, the Carts Service fetches the discounts from its own database and retrieves all items associated with this cart ID from the Items Service.</li><li>The Carts Service calculates the new total price based on the fetched discounts and items and updates the database.</li></ol><p>In the absence of unpredictable errors, do you see a potential flaky bug here? It might surprise you, but the <strong>new cart item might not be returned</strong> during the Carts Service fetch. This can lead to an incorrect total price calculation, causing a mismatch between the total price saved in the database and the actual items and discounts of the cart. The root cause is the <strong>implicit assumption that once an insertion is committed, it can be immediately queried</strong>.</p><p>To understand why the database might not return the most up-to-date data, we need to understand how a typical distributed database works:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CjIpCNK8PMTyURlYwAP8DA.jpeg" /></figure><p>As you probably know, most modern databases need to be able to:</p><ol><li><strong>Distribute Workload:</strong> Distribute at least some of the workload across multiple nodes, allowing the system to utilize more than a single unit of computation and memory. Even if today a single node is sufficient for the current workload, future growth could demand additional nodes. Planning for a distributed system from the start avoids the complexities and potential downtime associated with scaling up later.</li><li><strong>Ensure High Availability (HA):</strong> Continue processing new requests even if some nodes fail. High availability ensures that the system remains operational and can handle requests without significant downtime.</li></ol><p>These requirements necessitate having more than one node in the database system. The next critical decision involves determining how to distribute data among these nodes, assigning roles to each node, and defining the relationships and communication protocols between nodes.</p><p>A common solution to this challenge is replication, where more than one node holds the same data. This redundancy enhances the system’s ability to recover quickly with minimal data loss and allows multiple nodes to serve the same data. However, it also complicates data management. As you will soon see, handling data changes becomes more challenging, and certain trade-offs must be considered.</p><p>When multiple nodes can serve requests for the same data, there are at least three options for their relationships and responsibilities:</p><ol><li><strong>Single Primary Node:</strong> One of the nodes is elected to be the primary (leader/main), and the others are secondary (replicas/followers). The primary node exclusively handles write requests (requests that modify data), while secondary nodes are synchronized after the primary node is updated. Read requests can be distributed across all nodes or directed solely to the secondary replicas.</li><li><strong>Multi Primary Nodes:</strong> More than one node can receive write requests, a common setup in geographically distributed clusters. The implementation can involve multiple clusters, each with a single primary node, and a hierarchical structure between the different replications.</li><li><strong>No Primary Node:</strong> All nodes can receive both read and write requests, distributing the load and responsibilities equally among them.</li></ol><p>For simplicity, let’s assume our database is a typical SQL system with a single primary node. When the primary node receives a write request, it saves the change to its data store, and the replicas start to update asynchronously. The question now is when the primary can acknowledge to the server that the change has been safely saved:</p><p><strong>1. Without Waiting for Secondary Acknowledgments:</strong> <br>When the primary acknowledges immediately without waiting for secondaries, the latency for each request is minimized, and the write throughput of the entire system increases. However, this approach increases the risk of data loss. If the primary node crashes and there is data corruption, it might be hard to restore changes that have not yet synced to any secondary.</p><p><strong>2. Waiting for All Secondary Acknowledgments:</strong> <br>Waiting for all secondary nodes to acknowledge ensures all nodes are up-to-date, minimizing the potential for data loss in case of corruption and shortening the time needed to elect a new primary node after a crash.</p><p>However, this approach is <strong>often impractical</strong> due to several constraints.</p><p>The first problem is <strong>speed</strong>: issues with CPU, network, storage, etc., can cause a slow response from a node. For example, if the probability of getting a slow response from a secondary is around 0.7% and there are 12 secondary nodes, the chance of at least one secondary having a slow response is approximately 1 — (1–0.007)¹² ≈ 8.4%. Since the primary must wait for all nodes to acknowledge the change before responding, this can significantly increase the latency.</p><p>Another issue is availability: if one or more nodes are unavailable, the system’s ability to accept write operations is impacted, as all nodes must acknowledge the write.</p><p>As availability and the ability to handle increased workloads were some of the main reasons for having more than one node, these problems limit the practicality of this option.</p><p><strong>3. Waiting for Some Secondary Acknowledgments:</strong> This option allows configuration of the number or ratio of secondary acknowledgments required, balancing write latency with high availability. By configuring the right quorum, write latency can be minimized while ensuring that replicas maintain the desired level of high availability. The level of configurability of this option depends on the specific database and its version.</p><blockquote>Note: The point at which a node saves the change before acknowledging it has been saved depends on the specific database and its configuration. For example, in MongoDB, a node can be configured to acknowledge a write after the update is reflected in memory, not necessarily on disk. This means that in some cases, write operations might disappear if nodes crash before the data is persisted to disk.</blockquote><p>As you can see, in choices 1 and 3 (which are very popular), there is a chance you won’t get up-to-date data because you are fetching from a secondary that still hasn’t synced from the primary.</p><p>There are even more cases where you won’t get the most up-to-date data. For example, let’s take MySQL replication. In the default mode, Asynchronous, the primary (called the source) won’t wait for acknowledgment from any secondary (called replicas). As we just explained, there is no guarantee you will get the most up-to-date data unless you query from the primary. If you decide to change the default behavior and configure the Semisynchronous mode, you can set the number of acknowledgments required from replicas. The source waits for this configured number of acknowledgments before confirming the write to the client. Even if we are fortunate and query from a replica that just acknowledged the source, there is still no guarantee it will return the most up-to-date data. The reason is that the source waits for the replica to save the change in the relay log and flush it to disk (not necessarily in the storage engine), meaning that the change is not necessarily viewable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qsrzjtScQO1j7aL0VtW5lw.jpeg" /></figure><p>In our simple case, the solution is pretty obvious. The Carts Service knows which item was added, so if one of the nodes does not return the created cart, the Carts Service can try again to fetch the data. With some retries and waiting between them, the issue should be resolved.</p><p>In other cases, it might not be as straightforward, and it could be hard to detect that the data is not up-to-date. For example, if there is no indication in the event that the data is old. A possible solution is to enrich the event before submitting it.</p><p>Another solution is to read from the primary node.<strong> In a lot of cases, this might lead to performance bottlenecks</strong>, as one node has limited resources and can process only a limited number of requests simultaneously. Moreover, as in our example, the other service needs to allow fetching from the primary node in the API, which can lead to abuse in the future. However, <strong>there are cases where this decision could be appropriate</strong>. For example, if the same endpoint is changing the data table and our database follows a classic primary-secondary architecture. Since the primary node is already involved in the data change, fetching from the primary node of the same table might not be too resource-intensive (generally, write operations require more resources than read operations). This is especially true when the database is owned by the same service and the fetch and write operations occur in the same endpoint.</p><p>In a client-server relationship, another common solution is to do nothing. For example, imagine you are publishing an article on an open article platform. You submit your new article and tell your friend to check your feed to read your amazing article on Read-After-Write. It might not be a big deal if the article isn’t immediately visible, and your friend has to refresh the site to see it. In some circumstances, this behavior might be acceptable, as users are accustomed to “retrying” to see the most recent updates.</p><p>In conclusion, reading is a very healthy and important hobby, but it may be even more productive to read what someone else writes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=caf155ca83ab" width="1" height="1" alt=""><hr><p><a href="https://medium.com/wix-engineering/dont-get-caught-by-surprise-understanding-the-read-after-write-problem-caf155ca83ab">Don’t Get Caught by Surprise: Understanding the READ AFTER WRITE Problem</a> was originally published in <a href="https://medium.com/wix-engineering">Wix Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Long Running Tasks In Microservices Architecture]]></title>
            <link>https://medium.com/@dean15198/long-running-tasks-in-microservices-architecture-568c204da75e?source=rss-4d62080aae31------2</link>
            <guid isPermaLink="false">https://medium.com/p/568c204da75e</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Dean Bachar]]></dc:creator>
            <pubDate>Fri, 27 Aug 2021 15:05:57 GMT</pubDate>
            <atom:updated>2021-08-28T20:59:01.579Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Long Running Tasks in a Microservices Architecture</strong></h3><p><em>In this article I’m going to introduce the Tasks Service and its mechanism. <br>It’s one solution that solves two problems:</em></p><ol><li><em>How to efficiently keep the clients updated about long running tasks.</em></li><li><em>How to elegantly return a response to the client in an architecture of stateless microservices when using asynchronous communication.</em></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*Iel8MG5l4gJD53oN.jpg" /></figure><p>A year ago my team and I have started a new large project. It probably will continue to develop over the next few years. That’s why I had to pay attention to the system architecture, which might be one of the key factors in order for the project to succeed (especially in the long run).</p><p>The solution I’m going to talk about can be adapted to any technological choice, for example, framework, programming language, DB, communication mechanism, etc. <br>However, in this article, I will also discuss the choices I’ve made and most of their reasoning.</p><p><strong>The long running tasks problem</strong></p><p>My team’s previous project had a demand for a lot of complicated long running tasks. The platforms for all our projects run in a closed environment and the mainly backend programming language we use is Python. That’s the reason, we decided to use <a href="https://www.rabbitmq.com/">RabbitMQ</a>, <a href="https://www.mongodb.com/">MongoDB</a> and the Python libraries <a href="https://docs.celeryproject.org/en/stable/">Celery</a> and <a href="https://flask.palletsprojects.com/en/2.0.x/">Flask</a> to create a distributed processing tasks system. We chose to use RabbitMQ as our message broker because of its push-based approach, simplicity and the fact that it was fitting to our scale.<br>Celery, is a simple to use library with a great performance, a very good documentation, and has perfect integration with RabbitMQ.</p><p>The tasks creating mechanism was very simple:</p><ol><li>The client (<a href="https://reactjs.org/">React.js</a>) sent a HTTP request to the stateless server (Flask).</li><li>The server created a new document which represented the task in the DB (MongoDB) and produced it to the message broker (RabbitMQ) — all this was simply done by Celery.</li><li>The consumer (a Celery Worker) received the task and started doing the task logic. After the worker finished, it updated the DB.</li></ol><p>As you can see, Celery makes this mechanism quite easy to apply. <br>Therefore, we chose to use this library again in our new project. Celery’s code is very well documented so we could expand and modify it to our specific needs.</p><p>MongoDB was also a great choice. This technology helped us develop in the agile methodology. Its performances are great, it has excellent scalability and many more.</p><p>The big question was: “How will the client receive the task result?”<br>We used polling: the client asks the server what is the result of the task. <br>If the task is not finished, the client waits a second or two, and asks again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gUOuE0IwX8aKZ57li3smEg.png" /><figcaption>HTTP polling</figcaption></figure><p>The mechanism is simple but not good enough for medium-large projects. Its performance is subpar, it causes CPU time wasting of both the client and the server, partakes in unnecessary DB queries, etc. In addition, even though JavaScript and Flask (with <a href="http://www.gevent.org/intro.html">gevent monkey patching</a>) work in concurrency, there is still overhead.<br>In our new project we met the same demand but came up with a much better solution.</p><p><strong>Why is it relevant to a microservices architecture?</strong></p><p>Like a lot of new projects nowadays, I wanted to use the microservices architecture because of its famous advantages (independent scalability, simpler maintenance and debugging, fault isolation, etc.). <br>I understood that the synchronous communication approach between microservices will create a long HTTP/RPC chain. These chains could waste resources and make the system lose its independence. <br>My solution was that every request that involves more than one service will become a task. If a consumer needs to <a href="https://en.wikipedia.org/wiki/Create,_read,_update_and_delete">CRUD</a> other services, they can just use REST requests (HTTP). The communication between the consumers in different services will be by pub/sub or HTTP polling. Now I’m back to the original problem: <br><em>How to efficiently update the clients about the result and status of the individual task?</em></p><p>I know that my solution will involve a protocol that pushes data from the server to the client. I chose to use <a href="https://en.wikipedia.org/wiki/WebSocket">websocket</a> with socket.io because of its simplicity, wide use, and its adequate performance. The issue when using websocket was the fact that it is a stateful protocol which makes the server stateful as well.</p><p>Statelessness has a lot of advantages:</p><ul><li>It keeps the server simple because it doesn’t endure states as sessions.</li><li>Scalability won’t ever be a problem because it is always possible to deploy more servers.</li><li>It handles server crashes very well as there is no difference between one server and another.</li></ul><p>For the aforementioned reasons, it was very important for my team and I to keep as many servers stateless.</p><p>By creating an independent service, that updates the clients about the active tasks, we were able to keep the other services stateless. <br>We called it “The Tasks Service”.</p><p><strong>The Tasks Service</strong></p><p>The Tasks Service’s responsibility is to hold the tasks’ data and inform the clients about updates regarding active tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*7yL55-d8FVWBBzKGz1OgwQ.png" /></figure><p>It stores the active and previously completed tasks of the system in its inner DB.<br>We chose to store all the tasks’ data in MongoDB. <br>However, in a platform which has more frequent status updates, or when used in a higher scale system, it might be better to split the active tasks and store them in Redis.</p><p>By REST it is possible to:</p><ol><li>Fetch the previously completed tasks in chunks — the amount could be huge.</li><li>Create tasks, update statuses and close active tasks.</li></ol><p>The Tasks Service is holding the socket.io sessions with the clients — combined with Redis we are able to store big session groups and perform broadcasts even with multiple Tasks Service servers. <br>Each session group holds all the open sessions that have permission to view a specific kind of tasks. We named these groups the “Notification Groups”.</p><p>In our new project, we chose to use <a href="https://fastapi.tiangolo.com/">FastAPI</a> as our mainly backend framework. The framework is more modern, has better performance, asyncio support included, etc. <br>In the beginning of the project, FastAPI still didn’t have a good and stable integration with socket.io, that’s why we chose to use <a href="https://flask-socketio.readthedocs.io/en/latest/">Flask-SocketIO</a> for this special service as it is more stable and mature but is still maintained.</p><p>After the client performs the login:</p><ol><li>A socket is created between the client and one of the tasks services.</li><li>The service stores the client’s session in its Notification Groups (in Redis).</li><li>The server fetches from the DB (currently MongoDB) the active tasks that the client has permission to view (by its groups).</li><li>The server sends the fetched active tasks to the client using the socket.</li></ol><p><strong>What happens when the client requests a long running task or a request which involves two or more services?</strong></p><ol><li>The client requests a specific service to create a new task.</li><li>The server (of the specific service) requests the Tasks Service to create a new task. This request also contains the parameters and the Notification Groups of the task.</li><li>The Tasks Service saves the task in the DB and informs the relevant clients via the sockets in the Notification Groups that a new active task was created.</li><li>The server in the specific service produces the task to the message broker and responds “201 OK” to the client.</li><li>The consumer of the service consumes the task and performs its logic.</li><li>When the consumer finishes the task or wants to update its status, it performs an API call to the Tasks Service. In this case, the Tasks Service updates in its DB and informs the clients via the sockets in the Notification Groups.</li></ol><p>On the React.js side we are saving the active tasks by a context hook.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*11oPS4GkiTKNvH_d0Qs-uA.png" /></figure><p>The actions that the worker (the consumer) is capable of, are:</p><ol><li>CRUD his inner DB or CRUD via HTTP request to another service</li><li>Produce another task</li><li>Request another task by HTTP polling</li><li>Produce the task to a different service</li></ol><p>This generic mechanism allows the Tasks Service to only do its own job.<br>The code of the Tasks Service does not have reference to any of the tasks. It is completely possible to continue to develop more new tasks and services and keep the Tasks Service just as it is. The users that are eventually informed about the task are determent by the requesting service and not the Tasks Service (because the service sends the Notification Groups of the task in the request).</p><p>Because most of our services are written in Python we wrote a SDK that performed the task creating mechanism in a generic way. The other APIs/workers use it as a library so the logic is written only once.</p><p><strong>Can it be improved?</strong></p><p>If you suggest that this mechanism is causing a coupling between the Tasks Service and the other services, or the amount of connected users at the same time is huge, you can apply these changes (or at least one of them):</p><ul><li>Perform steps 2–4 as a background task and return “201 OK” to the client just after the server got the request for the new task. <a href="https://fastapi.tiangolo.com/tutorial/background-tasks/">For example in FastAPI it can be done easily.</a></li><li>The socket.io messaging will be by nested workers. The Tasks Service HTTP API will return “201 OK” just after it saves the task in the DB and produces it to a message broker.</li></ul><p>These changes make the solution harder and also have disadvantages. <br>For example, if there is a problem in steps 2–4, the client won’t get a notification that informs about the problem and the fact that the task failed to start. <br>In our perspective, the solution is great as it is. For our amount of users, we had a wonderful performance.</p><p><strong>Conclusion.</strong></p><p>As I stated previously, the technologies can be changed but the main idea of the Tasks Service and its mechanism can still be applied. For example, it is possible to use server side messaging instead of socket.io.<br>Another change that could have been made is to use Node.js for the Tasks Service server as it is a completely event driven API. <br>It may be obvious but the frontend framework can be replaced and the solution can be applied to mobile platforms as well.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=568c204da75e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>