<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Lucas Godoy on Medium]]></title>
        <description><![CDATA[Stories by Lucas Godoy on Medium]]></description>
        <link>https://medium.com/@godoy-lucas-e?source=rss-d5d525175f22------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*XcSHlcW1gQ91I2LtbzFysg.png</url>
            <title>Stories by Lucas Godoy on Medium</title>
            <link>https://medium.com/@godoy-lucas-e?source=rss-d5d525175f22------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 17:26:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@godoy-lucas-e/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Golang Concurrency — Building a Simple Rate Limiter - Token Bucket algorithm]]></title>
            <link>https://godoy-lucas-e.medium.com/golang-concurrency-building-a-simple-rate-limiter-token-bucket-algorithm-62de4f389039?source=rss-d5d525175f22------2</link>
            <guid isPermaLink="false">https://medium.com/p/62de4f389039</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[rate-limiting]]></category>
            <category><![CDATA[concurrency]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Lucas Godoy]]></dc:creator>
            <pubDate>Wed, 28 Feb 2024 18:04:25 GMT</pubDate>
            <atom:updated>2024-02-28T18:04:25.891Z</atom:updated>
            <content:encoded><![CDATA[<h3>Golang Concurrency — Building a Simple Rate Limiter - Token Bucket algorithm</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*TdqIVn9m4Jz6e6RVW7Ho0Q.jpeg" /></figure><p>Are you curious about how to control the rate at which your program makes requests or processes tasks? In this article, we’ll explore a simple yet powerful concept called a token bucket, and implement a basic rate limiter using Golang. This guide will help you understand the fundamentals behind rate limiting in just a few lines.</p><h3>Understanding the Token Bucket</h3><p>A token bucket is a classic algorithm used for rate limiting. In simple terms, it consists of a bucket filled with tokens, where each token represents permission to perform a specific action. Here’s a breakdown of the components in our Golang implementation:</p><ul><li>count: The size of the bucket, indicating the maximum number of tokens it can hold.</li><li>tokens: A channel representing the tokens in the bucket.</li><li>ticker: A timer that adds a new token to the bucket at regular intervals.</li></ul><h3>The Implementation</h3><p>Let’s delve into the Golang code to see how the token bucket rate limiter is created and utilized:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ef9a3636aec97f4b1dbf1a8f42e0a35f/href">https://medium.com/media/ef9a3636aec97f4b1dbf1a8f42e0a35f/href</a></iframe><h3>How It Works</h3><ol><li>NewTokenBucket initialization: We create a token channel and fill the bucket with the specified number of tokens during rate limiter instantiation.</li><li>The ticker: A ticker runs in the background, adding a new token to the bucket at regular intervals. Every interval of time a new message is delivered through the tb.ticker.C channel then a new token is added unless the bucket is full, it waits until the next tick.</li><li>Start Method: It launches a goroutine to manage the background process of adding tokens. It continuously adds tokens to the bucket as long as the context is not canceled.</li><li>Wait Method: It is used by your program to consume a token from the bucket. If the bucket is empty, it waits until a token becomes available.</li></ol><h3>Implementing Rate Limiting in Your Code</h3><p>Here’s a simple example of how you can use this token bucket for rate limiting:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ef26724d18a5f2c1f6397260c7215557/href">https://medium.com/media/ef26724d18a5f2c1f6397260c7215557/href</a></iframe><h3>Sum-up</h3><p>Fantastic job! You’ve successfully built a fundamental rate limiter using a token bucket in Golang. This versatile concept finds applications in scenarios like API rate limiting and task scheduling. While this approach is excellent for straightforward rate limits per service call, it may face challenges when dealing with more advanced limits based on specific request or entity IDs. In such cases, the current implementation requires two goroutines per limiter, impacting performance. The optimization of implementing the same token bucket algorithm with just one goroutine is a topic worth exploring, and we can delve into that in a separate article. Keep up the great work in your coding adventures!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=62de4f389039" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[Refactor] Go Worker Pool — A way around to the sync package]]></title>
            <link>https://itnext.io/refactor-go-worker-pool-a-way-around-to-the-sync-package-7d45b1afb768?source=rss-d5d525175f22------2</link>
            <guid isPermaLink="false">https://medium.com/p/7d45b1afb768</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Lucas Godoy]]></dc:creator>
            <pubDate>Mon, 05 Jul 2021 11:15:17 GMT</pubDate>
            <atom:updated>2021-07-05T11:25:50.575Z</atom:updated>
            <content:encoded><![CDATA[<h3>[Refactor] Go Worker Pool — A way around to the sync package</h3><h4>Synchronize worker executions by using Semaphore pattern instead of sync.WaitGroup.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*AfbInTo4PqRhyu8srHFmLA.png" /></figure><blockquote><strong>Before you move forward</strong>, I wanted to let you know, this article won’t tell that semaphores are better than using WaitGroups or vice versa, but a different approach you can choose for our WorkerPool implementation by purely using channels.</blockquote><h3>Introduction</h3><blockquote><strong>[TL:DR]</strong> <em>you can skip this and </em><a href="#8501"><em>jump right to the implementation</em></a><em>.</em></blockquote><p>Last week I came up with the<a href="https://itnext.io/explain-to-me-go-concurrency-worker-pool-pattern-like-im-five-e5f1be71e2b0"> <strong><em>“Explain to me Go Worker Pool Pattern like I’m five”</em></strong></a><strong> </strong>post, where we went through the pattern, its components, and how they work together. After having posted the article on my LinkedIn account, a former colleague of mine had pointed me out we could still have the WorkerPool working as is, without using WaitGroup from the sync package as a Workers synchronization mechanism.</p><p><a href="https://itnext.io/explain-to-me-go-concurrency-worker-pool-pattern-like-im-five-e5f1be71e2b0">Explain to me Go Concurrency Worker Pool Pattern like I’m five</a></p><p>Remember from the previous post, the WaitGroup’s goal is to block the goroutine on which the WorkerPool runs until every Worker completes the Jobs processing. Therefore, if we had to remove the WaitGroup we would have to emulate its functionality somehow. How do we achieve such behavior implementation?</p><h3>The key is using channels!</h3><p>In a sense, to remove the WaitGroup from the mix, we would have to:</p><ol><li>Create an execution slot counter. Every Worker to be allowed to process the Jobs must <strong>acquire</strong> an execution slot. This operation <em>decreases</em> slots availability by 1.</li><li>Mark the slot as <strong>release</strong>d. This <em>increases</em> slots availability by 1 after the Worker is done with it.</li><li>Prevent the WorkerPool from returning so we <strong>wait</strong> until Workers finish their execution.</li></ol><h4>A Channel to count slots</h4><p>The execution slots buffered channel is configured with worker count capacity will represent the slot total count.</p><pre><em>type </em>slot <em>struct</em>{}<br><em>type </em>slots <em>chan </em>slot<br><br><em>type </em>execution <em>struct </em>{<br>   slots slots<br>}<br><br><em>func </em>newExecutionSlots(capacity int) execution {<br>   slots := make(<em>chan </em>slot, capacity) // capacity = worker count<br>   <em>return </em>execution{slots: slots}<br>}</pre><pre><em>func </em>(e execution) <strong>acquire</strong>() { // <strong>1<br></strong>   e.slots &lt;- slot{}<br>}<br><br><em>func </em>(e execution) <strong>release</strong>() { // <strong>2</strong><br>   &lt;-e.slots<br>}<br><br><em>func </em>(e execution) <strong>wait</strong>() { // <strong>3</strong><br>   <em>for </em>i := 0; i &lt; cap(e.slots); i++ {<br>      e.slots &lt;- slot{} // empty struct type<br>   }<br>}</pre><pre>// close the slots channel when done with it<br><em>func </em>(e execution) close() {<br>   close(e.slots)<br>}</pre><ol><li>To <strong>acquire </strong>one of them we have to push an empty struct onto the execution’s slots channel. This <em>decreases </em>slot availability.</li><li>To <strong>release</strong> a slot reading out of the execution‘s slots channel is needed. This <em>increases</em> slot availability.</li><li>To <strong>wait</strong> on the WorkerPool for Workers to complete we attempt to push many empty structs as workers count onto the slots channel (aiming to take all of them up). But this won’t be allowed, since this channel is fully <strong>acquire</strong>d by Workers (no more free slots available), which blocks the current goroutine. This would happen until Workers are done and <strong>release</strong> their slot. Therefore, in this way, we prevent the WorkerPool from returning from its execution.</li></ol><p>As we can see here, the Worker synchronization will be governed by this “<em>pseudo</em>” counter shared across the Worker’s goroutines. Increasing/decreasing slot availability will be represented by the operations we perform over the slots channel.</p><ul><li><strong>slots count </strong>(Worker count) = <em>slot channel capacity</em></li><li><strong>taken </strong><strong>slots count</strong>= <em>slot channel length</em></li><li><strong>free </strong><strong>slots count</strong> = (<em>capacity — length</em>)</li></ul><p>What we have done here is nothing else than describe a <strong>semaphore</strong> implementation. In computer science, this is a mechanism that allows processes to access shared resources safely and concurrently.</p><pre><em>type </em><strong>semaphore</strong> <em>interface </em>{<br>   acquire()<br>   release()<br>   wait()<br>   close()<br>}</pre><p>Looking around for an easy-to-understand concept description, I stumbled upon a <a href="https://en.wikipedia.org/wiki/Semaphore_%28programming%29">Wikipedia</a> section that depicts it just fine:</p><blockquote>A useful way to think of a semaphore as used in a real-world system is as a record of <strong>how many units of a particular resource are available</strong>, coupled with operations to adjust that record safely (i.e., to avoid race conditions) as units are <strong>acquired</strong> or <strong>become free (released)</strong>, and, if necessary, <strong>wait</strong> until a unit of the resource becomes available.</blockquote><p>Consequently, we can conceive the execution slots channel as the counter for slots availability (slots are our “<em>shared resources</em>”). Every Worker has to claim a slot to run. While all the slots are <strong>acquired</strong> the WorkerPool will <strong>wait</strong> for all of them to complete to get the <strong>release</strong>d slot.</p><p>Finally the WorkerPool code will look like the following:</p><pre><em>func </em>(wp WorkerPool) Run(ctx context.Context) {<br>   eSlots := <strong>newExecutionSlots</strong>(wp.workersCount) <strong>// create new slots</strong><br>   <em>defer </em>eSlots.close() <em>// closing slots channel before WP returns</em><br><br>   <em>for </em>i := 0; i &lt; wp.workersCount; i++ {<br>      <strong>//</strong> <strong><em>for each worker acquire an execution slot</em></strong><br>      eSlots.<strong>acquire</strong>()</pre><pre><em>     // share the slot channel to workers to let them<br>     // mark them as released as soon they finish      <br>     go </em>worker(ctx, wp.jobs, wp.results, <strong>eSlots</strong>)<br>   }</pre><pre><strong>   // wait for workers to finish its execution<br></strong>   eSlots.<strong>wait</strong>()<br>   close(wp.results)<br>}</pre><h4>Summary</h4><p>This refactoring makes your code rely only on channels and gives you the chance to completely get rid of the sync package dependency since you can emulate the WaitGroup behavior by just using channels.</p><p>I leave you here the <strong>PR</strong> where you can check the changes I have made to the WorkerPool’s code and how the WaitGroup was replaced.</p><p><a href="https://github.com/godoylucase/workers-pool/pull/3">Replace WaitGroup using semaphore instead by godoylucase · Pull Request #3 · godoylucase/workers-pool</a></p><p>Any comments or suggestions are more than welcome, thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7d45b1afb768" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/refactor-go-worker-pool-a-way-around-to-the-sync-package-7d45b1afb768">[Refactor] Go Worker Pool — A way around to the sync package</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Explain to me Go Concurrency Worker Pool Pattern like I’m five]]></title>
            <link>https://itnext.io/explain-to-me-go-concurrency-worker-pool-pattern-like-im-five-e5f1be71e2b0?source=rss-d5d525175f22------2</link>
            <guid isPermaLink="false">https://medium.com/p/e5f1be71e2b0</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[concurrency]]></category>
            <dc:creator><![CDATA[Lucas Godoy]]></dc:creator>
            <pubDate>Sat, 26 Jun 2021 20:35:39 GMT</pubDate>
            <atom:updated>2021-07-05T11:29:19.803Z</atom:updated>
            <content:encoded><![CDATA[<h4>An easy explanation about how all the components of this pattern work together to concurrently process a batch of jobs.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/914/1*Ya3fa36roBBhZlMl-kChXw.png" /><figcaption>Concurrent WorkerPool Pattern</figcaption></figure><h4>Introduction</h4><blockquote><strong>[TL;DR]</strong> <em>Skip the intro and go straight to the </em><a href="#fe56"><em>Implementation Details</em></a><em> if you want to.</em></blockquote><p>When I first came into the Go language I was a bit reluctant to its syntax and verbosity. After a couple of months, I slowly began to fall in love with its simplicity, readability, performance superiority, and small memory footprints over other languages.</p><p>I especially had some particular interest in the language for its rich concurrency model. But despite the <em>“new language honeymoon”</em> stuff, I also struggled a bit trying to figure out how does this model work.</p><p>Because I was running behind a deadline, I went straight to the pattern implementation example instead of understanding the bases of concurrency first. <strong><em>Wrong!</em></strong></p><p>So I highly recommend you guys to go and have a look at the docs as a starting point. Even though Go handles concurrency with simplicity does not mean that programming concurrent code is that straightforward to grasp.</p><p>Nonetheless, I will do my best to be the more colloquial as possible here. I also provide the <a href="https://github.com/godoylucase/workers-pool"><strong><em>GitHub repo</em></strong></a> with an implementation of this pattern and tests for example usage.</p><h3>Implementation Details</h3><blockquote>One thing worth mentioning before moving forward: <em>this is </em><strong><em>NOT</em></strong><em> the post that talks about concurrency concepts</em>, but <strong>how concurrency natives are orchestrated to achieve concurrent execution for a set of defined jobs.</strong></blockquote><p>Having said that, the below diagram depicts the generalities of this pattern. A deeper and detailed description of what is happening behind the curtain is provided further below. (You can also check a <a href="https://drive.google.com/file/d/1yiOu9V3T8FsXSd0RbQVCguF-5IIHJhEh/view?usp=sharing">bigger diagram version here</a>. I made this on my own for you with so much love for better understanding!).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xe4DmSW7U1PNY8vzryKZ6Q.png" /><figcaption>WorkerPool components orchestration</figcaption></figure><h4>1. Jobs Batch</h4><p>I created a minimal work unit called Job, composed of an ExecutionFn that would let us write custom logic for the Job to returning a Result. The latter could be either a value or an error.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/382363fb0cf45d0736fdd3c89917c789/href">https://medium.com/media/382363fb0cf45d0736fdd3c89917c789/href</a></iframe><p>As the second step, I used the generator concurrency pattern to stream all the Jobs into the WorkerPool. What is this about? Generating a stream from ranging over some client’s defined Jobs slice pushing each of them into a channel, the Jobs channel. Which will be used to feed concurrently the WorkerPool.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6a2d89cdaaef5934cee6e43cee50b01c/href">https://medium.com/media/6a2d89cdaaef5934cee6e43cee50b01c/href</a></iframe><h4>2. Job’s channel</h4><p>It is a buffered channel (workers count capped) that once it’s filled up any further attempt to write will block the current goroutine (in this case the stream’s generator goroutine from 1). At any moment, if any Job is present on the channel will be consumed by a Worker function for later execution. In this way, the channel will be unblocked for new Job writes flowing from the generator from the previous point.</p><h4>3. WorkerPool</h4><p>This is the main piece of the puzzle, this entity is composed of Results, Jobs and Done channel, plus the number of Workers the pool will host. It will spawn as many Workers on different goroutines as the worker count indicates, AKA <strong><em>fanning-out</em></strong>.</p><p>The Workers themselves will be responsible for taking Jobs from the channel when available. Then they execute the Job and publishing it Result onto the Result s channel. As long as the cancel() function is not invoked upon Context, the Worker would do the previous mentioned. Otherwise, the loop brakes and WaitGroup is marked as Done(). This is quite similar to think of <em>“killing the </em><em>Worker“</em>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aa31339f6c9e490f60c611a7fb2c2c8e/href">https://medium.com/media/aa31339f6c9e490f60c611a7fb2c2c8e/href</a></iframe><p>After all the available Jobs have been drawn from their channel, the WorkerPool will finish its execution by closing its own Done and Results channels.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0b2e0ea04fb3bbb0b65834d160cfb736/href">https://medium.com/media/0b2e0ea04fb3bbb0b65834d160cfb736/href</a></iframe><h4>4. Results Channel</h4><p>As mentioned before, even though workers run on different goroutines, they publish the Job’s execution Results by multiplexing them onto the Result’s channel, AKA <strong><em>fanning-in</em></strong>. The WorkerPool‘s client can read out from this source even the channel is closed for any reason indicated above.</p><h4>5. Reading Results</h4><p>The WorkerPool’s clients can read out of the Results channel as long as there is at least one of them present on the buffered channel. Otherwise, reading from the empty Results channel blocks the client’s goroutine till a value is present or the channel is closed.</p><p>The for loop gets broken once closed WorkerPool‘s Done channel returns and moves forwards.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1129cc3017562b3a60d392ee755af0f6/href">https://medium.com/media/1129cc3017562b3a60d392ee755af0f6/href</a></iframe><h4>X. Cancel Gracefully</h4><p>In any case, if the client needs to shut down gracefully the WorkerPool execution, either it can call the cancel() function upon the given Context or have configured a timeout duration defined by with Context.WithTimeout method.</p><p>Whether one or other option happens (both end up calling cancel() function, one explicitly and the other after a time out happens) a closed Done channel will be returned from the Context which will be propagated to all Worker functions. This makes the for select loop break, hence the Workers stop consuming Jobs out of the channel. Then later, the WaitGroup is marked as done. But still, running workers will finish their job execution before WorkerPool is shut down.</p><h3>Sum Up</h3><p>When we make use of this pattern, we will achieve concurrent Jobs execution leveraging our system to be more performant and consistent across job executions.</p><p>This pattern could be hard to grasp at first glimpse. But, take your time to digest it, especially if you are new to the GoLang Concurrency Model.</p><p>One thing that could help is to think of channels as pipes, where data flows from one side to the other and the amount of data that could fit is limited. So if we want to inject more data, we just need to make some extra room for it by taking some data out first whilst we wait. On the other way around, if we want to consume from the pipe there has to be something, otherwise, we wait till that happens. In that way, we use these pipes to communicate and share data across goroutines.</p><h3>Resources</h3><p><strong>[Implementation]</strong> A full implementation for this pattern in this <a href="https://github.com/godoylucase/workers-pool"><strong><em>GitHub repo</em></strong></a><strong><em>.</em></strong></p><p><strong>[Book]</strong> I can recommend a book if you are interested in reading more about the concurrency topic: <a href="https://www.oreilly.com/library/view/concurrency-in-go/9781491941294/">Concurrency in Go — OReilly </a><br>For me has been a great resource as a go-to when I face this kind of issue to be solved using concurrent approaches.</p><p>Thanks for reading and I hope you find this useful!</p><p><strong><em>[UPDATE]</em></strong><em> I have written another way to implement this pattern but just using channels (no more </em><em>sync.WaitGroup). You can find it here:</em></p><p><a href="https://itnext.io/refactor-go-worker-pool-a-way-around-to-the-sync-package-7d45b1afb768">[Refactor] Go Worker Pool — A way around to the sync package</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e5f1be71e2b0" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/explain-to-me-go-concurrency-worker-pool-pattern-like-im-five-e5f1be71e2b0">Explain to me Go Concurrency Worker Pool Pattern like I’m five</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimize your data access by using CQRS Architecture Pattern — A theoretical and practical…]]></title>
            <link>https://itnext.io/optimize-your-data-access-by-using-cqrs-architecture-pattern-a-theoretical-and-practical-approach-part-1-b31fe259ea04?source=rss-d5d525175f22------2</link>
            <guid isPermaLink="false">https://medium.com/p/b31fe259ea04</guid>
            <category><![CDATA[event-sourcing]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[patterns]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[cqrs]]></category>
            <dc:creator><![CDATA[Lucas Godoy]]></dc:creator>
            <pubDate>Sat, 12 Jun 2021 20:04:08 GMT</pubDate>
            <atom:updated>2021-06-16T14:29:07.619Z</atom:updated>
            <content:encoded><![CDATA[<h3><em>Optimize your data access by using CQRS Architecture Pattern — Part II</em></h3><h4><em>A theoretical and practical approach</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/535/1*RKNRHm_7C5VoTsM9pArCag.jpeg" /></figure><p>This is the second part of the “<em>Optimize your data access by using CQRS Architecture Pattern — A theoretical and practical approach</em>” series. In <a href="https://godoy-lucas-e.medium.com/cqrs-architecture-pattern-c7f5c613c59c">Part I</a> of this series, we went over the CQRS pattern’s concepts, benefits, and also when this solution might be suitable and convenient to implement. Hence, if you have first reached this article, it is advisable for you, to look at it before moving forward on this one. Otherwise, you can skip the first part if you are just interested in walking through the PoC (Proof of Concept). Let’s dive into the problem and see how CQRS could help us to sort it out.</p><h3><strong>The Problem</strong></h3><p>Let’s say a company has a web application that allows users to save their favorite articles along with other related metadata like source URLs, hashtags, among others. Then, when the app was built this requirement was solved by implementing a simple CRUD API on which a web app POST and save the article to be queried later on by making a GET request using the returned article_id from the POST method. They used MongoDB as storage, which was suitable at that moment since they needed to query the articles collection by ID.</p><p>As the business starts to grow and the app becomes popular, there was a requirement to start linking <strong><em>users that have shared the same article</em></strong> and also what <strong><em>other articles across all of them do share the very same hashtags</em></strong>. At this point, the way they stored the Article’s data into the Mongo collection makes the query implementations hard to carry on. Doing so they would need to start polluting the collection with lots of indexes and treat the data mappings on the service code itself to make the final tweaks and convert it into a client readable DTO.</p><p>Not only that, but they expect, because of the popularity increase, to get more traffic on reads since multiple new business clients will start to read data from this CRUD service, especially interested in these new article data views.</p><p>The above-listed set of requirements would turn this solution almost inviable in terms of scalability and performance if we stayed on the CRUD model because two reasons:</p><ul><li>We don’t want to carry the data treatment to clients,</li><li>Would be hard enough to leverage those new queries on the given data structure.</li></ul><p>So let’s jump right into what we propose as a viable solution to solve these company’s issues.</p><h3><strong>General Architecture</strong></h3><p>As we already know, this pattern aims to pull the read operations apart from the write ones with the principal goal of increasing performance, scalability, and facilitate different views of our main data for query performance avoiding complex join operations or mapping operations after querying the data on the backend side.</p><p>On this opportunity, I picked a set of different tools to carry this PoC on (I don’t want to make this article a Golang, Mongo, Cassandra, or Kafka tutorial, therefore I won’t delve into them just a short description of what they do and how they collaborate towards the solution)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/873/1*vx4YJkp0QcByfTgBBiOsNg.png" /><figcaption>Architecture implementation</figcaption></figure><h3><strong>Divide and Conquer</strong></h3><p>As the architecture diagram yields, we split the problem up into two worlds. On one side there is the write domain. There, our legacy CRUD API seats in. Receiving as before, all the POSTs for creations, PUTs for updates, and DELETEs for article removals.</p><p>On the opposite side, the read domain. Upon which we will leverage all the specific queries. A brand new service will cover this end. Which will sit on top of Cassandra DB containing as many denormalized tables as many queries we are looking to perform. This reading service will expose a set of endpoints that allows to run those queries up and return the results as response bodies.</p><p>But wait, how are those responsibilities connected? How do we keep consistency (at some point) across both sides? To answer these questions a Kafka Topic is brought up into the mix. The latter is used as an event source to broadcast every event that affects the article entities. Every time something changes on the write side an event is posted onto the topic.</p><p>Once an event reaches the read service, it will ingest it and perform data transformations to store that into the corresponding Cassandra’s denormalized tables.</p><p>As you can see from the diagram you can notice many benefits:</p><ol><li>No need to modify your CRUD API logic at all. All the business rules to validate any article creation/update/deletion can still live here and only here.</li><li>The topic became the golden source of data. By replaying all the events you can have a snapshot of the original data’s current state.</li><li>All the queries require minimum business validations. So the read side is decoupled from such a thing, allowing it to scale rapidly when new queries are required to incorporate.</li><li>If new data views are required, you can still add new tables to represent them, write all the logic code to handle the new data transformation and create a new denormalized table to store the data. Once you do that, a new endpoint would expose the data to the clients.</li><li>Furthermore, if you have some other requirement to use another DB engine because you consider it’s best for certain use cases, but still you want to keep this one working, you can create a new read service that uses another different database engine. And still have the data up to date. Thereby replaying the whole event collection from the Kafka topic we can populate the new DB with a current data snapshot.</li><li>Inexpensive queries: the query performance Cassandra offers could be harnessed to generate different data views and in that way avoid expensive data treatments. Cassandra will let us group data by different fields thanks to its columnar data storage approach.</li></ol><p>Although many benefits were listed, complexity increases by the time we decide to move forward on this architecture. We need to start thinking of spanning new Kafka and Cassandra clusters. So we no longer have a simple architecture to maintain now we have a quite complex one. You will start facing new problems such as:</p><ul><li>Defining the proper sharding strategy for Kafka brokers and Cassandra cluster</li><li>Possible hotspots shards need to re-shard your brokers</li><li>Add much more infra for your storage</li><li>Resiliency strategies, etc</li></ul><p>Hope this article has been of interest to you. If you look forward to playing around just a bit and dive deep down on the implementation with an actual example application, here you can find this <a href="https://github.com/godoylucase/cqrs-pattern-impl"><strong>GitHub repo</strong></a> I have been working on. On the readme file, you will find the instructions to run it.</p><p>PS: this repo is not a production-ready service, it is just a PoC. Bear in mind you might find some ugly things, just wanted to prove some thoughts. Thanks for reading!</p><p><a href="https://godoy-lucas-e.medium.com/cqrs-architecture-pattern-c7f5c613c59c">Optimize your data access by using CQRS Architecture Pattern — A theoretical and practical approach</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b31fe259ea04" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/optimize-your-data-access-by-using-cqrs-architecture-pattern-a-theoretical-and-practical-approach-part-1-b31fe259ea04">Optimize your data access by using CQRS Architecture Pattern — A theoretical and practical…</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimize your data access by using CQRS Architecture Pattern — A theoretical and practical approach]]></title>
            <link>https://itnext.io/cqrs-architecture-pattern-c7f5c613c59c?source=rss-d5d525175f22------2</link>
            <guid isPermaLink="false">https://medium.com/p/c7f5c613c59c</guid>
            <category><![CDATA[cqrs]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[software-engineer]]></category>
            <category><![CDATA[software-architecture]]></category>
            <dc:creator><![CDATA[Lucas Godoy]]></dc:creator>
            <pubDate>Fri, 04 Jun 2021 17:29:04 GMT</pubDate>
            <atom:updated>2021-06-16T14:27:38.305Z</atom:updated>
            <content:encoded><![CDATA[<h3>Optimize your data access by using CQRS Architecture Pattern — Part I</h3><h4>A theoretical and practical approach</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*XqplesXCUmnVmKcU4UgpSg.jpeg" /></figure><p>Have you ever wondered how many times you have started to develop a new service with a simple CRUD architecture for a certain domain object from your bounded context and that was “ok” at that moment, whilst the surrounding ecosystem keeps growing and scaling over time till the point you start noticing either (or both):</p><ol><li>The need to execute queries around your object becomes complex to deal with (multiple HTTP calls to other services, expensive joins across tables, etc).</li><li>Performance degradation on your writes since your service ended up having more reading operation than writes.</li></ol><p>Certainly, some of the issues pointed out above could be sorted out in many ways. From the top of my head, I can come up with some data patterns such as sharding strategies, materialized views on DB, data reporting offload (data warehouse), different database access techniques for reads and updates, and there might be some others for sure by using a different combination of patterns and approaches that solves it too.</p><p>However, I will be focusing this writing on one of my favorites which is the CQRS pattern.</p><h3><strong>What problems does it solve?</strong></h3><p>When using traditional architectures like CRUD, where the same data model is used to update and query a database for large-scale solutions, it could end up being a burden. For example:</p><ul><li>The read endpoint may perform multiple queries on different sources on the query side to return complex DTOs mappings with different shapes. And we already know that mappings could become quite complicated.</li><li>On the writes side, the model may implement multiple and complicated business rules to validate creation and update operations.</li><li>We might want to query the model in other ways, maybe collapsing multiple records into one, or aggregating more information to the model that is not currently available on its domain, or just change the way the queries look through the records by using some secondary fields as a key.</li></ul><p>As a result, our CRUD service around our model object starts doing too many things and gets worst when it grows. Then is when this pattern comes into our tool-belt to help us out to solve these scalability issues.</p><h3><strong>The Pattern!</strong></h3><p>CQRS acronym states for <em>Command and Query Responsibility Segregation</em>. Its main purpose is based on the simple idea of separating the data mutation operations (command) from the reading operations (query). To accomplish this, it separates reads and writes into different models, using commands for creations/updates, and queries to read data from them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/0*W4FwjBAZgb8aKDBl" /><figcaption>General CQRS conceptual architecture</figcaption></figure><p>As revealed by the above diagram, you will notice there is a queue of events that connects both writes and reads worlds by pushing onto the topic an event, every time an instance of our domain is being created/updated on the write side. Then after that, the query service will read from the incoming event, denormalize, enrich, slice, and dice the data to create query-optimized models and store them to be read later on.</p><p>Particularly, I focus this article series on leveraging the CQRS pattern by adding Event Sourcing architecture into the mix. It suits properly when we want to keep this flow with a clear separation of concerns, asynchronous, and also to utilize the proper database engines aiming for query performance (let’s say a SQL database for writings and a NoSQL for querying operations over materialized view optimized for queries to avoid expensive joins).</p><p>In addition to this, when we use Event Sourcing architecture, the event topic will become our golden source of data, since it could be used at any time to replay the whole collections of events and reproduce the current state of the data. So that it is possible for us to asynchronously read the queue from the beginning and generate a new set of materialized views from the original data when the system evolves, or when the read model must change. The materialized views are in effect a durable read-only cache of the data.</p><p>Yet another benefit of having separated worlds is the chance to scale both separately, resulting in fewer lock contentions. Also by separating the models makes them more flexible and eases the maintenance due to most of the complex business logic goes into the write model. The read one can be relatively simple.</p><h3><strong>When is this pattern a convenient solution?</strong></h3><p>Like any pattern, CQRS is useful sometimes, but not always. We have learned on many occasions that there is no silver bullet to kill all our issues. But, it is going to be useful under these possible requirements:</p><ul><li>The performance of data reads must be fine-tuned separately from the performance of data writes, especially when the number of reads is much greater than the number of writes. In this scenario, you can scale out the read model, but run the write model on just a few instances.</li><li>Scenarios where one team of developers can focus on the complex domain model that is part of the write model, and another team can focus on the read model and the user interfaces.</li><li>The system is expected to evolve and might contain multiple versions of the model, or where business rules change regularly.</li><li>Integration with other systems, especially in combination with event sourcing, where the temporal failure of one subsystem shouldn’t affect the availability of the others.</li><li>Read data that is eventually consistent is allowed. Since the asynchronous nature of this pattern.</li></ul><h3><strong>Summary</strong></h3><p>As I stated above, this is not the solution to all our scaling/querying issues. We should be very cautious when using the CQRS pattern. It is useful sometimes but not always. Many systems do fit and do work very well with just a CRUD model because they are simple enough and consequently would be a waste of time and turn something easy-to-manage out on a very complex architecture to implement and maintain if we were looking to switch to the CQRS pattern. So try to keep things simple as possible unless it is required. Remember that reading from a queue and making all the data transformations to denormalize the original input could be very expensive regarding time consumption and resource usage.</p><p>However, despite the complexity, I can easily find this pattern very useful when facing problems like the need to have multiple ways to query our model without affecting write performance. Not only that, but also it is worth highlighting the fact we can just replay the whole queue to create new views of the original data to make queries simpler and inexpensive because we could avoid table joins and data merges.</p><h3><strong>Next Steps</strong></h3><p>Certainly, there are many other aspects of this pattern to be addressed in a longer article, but I wanted to highlight here at least the core principles that rule this architecture. So that, let&#39;s jump right into the second part of this series to see a practical example of this pattern and how the moving part interaction is by coding some silly examples using GoLang, Kafka, MongoDB, and Cassandra. See you at the next one!</p><p>[UPDATE] Part II is ready for you guys!</p><p><a href="https://godoy-lucas-e.medium.com/optimize-your-data-access-by-using-cqrs-architecture-pattern-a-theoretical-and-practical-approach-part-1-b31fe259ea04">Optimize your data access by using CQRS Architecture Pattern — A theoretical and practical…</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c7f5c613c59c" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/cqrs-architecture-pattern-c7f5c613c59c">Optimize your data access by using CQRS Architecture Pattern — A theoretical and practical approach</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Go concurrency applied to data pipelines]]></title>
            <link>https://godoy-lucas-e.medium.com/go-concurrency-applied-to-data-pipelines-a766b46a0999?source=rss-d5d525175f22------2</link>
            <guid isPermaLink="false">https://medium.com/p/a766b46a0999</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[concurrency]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[pipeline]]></category>
            <dc:creator><![CDATA[Lucas Godoy]]></dc:creator>
            <pubDate>Sat, 27 Mar 2021 19:03:20 GMT</pubDate>
            <atom:updated>2021-03-29T12:35:57.298Z</atom:updated>
            <content:encoded><![CDATA[<h4>A different approach to batch processing, and how to potentiate the power of data pipelines throughout the use of the Go concurrency model.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*GDsCfxs1yM1nrABsNo4YoA.jpeg" /></figure><h4>Introduction to pipelines</h4><p>The term <strong>pipeline</strong> applied to the computer science field, is nothing more than a series of stages that take data in, perform some operation on that data, and pass the processed data back out as a result.</p><p>Thus, when using this pattern, you can encapsulate the logic of each stage and scale your features quickly by adding/removing/modifying stages, each stage becomes easy to test, nor mentioning the huge benefit of leveraging it by using concurrency, which was the motto for this article.</p><h4>Previous problem and solution</h4><p>A few years back, I had a chance to work for a food and CPG delivery company as a software engineer, where I was part of a team responsible to build pieces of software to integrate retailer’s product availability within the main’s company app. After running that integration, the users were able to buy groceries with fewer stockouts risks.</p><p>To accomplish this feature, we built this “availability engine” in GoLang. So, what was it about? Long story short, it ingested several retailer’s CSV files with product availability information executing a couple of steps to enrich and filter data out based on certain business logic. At the end of the process, new files were made, with all the resultant products to be integrated into the company’s app for users to buy.</p><p>Overall, this data processing pipe looked something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qvOU2a45_q7zIagkrSmaiQ.png" /><figcaption>Example of architecture for handling in batches</figcaption></figure><p>As the diagram states, the first stage of the pipe takes in a set of CSV rows, processes them all, and puts the results on a new batch (new slice of maps). And the very same process repeats it as many times as the number of stages the pipe actually has. The particularity of this pattern was the fact that none of the next stages were able to start if the previous step in the pipe had not finished processing the whole set of rows. As you can see it was conceptually a batch processing pipeline.</p><p>To speed up and optimize the job we start using concurrency at the CSV file level, so we were able to process files concurrently. This approach fits us very well, but there is no silver bullet as we always say…</p><p>When working on this project, I gave in the concurrency to one of the more experienced engineers in the team, so I mostly took part in coding each of those business logic steps.</p><p>Motivated by the curiosity about different languages leads me to give it a try to the Go concurrency model and patterns to have more tools on my toolbelt. Hence I started reading some posts and books about concurrency in Go. While doing so, I stumbled upon a fantastic pattern, pipelines leveraged by the use of channels, which <strong><em>if I would have come across it by that time, I would rather suggest implementing it to solve this feature for sure!</em></strong></p><h4>A better approach for data pipelines: streams of data</h4><p>In my team’s approach, we were using batch processing between stages, which was good enough for us, but there are certainly other options that would have suited better to make it more performant.</p><p>Particularly we are speaking of <em>streaming data</em> across the different pipeline’s stages. This actually means that each stage receives and emits one element at a time instead of waiting for a full batch of results from the previous step to do its thing on them.</p><p>In contrast, if we have to compare the memory footprint between batching and streaming, the former is bigger, since each stage has to make a new slice of maps of equal length to store the results of its calculations. On the contrary, the streaming approach will end up receiving and emitting one element at a time so the memory footprint is back down to the size of the pipeline’s input.</p><h4>Implementation example</h4><p>To demonstrate the pipeline implementation in Go, I had coded a very silly example of this pattern (if you want you can find <a href="https://github.com/godoylucase/go-data-pipelines-example">the source code here</a>). This pipeline is built on top of a chain of stages whose first input would be a series of <em>UserIDs</em> (currently just mocked by code). So then, the pipeline stages are the following:</p><ul><li><strong>S0</strong>: Converts slice of <em>UserIDs</em> to a stream of data using a generator pattern</li><li><strong>S1</strong>: Fetch the users</li><li><strong>S2</strong>: Filter the inactive users</li><li><strong>S3</strong>: Fetch the user’s profile and aggregating it into the payload</li><li><strong>S4</strong>: Convert the overall aggregation as a plain object to be saved later on somewhere</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*9ggwWoIbKVDWHWSZUfp4gw.png" /></figure><p>Here below, at the <em>main</em> <em>function,</em> we can see how the pipeline invocation will look like at line 24.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6eff77f66856fab5365f49b66e5bc189/href">https://medium.com/media/6eff77f66856fab5365f49b66e5bc189/href</a></iframe><p>Let’s start from the innermost function call from our pipeline. The first stage <em>stream.UserIDs(done, userIDs…) </em>is the one that will feed the pipeline by streaming the <em>UserIDs</em> values. To accomplish this, I used a <strong>generator</strong> pattern, which receives a slice of <em>UserIDs</em> (input) and by ranging over it will start pushing each value into a <em>channel</em> (output). Therefore, the returned channel will be in turn the input for the next stage.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/964203c6f40446ab75fc7c10dfe9beeb/href">https://medium.com/media/964203c6f40446ab75fc7c10dfe9beeb/href</a></iframe><p>As I mentioned before, conceptually, every stage is fed by input and spits an output out for results. So then, the stages are going to <em>receive a read-only</em> <em>channel</em> as input to do the data processing and will <em>return a read-only channel</em> on which the resulting outputs will be posted on. Doing so, the next stage on the pipeline will take the previous stage’s output as its input for executing its own logic, and so forth till the end of the pipe.</p><p>Because of that, the use of channels across the pipe will allow us to safely execute concurrently each pipeline stage because our inputs and outputs are safe in concurrent contexts.</p><p>Let’s have a look at the following stages on the chain, where based on the streamed data from the first stage (generator) we fetch the actual user data, filter inactive users out, enrich them with its profile, and finally rip some data apart to make a plain object out of the whole aggregation/filtering process.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bde4e0dab902a1d3c95080899737eaab/href">https://medium.com/media/bde4e0dab902a1d3c95080899737eaab/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ca3b9e10b13a3c8cf224a3cc5a07de9a/href">https://medium.com/media/ca3b9e10b13a3c8cf224a3cc5a07de9a/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ad8bdb553d40bf8bf9a4f104679625bc/href">https://medium.com/media/ad8bdb553d40bf8bf9a4f104679625bc/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a2cb05deb154039ed247fe62f6400776/href">https://medium.com/media/a2cb05deb154039ed247fe62f6400776/href</a></iframe><p>For sure you have noticed I passed a <em>done chan interface{}</em> around the stages. What is this for? It is worth mentioning that goroutines are not garbage collected in runtime, so we have to make sure as programmers to make them all preemptable. By doing that, consequently, we will not leak any goroutine (I will be writing more about this in another post later on) and freeing up memory. Thus, any invocation to the pipeline could be stopped by just closing the <em>done</em> channel. This action will lead to the termination of all the spawn children&#39;s goroutines and cleaning them up.</p><p>All in all, after the latest stage on the pipe, start pushing data out through its output channel another routine (in this case the main one) could start reading out of that channel by iterating over it as stated on main function line 31 and do something with it.</p><p>And that is pretty much it! We implemented our first data pipeline, elegant and efficient! (more than batching pipes).</p><h4>Final thoughts</h4><p>In brief, if I any time on I got the chance to work with a similar problem as I had before, I will go for sure for this pattern, which not only is way more performant in terms of memory footprint but also it is faster than doing a batching approach since we can get data being processed concurrently.</p><p>Also, there are many other things we can do with pipelines, like rate limiting and fanning it in/out. This topic will be covered later in upcoming posts, where the idea is to keep iterating over this pattern by adding and combining more concurrency patterns to it.</p><p>This is my first time publishing an article, so any critics are more than welcome, so I can keep learning from all of you guys.</p><p>Cheers!</p><p>PS: there are a couple of things to be improved in this example that will do make it happen in one upcoming post in which we can talk about error handling for these types of concurrency pattern implementations.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a766b46a0999" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>