<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Abdulsamet İLERİ on Medium]]></title>
        <description><![CDATA[Stories by Abdulsamet İLERİ on Medium]]></description>
        <link>https://medium.com/@abdulsamet-ileri?source=rss-77397c9e27------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*GGGftow6Cy3wRCJh20g6fQ.jpeg</url>
            <title>Stories by Abdulsamet İLERİ on Medium</title>
            <link>https://medium.com/@abdulsamet-ileri?source=rss-77397c9e27------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 01:23:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@abdulsamet-ileri/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Tuning Go Application, which has GC issues with a few steps]]></title>
            <link>https://itnext.io/tuning-go-application-which-has-gc-issues-with-a-few-steps-4487624b313e?source=rss-77397c9e27------2</link>
            <guid isPermaLink="false">https://medium.com/p/4487624b313e</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[garbage-collection]]></category>
            <category><![CDATA[go]]></category>
            <dc:creator><![CDATA[Abdulsamet İLERİ]]></dc:creator>
            <pubDate>Mon, 25 Nov 2024 10:24:11 GMT</pubDate>
            <atom:updated>2024-12-03T10:58:00.001Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Lessons Learned in Performance Tuning and Garbage Collection</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/453/1*SwmmvjovBaJ3LVvmEzS3Ag.png" /><figcaption>Figure taken from <a href="https://www.linkedin.com/pulse/garbage-collector-golang-trong-luong-van-swlcc/">this article</a></figcaption></figure><h3>Motivation</h3><p>I am working on a project that pools the messages from Kafka and writes them to Google Cloud Storage. When I compared it to the existing solution <a href="https://docs.confluent.io/cloud/current/connectors/cc-gcs-sink.html">Confluent Google Cloud Sink Connector</a> under the same environment (millions of messages topic, 50m+), it could not perform well 😔. It finishes <strong>one hour</strong> after the connector finishes. When I tried to investigate why it has such latency, I met some GC issues on our app. I wanted to share my investigation and implementation. <strong><em>As a result of the journey, we didn’t fall behind; we finished 10 minutes before.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*848EuDil5vDuf2XsNZhjtw.png" /><figcaption>Figure: Before improvement (-1h)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZOItfZGcrQyrkEAL0P4eKA.png" /><figcaption>Figure: After improvement (+10min)</figcaption></figure><blockquote>During this journey writing benchmark tests and reading pprof results is important because premature optimization is the root of all evil, if you don’t know well, you can take a look at <a href="https://blog.stackademic.com/profiling-go-applications-in-the-right-way-with-examples-e784526e9481">my pprof article</a>.</blockquote><h3>Little Bit Stack &amp; Heap Allocations</h3><p>In Go, memory allocation happens in one of two places: the stack or the heap. Each goroutine has its own stack, which is a continuous block of memory. In contrast, the heap is a large shared memory area accessible by all goroutines. The diagram below illustrates this concept.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1015/1*GJMBF86hKNlsFAmILEekdQ.png" /><figcaption>Figure: Three goroutines that its own stack and shared one heap</figcaption></figure><p>The stack is self-managing and used exclusively by a single goroutine. In contrast, the heap relies on the garbage collector (GC) for cleanup. Increased heap allocations put more pressure on the GC. When the GC runs, it consumes 25% of the available CPU resources and can introduce “stop-the-world” latency, during which the application is momentarily paused for several milliseconds.</p><blockquote>In general, the cost of garbage collection is directly proportional to the volume of heap allocations made by your program.</blockquote><p>According to our experience, here is our list, for reducing heap 👇</p><ul><li>Using <a href="https://pkg.go.dev/sync#Pool">sync.Pool</a> (for reusing objects instead of initializing every time). <em>Be careful about the implementation, misimplementation causes more harm than benefit.</em></li><li>Prefer <a href="https://pkg.go.dev/strings#Builder">strings.Builder</a> over + concentration</li><li>Try to preallocate slices and maps if their size can be known</li><li>Reduce pointer usage</li><li>Try to avoid large local variables in a function</li><li>Observe compiler decisions via <em>go build -gcflag=&quot;-m&quot; ./... </em>output. Take a look <a href="https://dave.cheney.net/2020/04/25/inlining-optimisations-in-go">at inlining optimisations</a>. <em>(related -l flag)</em></li><li>Struct data alignment technique, use <a href="https://pkg.go.dev/golang.org/x/tools/go/analysis/passes/fieldalignment">fieldalignment</a> linter</li></ul><p>You can also take a look at the great resources below. 👇</p><ul><li><a href="https://github.com/DataDog/go-profiler-notes/blob/main/guide/README.md#garbage-collector">Go Profiler Notes</a></li><li><a href="https://go.dev/doc/faq#stack_or_heap">How do I know whether a variable is allocated on the heap or the stack?</a></li><li><a href="https://medium.com/eureka-engineering/understanding-allocations-in-go-stack-heap-memory-9a2631b5035d">Understanding allocations in a go article</a></li><li><a href="https://www.amazon.com/100-Mistakes-How-Avoid-Them/dp/1617299596">100 Go Mistakes book</a></li></ul><h3>Changing json library</h3><ul><li>We used the <a href="https://github.com/bytedance/sonic">bytedance/sonic</a> instead of the <a href="https://pkg.go.dev/encoding/json">encoding/json</a> library for serialization and deserialization with the size of ~1.50KB objects. Local load test (100k Kafka messages), pprof results (from 2.55s to 0.68s) 👇</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LHSOD0AXzvKHedrUSrLIlg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FMZW1-6EGSVF-s1nPe6Yug.png" /><figcaption>Figure: Sonic performs very well</figcaption></figure><p><em>By changing your json library, you gain speed and performance but sacrifice stability, that’s the tradeoff. encoding/json is a more suitable one in comparison to others.</em></p><h3>GOGC and GOMEMLIMIT Tuning</h3><ul><li><strong>GOGC</strong> (Garbage Collection target percentage) is an environment variable in Go that controls the garbage collector’s aggressiveness by specifying the target heap growth rate. For example, GOGC=100 means the heap can grow by 100% before triggering garbage collection, while lower values result in more frequent collections but smaller memory usage.</li><li><strong>GOMEMLIMIT</strong> is an environment variable introduced in Go 1.19 that sets a soft limit on memory usage for the Go runtime. When the total memory used by the program approaches this limit, the garbage collector becomes more aggressive in staying within the defined budget, helping manage memory in memory-constrained environments.</li></ul><blockquote>You can take a look <a href="https://www.youtube.com/watch?v=07wduWyWx8M">at this great presentation</a> and <a href="https://tip.golang.org/doc/gc-guide#GOGC">the GC guide article</a> for more detail.</blockquote><p>We used <strong>GOGC=off and GOMEMLIMIT=90% of </strong><a href="https://man7.org/linux/man-pages/man7/cgroups.7.html"><strong>cgroup’s</strong></a><strong> memory limit</strong> by using t<a href="https://github.com/KimMachineGun/automemlimit">he automemlimit</a> library. Setting GOGC=off disables regular garbage collection based on heap growth, and GC only runs when memory usage approaches the GOMEMLIMIT, which acts as a soft limit for the heap.</p><p>Our app’s mem and CPU as shown below, (<a href="https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/">ref</a>)</p><pre>resources:<br>  limits:<br>    cpu: &#39;2&#39;<br>    memory: 4Gi<br>  requests:<br>    cpu: &#39;1&#39;<br>    memory: 2Gi</pre><h3>Results</h3><ul><li><strong>GC Pause Frequency</strong> is decreased from <strong>350 to 30</strong>. <em>(calls per minute)</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K_0_WmexhnF8awtNSPkckg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*tzDsKkCZXSoT7Y2aazCzQw.png" /><figcaption>Figure: <strong>GC Pause Frequency </strong>(c<em>alls per minute of stop-the-world garbage collection</em>)</figcaption></figure><ul><li><strong>GC Pause Time </strong>is decreased<strong> </strong>from <strong>40ms to 400µs</strong> at peak.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t5u-uuqJ0bS6buFd-Fx6Ww.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*So_yI3hAw7pKIFM4tRpMnA.png" /><figcaption>Figure: <strong>GC Pause Time</strong> (milliseconds spent in stop-the-world garbage collection)</figcaption></figure><ul><li><strong>GC Time Per Min is </strong>decreased from<strong> 60µs to 2.5µs</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s0XzTKdkMhe0wO3fydwyAg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/1*Y5EVSuyHAFL8T7Ub_XBKWA.png" /><figcaption>Figure: GC Time Per Min</figcaption></figure><ul><li><strong>CPU utilization</strong> is increased from <strong>%3000 to %5000.</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gcv4INdDUXSA91hkgWYGCQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/1*cnHMvA4q20h0Fh-S8CqTfg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mb2rc7e78uVdXxR40YKmMA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/1*j42RRaNOhoC6XkeWYoE79Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xJ8nC5Hil-Q10o7YL6gpFg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/963/1*TDQVKcny1oBN_lt436thmw.png" /><figcaption>Figure: CPU utilization &amp; usage in different network bandwidths</figcaption></figure><ul><li><strong>Our total memory usage</strong> is increased <em>(we allowed that for no aggressive GC pressure)</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pE_QtE63sK9q7VfoW9VQLg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*pWIEGyOdQd7XnH7707lcug.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aiythj7nGrD9GQczTmJSLw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/1*oN_2iCDYwbUR3hVwTBYEXQ.png" /><figcaption>Figure: Memory Usage</figcaption></figure><p>Yes, we allocated more memories but reduced overall GC &amp; CPU time.🚀</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*eGasskLH402EaSO_4p4Y9Q.png" /></figure><p>Thank you for reading so far 💛. All feedback is welcome 🙏</p><p>We are still trying to optimize our app; this is not the end, but the beginning 😃 💃</p><p>I also thank <a href="https://medium.com/u/df34a29d57e6">Emre Odabas</a> and <a href="https://medium.com/u/51bc817a3d69">Mehmet Sezer</a> for supporting me on this journey. 💪</p><h3>Further Reading</h3><ul><li><a href="https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to-stop-worrying-and-love-the-heap/">https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to-stop-worrying-and-love-the-heap/</a></li><li><a href="https://github.com/uber-go/automaxprocs">https://github.com/uber-go/automaxprocs</a> <em>(creates thread with a number of k8s core CPU limit, not node cpu limit.)</em></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4487624b313e" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/tuning-go-application-which-has-gc-issues-with-a-few-steps-4487624b313e">Tuning Go Application, which has GC issues with a few steps</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kafka Konsumer Two Years Journey ]]></title>
            <link>https://medium.com/trendyol-tech/kafka-konsumer-two-years-journey-3e00b46c9ea3?source=rss-77397c9e27------2</link>
            <guid isPermaLink="false">https://medium.com/p/3e00b46c9ea3</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[distributed-tracing]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[exception-handling]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Abdulsamet İLERİ]]></dc:creator>
            <pubDate>Thu, 18 Jul 2024 06:45:05 GMT</pubDate>
            <atom:updated>2024-11-01T11:16:00.152Z</atom:updated>
            <content:encoded><![CDATA[<p>Last year, we introduced <a href="https://medium.com/trendyol-tech/kafka-konsumer-c47b4b8c1599">kafka-konsumer</a> and <a href="https://medium.com/trendyol-tech/kafka-exception-c-r-onsumer-37c459e4849d">kafka-cronsumer</a> libraries in their articles. After one year of development, we are thrilled to write a new blog post introducing other useful features and our use cases. During those two years of journey, these two libraries have been used in <strong>100+ projects</strong> in Trendyol. 🚀</p><p>First of all, let’s remember what these libraries are. Basically,</p><ul><li><a href="https://github.com/Trendyol/kafka-konsumer"><strong>Kafka Konsumer</strong></a>: ensures the easy implementation of Kafka consumer with a built-in exception manager (Kafka-cronsumer).</li><li><a href="https://github.com/Trendyol/kafka-cronsumer"><strong>Kafka Cronsumer</strong></a><strong>:</strong> cron-based Kafka exception consumer with the power of auto retry &amp; concurrency</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*_oiD30kcMQ9Jb2Ao" /><figcaption>Figure: Logo</figcaption></figure><p>As a Seller Product Indexing Team, we have used these libraries for every consumer project. It has worked in production for a very long time; it gave a good performance 💪. For example,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Ku5OsNyO5cIkWYsx2cjNA.png" /><figcaption>Figure: Lag Graph, 16 pods and 16 partitions for the international category</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c4hsUHyrIxEz81hR5Kufag.png" /><figcaption>Figure: Lag Graph, 16 pods and 16 partitions for the content features</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JCLgA6i6rvxCKR2TRUw0fA.png" /><figcaption>Figure: TP Graph, Lots of different types of messages under the heavy load</figcaption></figure><p>Last year, we also wrote <a href="https://abdulsamet-ileri.medium.com/new-winner-of-kafka-consumers-scala-to-go-journey-604c6bdd7041">New Winner of Kafka Consumers: Scala to Go Journey 350 Million Messages per Day</a>. You can find our re-platforming journey and more performance insights!</p><p>Let’s take a look at what we develop and our use cases.👀</p><h4>Table of Contents</h4><ul><li><a href="#4672">Distributed Tracing (Opentelemetry) Support</a></li><li><a href="#b122">Pause/Resume consumer any time</a></li><li><a href="#0518">Run pre-batch function for batch mode</a></li><li><a href="#e97b">Error-Only Retry for batch consuming</a></li><li><a href="#f3c0">Skip Message By Header Feature</a></li><li><a href="#44d5">Exposing Prometheus Metrics</a></li><li><a href="#82b1">What’s new on Kafka Cronsumer?</a></li><li><a href="#8311">Conclusion</a></li></ul><h3>#1 — Distributed Tracing (Opentelemetry) Support</h3><p>Kafka Konsumer uses the <a href="https://github.com/segmentio/kafka-go">segmentio/kafka-go</a> library inside, but unfortunately, Kafka-go does not support open-telemetry. There is also an ongoing <a href="https://github.com/segmentio/kafka-go/issues/1025">issue</a> about it. Based on some work on that issue, we created the <a href="https://github.com/Trendyol/otel-kafka-konsumer">otel-kafka-konsumer</a> project.</p><p>By integrating the <a href="https://github.com/Trendyol/otel-kafka-konsumer">otel-kafka-konsumer</a> project with the <a href="https://github.com/Trendyol/kafka-konsumer">kafka-konsumer</a>, we successfully implemented <a href="https://aws.amazon.com/what-is/distributed-tracing/">distributed tracing</a> in consuming and producing operations. 💃</p><blockquote>You can run the <a href="https://github.com/Trendyol/kafka-konsumer/tree/v2/examples/with-tracing">demo</a>.</blockquote><p>In simple terms, only two settings are significant.<br> - <strong><em>trace.TracerProvider</em></strong> <em>(you can set jaeger,zipkin etc.)<br> - </em><strong><em>propagation.TextMapPropagator</em></strong> <em>(please refer to </em><a href="https://opentelemetry.io/docs/specs/otel/context/api-propagators/"><em>here</em></a><em>)</em></p><p>Enable with <strong><em>DistributedTracingEnabled: true</em></strong><em>; kafka-konsumer</em> does the rest.</p><pre>consumerCfg := &amp;kafka.ConsumerConfig{<br>  DistributedTracingEnabled: true,<br> }</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZO_-NyxuHpaabT0aP5v87Q.png" /><figcaption>Figure: Demo Jaeger dashboard example</figcaption></figure><p><strong>Our Use Case:</strong> Distributed tracing is crucial for identifying which part of your architecture is the bottleneck. We followed an input Kafka message <em>(via traceId)</em> at every step and saw how our architecture performed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x7N9eGRRLlr9zNehQAmrVg.png" /><figcaption>Figure: A trace captured from production</figcaption></figure><h3>#2 — Pause/Resume consumer any time</h3><p>Pause/resume consumer functionality has good benefits. For example,</p><ul><li>It helps to process long-running messages and implement delaying messages.</li><li>Easily implement backpressure or control flow mechanism based on processing time or buffer limit.</li></ul><p>You can control the Pause and Resume functions. This way, the consumer group state is unaffected because it still sends a heartbeat under the hood. You can read <a href="https://github.com/Trendyol/kafka-konsumer/blob/v2/examples/with-pause-resume-consumer/how-it-works.md">how it works</a>.</p><pre>consumer, _ := kafka.NewConsumer(consumerCfg)<br>defer consumer.Stop()<br><br>consumer.Consume()<br>fmt.Println(&quot;Consumer started...!&quot;)<br><br>consumer.Pause()<br>// ..<br>consumer.Resume()</pre><blockquote>You can run the <a href="https://github.com/Trendyol/kafka-konsumer/tree/v2/examples/with-pause-resume-consumer">demo</a><a href="https://github.com/Trendyol/kafka-konsumer/blob/v2/examples/with-pause-resume-consumer/how-it-works.md">.</a></blockquote><p><strong>Our Use Case: </strong>We processed some types of messages (brand, category, etc.) in the evening. Why? Because a category can have more than 50 million products, we must change the category field for more than 50 million products. It&#39;s really heavy work, so we need to do this in the evening. So, this feature is also great for these use cases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fS_OADdeyLjjMV0Topa3qQ.png" /><figcaption>Figure: Lag graph of Brand messages processed in the evening</figcaption></figure><h3>#3 — Run pre-batch function for batch mode</h3><p>The <a href="https://github.com/Trendyol/kafka-konsumer/blob/v2/examples/with-kafka-batch-consumer-with-prebatch/main.go#L22">pre-batch function</a> has good benefits. For example,</p><ul><li><strong>Same-Key Compaction</strong>: Retain only the latest message for each key to reduce the number of processed messages.</li><li><strong>Merging</strong>: Combine related messages into one message to simplify downstream processing and reduce message volume.</li><li><strong>Pre-Processing</strong>: Apply transformations or filtering to the data before processing the batch.</li></ul><p>You can quickly implement a pre-batch function for batch mode consumption. For example, if we have <a href="https://github.com/Trendyol/kafka-konsumer/blob/v2/examples/with-kafka-batch-consumer-with-prebatch/main.go#L53">a version based pre batch function</a> <em>(getting the latest version of a specific message)</em></p><pre>{ &quot;id&quot;: 1, &quot;version&quot;: 1, &quot;payload&quot;: &quot;foo&quot; }<br>{ &quot;id&quot;: 1, &quot;version&quot;: 2, &quot;payload&quot;: &quot;foobar&quot; }<br>{ &quot;id&quot;: 1, &quot;version&quot;: 3, &quot;payload&quot;: &quot;foobarfoobar&quot; }<br>{ &quot;id&quot;: 2, &quot;version&quot;: 1, &quot;payload&quot;: &quot;x&quot; }<br>{ &quot;id&quot;: 3, &quot;version&quot;: 2, &quot;payload&quot;: &quot;y&quot; }<br><br>after preBatch<br>{ &quot;id&quot;: 1, &quot;version&quot;: 3, &quot;payload&quot;: &quot;foobarfoobar&quot; }<br>{ &quot;id&quot;: 2, &quot;version&quot;: 1, &quot;payload&quot;: &quot;x&quot; }<br>{ &quot;id&quot;: 3, &quot;version&quot;: 2, &quot;payload&quot;: &quot;y&quot; }</pre><p>As a result, the number of messages decreases from 5 to 3.</p><blockquote>You can run the <a href="https://github.com/Trendyol/kafka-konsumer/blob/v2/examples/with-kafka-batch-consumer-with-prebatch">demo</a>.</blockquote><p><strong>Our Use Case:</strong> We merged some category messages to decrease the number of messages processed. If we don’t do that, we create extra work for the system. It saves us from processing millions of extra events.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2JzOrHSez4_tvq2wcyCmnw.png" /><figcaption>Figure: Category messages need to be merged</figcaption></figure><p>We couldn’t handle the high load if we didn&#39;t merge this. As you can see below, the number of produced events is about 250m+ before closing the consumer. <em>(Yes, we couldn’t let it continue like this)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gLthHI7643licpBc5AsDXg.png" /><figcaption>Figure: Category messages without merging</figcaption></figure><p>After merging, we decreased category messages from 12.330 to 571 💪 And by merging, we removed extra messages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bqCmhZS95A0C7N8vRlCgCQ.png" /><figcaption>Figure: Category messages enabling merging</figcaption></figure><h3>#4 — Error-Only Retry for batch consuming</h3><p>Assume you are using batch-consuming mode and polling 100 messages from the topic. Some messages failed while processing this group of messages. If you want to retry only failed messages (not all), you can disable the transactional-retry feature easily.</p><pre>consumerCfg := &amp;kafka.ConsumerConfig{<br>  ..<br>  RetryEnabled:       true,<br>  TransactionalRetry: kafka.NewBoolPtr(false),<br>  ...<br> }<br><br>func batchConsumeFn(messages []*kafka.Message) error {<br> for i := range messages {<br>  if i &lt; 2 {<br>   messages[i].IsFailed = true<br>   messages[i].ErrDescription = fmt.Sprintf(&quot;Key = %s error, retry count %s&quot;, string(messages[i].Key), retryCount)<br>  }<br> }<br><br> // you must return error here to retry only failed messages<br> return errors.New(&quot;err&quot;)<br>}</pre><blockquote>You can run the <a href="https://github.com/Trendyol/kafka-konsumer/blob/v2/examples/with-kafka-transactional-retry-disabled/main.go">demo</a>.</blockquote><p><strong>Our Use Case: </strong>Let’s assume we take 100 messages from the topic and query elastic search for every one of them. While some of these are available in elastic search, some are not. We produce unavailable messages to the retry topic and retry after a fixed cron interval.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X1E7QGnahj_FmX6TDIXhxQ.png" /><figcaption>Figure: Query every message, send to retry topic unavailable ones</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xowPXjKSBb6YCj20W8OClA.png" /><figcaption>Figure: Three messages in the same batch have different errors</figcaption></figure><h3>#5 — Skip Message By Header Feature</h3><p>In the event streaming world, you must pass over every message step by step. Sometimes, you need to skip and continue processing to the others. Ideally, a message’s selection value should be specified in its header, not its body, so the function accepts message headers. <a href="https://www.amazon.com/Enterprise-Integration-Patterns-Designing-Deploying/dp/0321200683">[Ref]</a></p><pre>consumerCfg := &amp;kafka.ConsumerConfig{<br>  ..<br>  RetryEnabled:       true,<br>  SkipMessageByHeaderFn: skipMessageByHeaderFn,<br>  ...<br> }<br><br>func skipMessageByHeaderFn(headers []kafka.Header) bool {<br> for _, header := range headers {<br>  if header.Key == &quot;SkipMessage&quot; {<br>   return true<br>  }<br> }<br> return false<br>}</pre><blockquote>You can run the <a href="https://github.com/Trendyol/kafka-konsumer/blob/v2/examples/with-header-filter-consumer/main.go">demo</a>.</blockquote><p><strong>Our Use Case: </strong>Let’s return to our category example; above, we said that we processed category events at night, but there are exceptional category event types we need to process immediately. For this, we created two consumers in the same app, used SkipMessageByHeaderFn for each, and ran at production flawlessly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vbKU7sheWQWqNL6Ac2TtLQ.png" /><figcaption>Figure: SkipMessageByHeader Func</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZCa4FYydoR0rnVlEZ033Bw.png" /><figcaption>Figure: Lag Graph of the instant and night consumers</figcaption></figure><h3>#6 — Exposing Prometheus Metrics</h3><p>Kafka Konsumer exposes its own three metrics and exception metrics from kafka-cronsumer. We can easily create a Grafana dashboard and Prometheus alerts based on these.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GAASLhi9wp78jaSs.png" /><figcaption>Figure: Sample dashboard based on the below metrics</figcaption></figure><pre>kafka_konsumer_processed_messages_total_current<br>kafka_konsumer_unprocessed_messages_total_current<br>kafka_konsumer_error_count_during_fetching_message_total_current<br>kafka_cronsumer_retried_messages_total<br>kafka_cronsumer_discarded_messages_total</pre><p>You can use <a href="https://github.com/Trendyol/kafka-konsumer/blob/v2/examples/with-grafana/main.go#L58">APIEnabled: true</a> If you want to use Kafka konsumer internal API</p><pre>consumerCfg := &amp;kafka.ConsumerConfig{<br>  APIEnabled:  true,<br>  ..<br>}</pre><blockquote>You can run the <a href="https://github.com/Trendyol/kafka-konsumer/tree/v2/examples/with-grafana">demo</a>.</blockquote><blockquote>(If you want to change the prefix of the exposed metrics, you can provide the <a href="https://github.com/Trendyol/kafka-konsumer/blob/v2/examples/with-metric-collector/main.go#L11">MetricPrefix</a> field)</blockquote><p>Or you can get metrics collectors and inject them on your own API.</p><pre>firstConsumer, _ := kafka.NewConsumer(&amp;kafka.ConsumerConfig{})<br>secondConsumer, _ := kafka.NewConsumer(&amp;kafka.ConsumerConfig{})<br><br>allCollectors := append(firstConsumer.GetMetricCollectors(), secondConsumer.GetMetricCollectors()...)<br><br>StartAPI(allCollectors...) # Your own api</pre><blockquote>You can run the <a href="https://github.com/Trendyol/kafka-konsumer/tree/v2/examples/with-metric-collector">demo</a>.</blockquote><h3>What’s new on Kafka Cronsumer?</h3><ul><li>Exception management is based on back-off strategy support (linear, fixed, and exponential options). <a href="https://github.com/Trendyol/kafka-cronsumer/blob/main/backoff-strategy-structure.md">Here</a> details</li><li>Added x-error-message to see what was the error of the message during processing. To learn why any message is in an exception/retry topic, look at its own x-error-message header.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OCDD3LCVgIRPxvjk7De-vA.png" /><figcaption>Figure: See why this message in the exception by looking at x-error-message header</figcaption></figure><h3>Conclusion</h3><p>Thank you to all the contributors <strong>(20+) </strong>who have participated in our two-year journey. I especially want to thank <a href="https://emre-odabas.medium.com/">Emre Odabaş</a> for all your support.</p><p>We are open to hearing your voice and feedback. Don’t hesitate to contact us and give ⭐️️ ⭐ ⭐️ 😄</p><ul><li><a href="https://github.com/Trendyol/kafka-konsumer">GitHub - Trendyol/kafka-konsumer: Easy implementation of kafka consumer with built-in exception manager (kafka-cronsumer)</a></li><li><a href="https://github.com/Trendyol/kafka-cronsumer">GitHub - Trendyol/kafka-cronsumer: Cron based Kafka exception consumer with the power of auto retry &amp; concurrency</a></li><li><a href="https://github.com/Trendyol/otel-kafka-konsumer">GitHub - Trendyol/otel-kafka-konsumer: open telemetry wrapper for kafka-go and kafka-konsumer</a></li></ul><h3>About Us</h3><p>Want to be a part of our growing company? We’re hiring! Check out our open positions and other media pages from the links below.</p><p><a href="https://careers.trendyol.com/">Home - Trendyol Careers</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e00b46c9ea3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/trendyol-tech/kafka-konsumer-two-years-journey-3e00b46c9ea3">Kafka Konsumer Two Years Journey 🚀</a> was originally published in <a href="https://medium.com/trendyol-tech">Trendyol Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Let's implement a basic hotel reservation system with 2PC using pessimistic and optimistic locks in…]]></title>
            <link>https://blog.stackademic.com/lets-implement-a-basic-hotel-reservation-system-with-2pc-using-pessimistic-and-optimistic-locks-in-581256d142e8?source=rss-77397c9e27------2</link>
            <guid isPermaLink="false">https://medium.com/p/581256d142e8</guid>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[2pc]]></category>
            <category><![CDATA[concurrency]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[Abdulsamet İLERİ]]></dc:creator>
            <pubDate>Sun, 29 Oct 2023 09:30:35 GMT</pubDate>
            <atom:updated>2023-10-30T06:42:52.474Z</atom:updated>
            <content:encoded><![CDATA[<h3>Let&#39;s implement a basic hotel reservation system with 2PC using pessimistic and optimistic locks in Go</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qDWhxBHFfbrRbXmF" /><figcaption>Photo by <a href="https://unsplash.com/@calina?utm_source=medium&amp;utm_medium=referral">Georg Bommeli</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Hello all! In this article, I will implement <a href="https://en.wikipedia.org/wiki/Two-phase_commit_protocol">2PC</a> (Two-Phase Commit) with pessimistic and optimistic locking in Go within a basic hotel reservation system. In this system, we will focus on creating a reservation process and will use the PostgreSQL database. You can find the source code <a href="https://github.com/Abdulsametileri/hotel-reservation-service/blob/edb096cdbdb265512b5ce54d1a6293d2718d7fb3/monolith/internal/reservation/repository.go">here</a>!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/757/1*wu2KUH5fwEx0ONGEIGXHIQ.png" /><figcaption>Figure: Problem space</figcaption></figure><p>To understand the problem and its solutions better, first, we will start with monolith architecture (only one database) and then microservice architecture (multiple databases). Let&#39;s get started!</p><p>We have two tables named as room_type_inventory and reservation</p><ul><li><strong>room_type_inventory</strong> table includes hotel room reservations and rooms. There are some important points for its data model. First, users reserve a type of room in a given hotel <em>(standard room, king-size room, two-bedroom, etc.), </em>and<em> </em>room numbers are given only at user check-in. Second, we keep one row per date. In this way, managing reservations with a given date range and queries is easy.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/804/1*xddCaumY9s5SkzBqRz3DDg.png" /><figcaption>Figure: room_type_inventory table</figcaption></figure><p>(<em>hotel_id, room_type_id, and date)</em> are used as <a href="https://www.geeksforgeeks.org/composite-key-in-sql/">composite primary keys</a>. It has some example records, as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BWV39-XptizDJDgr2voczQ.png" /></figure><ul><li><strong>reservation </strong>table includes user reservation data.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/448/1*UExqbasrMWlRtjBYDKa75Q.png" /></figure><p>Its field is self-explanatory. Status indicates reservation status. It can be pending , cancelled , paid , refunded and rejected .</p><h3>Implement Create Reservation in the Monolith</h3><p>Let&#39;s start with monolith architecture. It has only one database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1017/1*33W1TPyzPf2PVGoiFp43Ew.png" /><figcaption>Figure: Create reservation flow</figcaption></figure><p>In the implementation step, all we can do is use transactions for <strong>room_type_inventory</strong> and <strong>reservation</strong> table. I implemented this like</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yBlI1ECUUT_TAorLIHFJrw.png" /><figcaption>Figure: use local transaction</figcaption></figure><blockquote>You can find implementation <a href="https://github.com/Abdulsametileri/hotel-reservation-service/blob/edb096cdbdb265512b5ce54d1a6293d2718d7fb3/monolith/internal/reservation/repository.go">here</a>.</blockquote><p>Okay, fine 😌. <strong>What about the user clicks the &quot;Create Reservation&quot; button multiple times?</strong> If we don&#39;t solve this problem, at least two reservations will be made 😢. We must implement some idempotency <em>(applying the operation several times always produces the same result)</em> mechanism here! 🤔</p><p>There are a couple of solutions to this problem. One of them is making <strong>reservation_id</strong> a primary key. In this way, if the user tries to insert multiple records with the same reservation ID, it gets <strong>pq: duplicate key value violates unique constraint &quot;reservation_pk&quot;</strong> error.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/843/1*vCTwYQs1SF_ex1UY6ApYHA.png" /><figcaption>Figure: trying to make the same reservation multiple times</figcaption></figure><p><strong>What about multiple users trying to book the same room simultaneously? 😕</strong>. We need to solve that one, too. To understand the problem better, assume that the <a href="https://www.postgresql.org/docs/current/transaction-iso.html">database isolation level</a> is not serializable, and as <a href="https://www.geeksforgeeks.org/acid-properties-in-dbms/">ACID</a> requires database transactions, they must complete their tasks independently from other transactions. As a result of this, a race condition occurs in the following.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yecrK3wSFhNq7XcNfe8p8A.png" /><figcaption>Figure: Race condition occurs</figcaption></figure><p>We need to implement some locking mechanism here. There are a couple of techniques. We will explore two of them. Pessimistic Locking and Optimistic Locking.</p><blockquote>Pessimistic and optimistic concurrency control is the difference between apologizing and asking permission. Pessimistic strategies request access to the resource and will only act based on the response, while optimistic strategies assume they have access to the resource, and when they don’t, they apologize by acting accordingly. <a href="https://www.amazon.com/Practical-Event-Driven-Microservices-Architecture-Sustainable/dp/1484274679">[1]</a></blockquote><h4>Pessimistic Locking</h4><blockquote>Pessimistic concurrency strategies lock the resource upfront. The services request access to the resource; if it is available, the application locks it, and it becomes unavailable for every other access that might happen until the application frees the resource. Suppose the resource is unavailable (another consumer or thread already has a lock on it). In that case, the access will fail, and the application will either wait to release the lock or fail the operation. <a href="https://www.amazon.com/Practical-Event-Driven-Microservices-Architecture-Sustainable/dp/1484274679">[2]</a></blockquote><p>To implement pessimistic locking in PostgreSQL, we can use the <a href="https://www.postgresql.org/docs/current/explicit-locking.html#LOCKING-ROWS">FOR UPDATE</a> statement. All we need to do is lock the row we want to change.</p><pre>checkInventoryQuery := `<br>   SELECT date, total_inventory, total_reserved<br>   FROM room_type_inventory<br>   WHERE hotel_id = $1 and room_type_id = $2 AND date between $3 AND $4<br>   FOR UPDATE`</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qw5-CEhpEroKL3V5bFzcPQ.png" /><figcaption>Figure: Pessimistic Locking Example</figcaption></figure><blockquote>You can find implementation <a href="https://github.com/Abdulsametileri/hotel-reservation-service/blob/3c338fbb7b2a446986fc9ef8414a0d738ddf9453/monolith/internal/reservation/repository.go">here</a>.</blockquote><p>There is a couple of good documentation on using it; I strongly recommend <a href="https://linuxhint.com/select-update-postgres/">this</a> to check it out.</p><blockquote>We can also achieve pessimistic concurrency with distributed locks by resorting to an external tool like Consul or ZooKeeper.</blockquote><p>Unfortunately, pessimistic locking is not a silver bullet that fits everywhere perfectly. Let&#39;s review the pros and cons of it.</p><h4>Pros</h4><ul><li>Prevents conflicts between concurrent business transactions by allowing only one business transaction at a time to access data.</li><li>It is appropriate when the chance of conflict is high, or the expense of a conflict is unacceptable.</li></ul><h4>Cons</h4><ul><li><a href="https://en.wikipedia.org/wiki/Deadlock">Deadlock</a> may occur. You must do deadlock-free implementation, like providing lock time or implementing a deadlock detection mechanism.</li><li>It affects database performance if transactions lock the resource long scalability may hurt.</li></ul><h4>Optimistic Concurrency Control</h4><blockquote>Optimistic concurrency strategies assume there is no concurrency and act when concurrency occurs. Typically, there is no locking involved; the flow of the application runs without synchronization. When the application is about to persist the changes, it validates if anything has changed since the start of the operation. If it did, the application aborts or retries the operation. <a href="https://www.amazon.com/Practical-Event-Driven-Microservices-Architecture-Sustainable/dp/1484274679">[3]</a></blockquote><blockquote>A good way of thinking about this is that Optimistic lock is about conflict detection, while a pessimistic lock is about conflict prevention.</blockquote><p>Optimistic locks usually base their conflict detection on a version marker on the data. This can be a timestamp or a sequential counter. I implemented it with an integer <strong>version</strong> column in the <strong>room_type_inventory</strong> table. First, we can get the latest version value with the Select query; second, use it to update the query&#39;s where clause.</p><pre>checkInventoryQuery := `<br>   SELECT date, total_inventory, total_reserved, version<br>   FROM room_type_inventory<br>   WHERE hotel_id = $1 and room_type_id = $2 AND date between $3 AND $4`</pre><pre>updateInventoryQuery := `<br>  UPDATE room_type_inventory<br>  SET total_reserved = total_reserved + 1, version = version + 1 <br>  WHERE hotel_id = $1 AND room_type_id = $2 AND date between $3 AND $4 <br>    AND version = $5`</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ykM7Hr9R2WgiqJyMbNm2Ag.png" /><figcaption>Figure: Optimistic Locking Example</figcaption></figure><blockquote>You can find implementation <a href="https://github.com/Abdulsametileri/hotel-reservation-service/blob/f15be384b9b2fb21441dba0fe1c6ba4a6f20d15f/monolith/internal/reservation/repository.go">here</a>.</blockquote><blockquote>One of the nice things about the optimistic locking pattern that you can extend it so the client passes the version number that they expect in an If-Not-Match or X-Expected-Version header. In certain applications, this can be useful to help the client ensure they are not sending their update request based on outdated information. <a href="https://lets-go-further.alexedwards.net/">[4]</a></blockquote><p>Let&#39;s review the pros and cons of it.</p><h4>Pros</h4><ul><li>it shines in environments with low chances of concurrency. They don&#39;t involve locking and often require retrying the process when concurrency is detected.</li></ul><h4>Cons</h4><ul><li>it has a performance penalty in high concurrency environments.</li></ul><p>Having one database and using transactions with a couple of concurrency controls is nice. What about how we can manage this in a distributed environment like multiple databases in microservices? Let&#39;s take a look 👀.</p><h3>Implement Create Reservation in the Microservices</h3><p>We have two databases: inventory and reservation.</p><ul><li>The inventory database has only one table, <strong>room_type_inventory</strong>, with the same fields mentioned above.</li><li>The reservation database has only one table, <strong>reservation</strong>, with the same fields mentioned above.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A6eDfT_ou5pV6ez1xxEeGA.png" /><figcaption>Figure: microservice architecture</figcaption></figure><p>In this architecture, handling data consistency is very challenging. What about we reserve some inventory but cannot insert any reservation? We must address these failure cases. There are a couple of solutions for handling data inconsistency in the microservice environment: 2PC (two-phase commit) and <a href="https://microservices.io/patterns/data/saga.html">Saga</a> <em>(Choreographed and Orchestrated variants)</em>. In this article, I implement 2PC with <a href="https://www.postgresql.org/docs/current/sql-prepare-transaction.html">PostgreSQL prepared transaction</a> feature. 🚀</p><h4>2PC (Two Phase Commit)</h4><blockquote>Two-phase commit is an algorithm for achieving atomic transaction commit across multiple nodes — i.e., to ensure that either all nodes commit or all nodes abort. <a href="https://www.amazon.com/Designing-Data-Intensive-Applications-Reliable-Maintainable/dp/1449373321">[4]</a></blockquote><p>We can implement 2PC with pessimistic or optimistic locking, as mentioned above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pB93sGsKajwkH3uvmT8T5g.png" /><figcaption>Figure: Microservice architecture</figcaption></figure><p>Before diving into the implementation, let&#39;s look at how to use the prepared transaction feature in PostgreSQL. First of all, you need to pass <a href="https://github.com/Abdulsametileri/hotel-reservation-service/blob/main/docker-compose.yml#L10">max_prepared_transactions</a> config to the PostgreSQL server. After that, you can use, for example</p><pre>BEGIN;<br>UPDATE room_type_inventory<br>SET total_reserved = total_reserved + 1<br>WHERE hotel_id = 100 AND room_type_id = 1 AND date between &#39;2023-10-12&#39; AND &#39;2023-10-13&#39;;<br>PREPARE TRANSACTION &#39;transaction-id&#39;;<br><br>COMMIT PREPARED &#39;transaction-id&#39;; # to commit update transaction<br>ROLLBACK PREPARED &#39;transaction-id&#39;; # to rollback update transaction</pre><pre>BEGIN;<br>INSERT INTO reservation (reservation_id, hotel_id, room_type_id, start_date, end_date, status)<br>VALUES (&#39;65ac63c6-0f65-42c2-a05c-78a1ac9a384a&#39;, &#39;100&#39;, &#39;1&#39;, &#39;2023-10-12&#39;, &#39;2023-10-13&#39;, &#39;pending_pay&#39;);<br>PREPARE TRANSACTION &#39;transaction-create-id&#39;;<br><br>COMMIT PREPARED &#39;transaction-create-id&#39;; # to commit insert transaction<br>ROLLBACK PREPARED &#39;transaction-create-id&#39;; # to rollback insert transaction</pre><blockquote>You should specify the transaction id.</blockquote><p>You can also query living-prepared transactions in the pg_prepared_xacts table</p><pre>SELECT * from pg_prepared_xacts;</pre><blockquote>In the implementation step, I connect two different database in the same application for simplicity.</blockquote><pre>func (d *defaultService) CreateReservation(ctx context.Context, reservation *Reservation) error {<br>   inventoryTxID := uuid.NewString()<br>   reservationTxID := uuid.NewString()<br><br>   _, err := d.inventoryService.UpdatePrepared(ctx, reservation, inventoryTxID)<br>   if err != nil {<br>      d.inventoryService.RollbackPrepared(ctx, inventoryTxID)<br>      return err<br>   }<br><br>   if err = d.reservationService.CreatePrepared(ctx, reservation, reservationTxID); err != nil {<br>      d.inventoryService.RollbackPrepared(ctx, inventoryTxID)<br>      return err<br>   }<br><br>   d.inventoryService.CommitPrepared(ctx, inventoryTxID)<br>   d.reservationService.CommitPrepared(ctx, reservationTxID)<br><br>   return nil<br>}</pre><blockquote>You can find <a href="https://github.com/Abdulsametileri/hotel-reservation-service/tree/7f8ab37eafe7c62623edf89521aa2db544077c64">pessimistic</a> and <a href="https://github.com/Abdulsametileri/hotel-reservation-service/tree/54bf05d466215804c5e615d053ea9def0309f9bc">optimistic</a> implementations. It is the same thing. The only difference is using prepared transactions.</blockquote><p>Thank you for reading so far 💛. All feedback is welcome 🙏</p><h3>References</h3><ul><li><a href="https://www.amazon.com/System-Design-Interview-Insiders-Guide/dp/1736049119">System Design Interview — An Insider&#39;s Guide: Volume 2</a></li><li><a href="https://www.amazon.com/Practical-Event-Driven-Microservices-Architecture-Sustainable/dp/1484274679">Practical Event-Driven Microservices Architecture</a></li><li><a href="https://lets-go-further.alexedwards.net/">Let&#39;s Go Further</a></li><li><a href="https://www.amazon.com/Designing-Data-Intensive-Applications-Reliable-Maintainable/dp/1449373321">Designing Data-Intensive Applications</a></li></ul><h3>Stackademic</h3><p><em>Thank you for reading until the end. Before you go:</em></p><ul><li><em>Please consider </em><strong><em>clapping</em></strong><em> and </em><strong><em>following</em></strong><em> the writer! 👏</em></li><li><em>Follow us on </em><a href="https://twitter.com/stackademichq"><strong><em>Twitter(X)</em></strong></a><em>, </em><a href="https://www.linkedin.com/company/stackademic"><strong><em>LinkedIn</em></strong></a><em>, and </em><a href="https://www.youtube.com/c/stackademic"><strong><em>YouTube</em></strong></a><strong><em>.</em></strong></li><li><em>Visit </em><a href="http://stackademic.com/"><strong><em>Stackademic.com</em></strong></a><em> to find out more about how we are democratizing free programming education around the world.</em></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=581256d142e8" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/lets-implement-a-basic-hotel-reservation-system-with-2pc-using-pessimistic-and-optimistic-locks-in-581256d142e8">Let&#39;s implement a basic hotel reservation system with 2PC using pessimistic and optimistic locks in…</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Profiling Go Applications in the Right Way with Examples]]></title>
            <link>https://blog.stackademic.com/profiling-go-applications-in-the-right-way-with-examples-e784526e9481?source=rss-77397c9e27------2</link>
            <guid isPermaLink="false">https://medium.com/p/e784526e9481</guid>
            <category><![CDATA[pprof]]></category>
            <category><![CDATA[cpu]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[memory-leak]]></category>
            <dc:creator><![CDATA[Abdulsamet İLERİ]]></dc:creator>
            <pubDate>Wed, 13 Sep 2023 08:18:14 GMT</pubDate>
            <atom:updated>2025-08-06T05:13:55.916Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lqQvB8_jBU5U7o-e.png" /><figcaption>Illustration created for “A Journey With Go,” made from the original Go Gopher, created by Renee French.</figcaption></figure><h4>What is profiling?</h4><p><strong>Profiling</strong> is an essential technique for analyzing applications and identifying bottlenecks hindering the application’s performance. It is helpful to detect which parts of the code are taking <strong>too long to execute</strong> or <strong>consuming too many resources,</strong> such as CPU and memory.</p><h4>Table of Contents</h4><ul><li><a href="#def9">Benchmarking </a><br>- <a href="#5bce">Profiling CPU</a><br>- <a href="#0110">Profiling Memory</a></li><li><a href="#0295">Runtime profiling</a><br>- <a href="#b79b">Profiling CPU</a><br>- <a href="#1e2d">Profiling Memory</a></li><li><a href="#6063">Web profiling</a><br>- <a href="#1a86">Getting CPU Profile and tricks</a><br>- <a href="#7d93">Getting Heap Profile and tricks</a><br>- <a href="#833f">Getting Allocation Profile and tricks</a><br>- <a href="#3055">Investigating source code on remote pprof endpoint</a></li></ul><h4>Methods of Profiling</h4><p>You have <strong>three</strong> methods for profiling.</p><ul><li><strong>Go test</strong> (with benchmarking).</li><li><strong>Runtime profiling</strong> with <a href="https://pkg.go.dev/runtime/pprof">runtime/pprof</a>.</li><li><strong>Web profiling</strong> with <a href="https://pkg.go.dev/net/http/pprof">net/http/pprof</a>.</li></ul><h4>Types of Profiling</h4><ul><li><strong>CPU</strong> <em>(collects data on the application CPU usage)</em></li><li><strong>Heap</strong> / <strong>Memory</strong> <em>(collects data on application memory usage)</em></li><li><strong>Goroutine</strong> (<em>identifying the functions that create the most number of goroutines</em>)</li><li><strong>Block</strong> (<em>identifying the functions that cause the most blocking</em>)</li><li><strong>Thread</strong> (<em>identifying the functions that create the most threads</em>)</li><li><strong>Mutex</strong> (<em>identifying the functions that have the most </em><a href="https://en.wikipedia.org/wiki/Resource_contention"><em>contention</em></a>)</li></ul><p><em>In this article, I mainly focus on CPU and Memory profiling with the abovementioned methods.</em></p><h3>1. Benchmarking</h3><p>I wanted to implement <a href="https://leetcode.com/problems/two-sum/">the famous Two Sum algorithm</a> problem. I don’t focus on the details about it. Let’s run</p><blockquote>go test -bench=.</blockquote><p><em>-bench option runs all of the benchmark tests in our project.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9qBrFITo8kpYkdjhosNHTQ.png" /><figcaption>Go test bench output.</figcaption></figure><p>Based on the output above, TwoSumWithBruteForce is the most efficient one compared to others. Don’t forget that it depends on the function’s inputs. If you give a large array, you will get different results. 😅</p><p>If you type go help testflag , you will see lots of flags and their explanations. Like count, benchtime etc. I will explain the most used ones.</p><ul><li>If you want to run a specific function, you can specify it as</li></ul><blockquote>go test -bench=&#39;BenchmarkTwoSumWithBruteForce&#39;</blockquote><ul><li>By default, benchmark functions run only once; if you want to override this behavior, you can use count a flag. For example,</li></ul><blockquote>go test -bench=&#39;.&#39; -count=2</blockquote><p>gives output as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*yre7QpFpOfBTJQ6FhX_DSA.png" /><figcaption>Benchmarking with count flag</figcaption></figure><ul><li>By default, Go decides how long each benchmarking operation runs. You can override this by providing. benchtime=&#39;2s&#39; .</li></ul><blockquote>You can use count and benchtime flags together for better measurement of your benchmarked functions. Y<a href="https://dave.cheney.net/2013/06/30/how-to-write-benchmarks-in-go">ou can also refer to this article.</a></blockquote><blockquote>You can find source code <a href="https://github.com/Abdulsametileri/pprof-examples/tree/main/benchmarking">here</a>!</blockquote><p>In the real world, our functions can be complex and long. The timing is not helpful here, so we need to extract the CPU and memory profile for further analysis. You can type</p><blockquote>go test -bench=&#39;.&#39; -cpuprofile=&#39;cpu.prof&#39; -memprofile=&#39;mem.prof&#39;</blockquote><p>Let’s analyze them using the <a href="https://linux.die.net/man/1/pprof">pprof tool</a>.</p><h4>1.1 Profiling CPU</h4><p>If you type in</p><blockquote>go tool pprof cpu.prof</blockquote><p>and hit enter, you will have a pprof interactive console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*JEtNuVWFUPAXhcRi_yGf8g.png" /><figcaption>go tool pprof cpu.prof output</figcaption></figure><p>Let’s take a look at the most popular ones.</p><ul><li>You can type top15 to see the top 15 resource-intensive functions during the execution time. <em>(15 indicates the number of nodes that will displayed.)</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cxw1P0bswmKSfZQ7XC7s5Q.png" /></figure><p>To be clear, let’s assume that we have a A function.</p><pre>func A() {<br>   B()             // takes 1s<br>   DO STH DIRECTLY // takes 4s<br>   C()             // takes 6s<br>}</pre><p>Flat and cumulative values are calculated as; <strong>A=4 for flat</strong> and <strong>A=11 for cum </strong><em>(1s + 4s + 6s)</em>.</p><ul><li>If you want to sort based on cum value, you can type top15 -cum . You can also use sort=cum and top15 commands respectively.</li><li>If you want to get more verbose output in the top, you can specify granularity option. For example, if we pass granularity=lines , it shows the lines of the functions.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x3mg_gnc2QrSrZZP98tozQ.png" /><figcaption>top15 output with granularity lines</figcaption></figure><p>Thanks to this, we can identify specific lines of the function causing performance issues. 😌</p><ul><li>The output also shows us runtime and user-defined functions together. If you want to focus on only your functions, you can provide hide=runtime and execute top15 again.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/938/1*qj46iS6YkB8trvJ0ye8Upw.png" /><figcaption>top15 with hide option</figcaption></figure><p>You can reset it by typing hide= .</p><ul><li>Conversely, you can use show command. For example, by typing show=TwoSum</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*5N6jVaRlxCmrGg0mOELd4Q.png" /></figure><ul><li>If you want to focus on a specific node, you can use focus command. Let’s focus on TwoSumOnePassHashTable . The output is displayed as</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*M7rP3vlKFO1BIH8bq9uZJg.png" /><figcaption>top output with hide and focus together</figcaption></figure><p>You can reset it by typing focus= .</p><ul><li>If you want to get details of the function, you can uselist command. For example, I want to get details about TwoSumWithTwoPassHashTable function. Type list TwoSumWithTwoPassHashTable</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/987/1*gQTk7a_R2_NwczI4thTaSQ.png" /><figcaption>list TwoSumWithTwoPassHashTable output</figcaption></figure><ul><li>If you want to see a graphical representation of the call stack, you can type web .</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rn12kXsdlpYm3AQxyN5FLQ.png" /><figcaption>web output</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/527/1*zovPVE19VxzDnhHgJFq2CQ.png" /><figcaption>Node detail</figcaption></figure><p>I will give more details about reading the graphs in the following sections.</p><ul><li>You can also type gif or pdf to share profiling data in the corresponding format with someone. 😃</li></ul><h4>1.2 Profiling Memory</h4><p>If you type in go tool pprof mem.prof and hit enter</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/913/1*94xFDjxc8NL7pd7KZYSxqQ.png" /><figcaption>go tool pprof mem.prof output</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/938/1*OqVysc8D241D18dWy6gOEg.png" /><figcaption>top10 output</figcaption></figure><blockquote>Note that flat and cum are the same thing mentioned above but only different measurements <em>(ms in CPU, MB in Memory profiling)</em>.</blockquote><ul><li>list command:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/1*S_YekNIAPY_CMR2w-i-MZQ.png" /><figcaption>list TwoSumWithBruteForce output</figcaption></figure><ul><li>web command:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cLtoqGa-75peutAlv7l4Ew.png" /><figcaption>web command output</figcaption></figure><p>We can use all the commands mentioned in the CPU profiling section.</p><p>Let’s take a look at another method, runtime/pprof. 🚀</p><h3>2- Runtime profiling with <a href="https://pkg.go.dev/runtime/pprof">runtime/pprof</a>.</h3><p>Benchmark tests are useful for an isolated function’s performance but are insufficient to understand the whole picture. Here runtime/pprof shines 💠.</p><h4>2.1 Profiling CPU</h4><p>In benchmark tests, CPU and memory profiling are supported built-in. If you need to add CPU profiling support to your application, you must enable it first.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-l4K4g8B2zFBJtYwAps5UQ.png" /></figure><p>If you run go run . here, you will see the generated cpu.prof file. You can analyze it go tool pprof cpu.prof as mentioned in the benchmarking section.</p><p>In this section, I want to introduce to you one of my favorite features is pprof.Labels . T<em>his feature is available </em><a href="https://pkg.go.dev/runtime/pprof#Labels"><em>only on CPU and goroutine profiling</em></a><em>.</em></p><p>If you want to add a label/labels to a specific function, you can use <a href="https://pkg.go.dev/runtime/pprof#Do">pprof.Do</a> func.</p><pre>pprof.Do(ctx, pprof.Labels(&quot;label-key&quot;, &quot;label-value&quot;), func(ctx context.Context) {<br>    // execute labeled code<br>})<br></pre><p>For example,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c05MDOcUpTUsfHKa9YxF2w.png" /><figcaption>Adding a label to a specific function</figcaption></figure><p>In the pprof interactive console, type tags . It shows your labeled functions with useful information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WMaeIk0knlD5yWfOhOwxKg.png" /><figcaption>tags output</figcaption></figure><p>You can do <a href="https://github.com/google/pprof/blob/main/doc/README.md#tags">lots of things</a> with labels. You can get more information from <a href="https://rakyll.org/profiler-labels/">this article</a>.</p><blockquote>pprof also has a great web interface. It allows us to analyze profiling data using various visualizations.</blockquote><p>Type go tool pprof -http=:6060 cpu.prof , localhost:6060 will be opened. <em>(To be more clear, I removed pprof.Labels)</em></p><p>Let’s dive into graph representation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/1*AshjuOlAf0jh12lTXl1g0g.png" /><figcaption>Graph for CPU analysis</figcaption></figure><p>Node color, font size, edge weight, and many more have different meanings. Please refer to it <a href="https://github.com/google/pprof/blob/main/doc/README.md#interpreting-the-callgraph">here</a>. This visualization allows us to identify and fix performance issues more easily.</p><p>After clicking a node in the graph, you can refine it. It allows us to filter the visualization based on our choice. I showed some of the above <em>(focus, hide, etc.)</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/142/1*N78BhkAMu6ppbvk7o2BuWw.png" /><figcaption>Refine options</figcaption></figure><p>You can also see other visualization options.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/162/1*93R7XsYlvst7rq_pbjndQQ.png" /><figcaption>View options</figcaption></figure><p>I showed peek and source <em>(as the list command),</em> so I want to introduce you to <a href="https://github.com/google/pprof/blob/main/doc/README.md#flame-graph">Flame Graph</a>. Flame graphs provide a high-level view of where time is spent in your code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8QtbEZhoN8_7EOlXsRNDzQ.png" /><figcaption>Flame Graph</figcaption></figure><p>Each function is represented by a colored rectangle where the width of the rectangle is proportional to the amount of time spent in that function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lGnpPcEZh-CaK_9-6mY1_g.png" /></figure><blockquote>You can find the source code <a href="https://github.com/Abdulsametileri/pprof-examples/tree/main/runtimepprof/cpu">here</a>!</blockquote><h4>2.2 Profiling Memory</h4><p>If you need to add memory profiling support to your application, you must enable it first.</p><blockquote>You can find the source code <a href="https://github.com/Abdulsametileri/pprof-examples/tree/main/runtimepprof/mem">here</a>.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zJHGeP6tIP1z40tF4SMy-Q.png" /></figure><p>If you run go run . here, you will see the generated mem.prof file. You can analyze it go tool pprof mem.prof as mentioned in the benchmarking section.</p><p>I want to introduce to you two more useful commands tree and peek .</p><ul><li>The tree shows all callers and callees of the execution flow.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QWJgfL5m1K9ahzRPbUhEDw.png" /><figcaption>tree output</figcaption></figure><p>This allows us to identify the execution flow and spot the objects that consume the most memory. <em>(Don’t forget to use </em><em>granularity=lines It provides a more readable format.)</em></p><ul><li>If you want to see a specific function execution flow, you can use peek command. For example peek expensiveFunc shows</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*M6qlcmp0SIqQ45th5hPStQ.png" /></figure><ul><li>You can also use the pprof web interface for memory analysis. Type go tool pprof -http=:6060 mem.prof , localhost:6060 will be opened.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*mpG5c-HWeQU_X15VnjzIVw.png" /><figcaption>Graph for memory analysis</figcaption></figure><blockquote>You can find the source code <a href="https://github.com/Abdulsametileri/pprof-examples/tree/main/runtimepprof/mem">here</a>!</blockquote><h3>3. Web profiling with net/http/pprof</h3><p>The <strong>runtime/pprof</strong> package offers a lower-level interface for our Go program’s performance. On the other hand, <strong>net/http/pprof</strong> provides a higher-level interface for profiling. It allows us to collect our program profiling information through HTTP 💃 All you can do is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jb6ZHrhnmuxdTvKb6p5sGw.png" /><figcaption>Adding net HTTP pprof</figcaption></figure><p>If you type localhost:5555/debug/pprof , you will see all available profiles on your browser <em>if you are not using stdlib, check </em><a href="https://docs.gofiber.io/api/middleware/pprof/"><em>fiber</em></a><em>, </em><a href="https://github.com/gin-contrib/pprof"><em>gin</em></a>,<em> or </em><a href="https://pkg.go.dev/github.com/labstack/echo-contrib/pprof"><em>echo</em></a><em> pprof implementations.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/514/1*w97DulOf-umNRBCXxVVW3Q.png" /><figcaption>debug/pprof view</figcaption></figure><p>All other <a href="https://pkg.go.dev/net/http/pprof#hdr-Usage_examples">usages</a> and <a href="https://pkg.go.dev/net/http/pprof#hdr-Parameters">parameters</a> are documented here. Let’s review popular ones.</p><h4>Getting CPU profile and Tricks</h4><blockquote>go tool pprof <a href="http://localhost:5555/debug/pprof?profile?seconds=30">http://localhost:5555/debug/pprof/profile?seconds=30</a></blockquote><p>During the CPU profiling, be aware</p><p>runtime.mallogc that → means you can optimize the number of small heap allocations.</p><p>syscall.Read or syscall.Write that → The application spends a significant amount of time in Kernel mode. You can try to do IO/buffering.</p><h4>Getting Heap (a sampling of live objects’ memory allocations) profile and Tricks</h4><blockquote><em>go tool pprof </em><a href="http://localhost:5555/debug/pprof?profile?seconds=30"><em>http://localhost:5555/debug/pprof/heap</em></a></blockquote><blockquote><em>go tool pprof </em><a href="http://localhost:5555/debug/pprof/heap?gc=1"><em>http://localhost:5555/debug/pprof/heap</em></a><a href="http://localhost:5555/debug/pprof/heap?gc=1">?gc=1</a></blockquote><p>Personally, I loved playing with GC parameters for diagnosing problems. For example, if your application has memory leak issues, you can do the following:</p><ul><li>Trigger GC <em>(Go to /debug/pprof/heap?gc=1 in your browser)</em></li><li>Download heap data; <em>give the downloaded file name -&gt; file1</em></li><li>Wait for seconds/minutes.</li><li>Trigger another GC <em>(Go to /debug/pprof/heap?gc=1 in your browser)</em></li><li>Download another heap of data; <em>give the downloaded file name -&gt; file2</em></li><li>Compare using the <a href="https://github.com/google/pprof/blob/main/doc/README.md#comparing-profiles">diff_base</a></li></ul><blockquote>go tool pprof -http=:6060 -diff_base file2 file1</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ay3MNddeoDahEYa0paUvLw.png" /><figcaption>diff_base output</figcaption></figure><h4>Getting Allocation (a sampling of all past memory allocations) profile and Tricks</h4><blockquote><em>go tool pprof </em><a href="http://localhost:5555/debug/pprof?profile?seconds=30"><em>http://localhost:5555/debug/pprof/a</em></a>llocs</blockquote><p>During the allocs profiling, you can do it</p><ul><li>If you saw bytes.growSlice , you should think about using <a href="https://pkg.go.dev/sync#Pool">sync.Pool</a> .</li><li>If you saw your functions, check whether you define fixed capacity in your slice or maps.</li></ul><h4>Investigating source code on remote pprof endpoint</h4><p>You can use -source_path and -trim_path . For example,</p><p>If you are using /app in your Dockerfile, change your directory in your project and write down</p><pre>go tool pprof -source_path=. -trim_path=/app/ https://remote-project-url/debug/pprof/allocs<br>top15<br>granularity=lines</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N84nx1Egv7MzAwQ_tWdL9g.png" /></figure><p>Thank you for reading so far 💛. All feedback is welcome 🙏 Special thanks to <a href="https://medium.com/u/df34a29d57e6">Emre Odabas</a> for the review and <a href="https://medium.com/u/7ebbb29834d">Mert Bulut</a> for encouraging me.</p><h3>Useful Links</h3><p><a href="https://github.com/google/pprof/blob/main/doc/README.md">1- pprof Github Readme</a></p><p>2- <a href="https://blog.golang.org/2011/06/profiling-go-programs.html">Profiling Go Programs by Russ Cox</a></p><p>3- p<a href="https://linux.die.net/man/1/pprof">prof man page</a></p><p>4- <a href="https://www.youtube.com/watch?v=nok0aYiGiYA">GopherCon 2019: Dave Cheney — Two Go Programs, Three Different Profiling Techniques</a></p><p>5- <a href="https://www.youtube.com/watch?v=7hg4T2Qqowk">GopherCon 2021: Felix Geisendörfer — Go Profiling and Observability from Scratch</a></p><p>6- <a href="https://www.youtube.com/watch?v=19bxBMPOlyA">GopherConAU 2019 — Alexander Else — Profiling a go service in production</a></p><p>7- <a href="https://www.practical-go-lessons.com/chap-36-program-profiling">Practical Go Lessons Profiling Chapter</a></p><h3>Stackademic 🎓</h3><p>Thank you for reading until the end. Before you go:</p><ul><li>Please consider <strong>clapping</strong> and <strong>following</strong> the writer! 👏</li><li>Follow us <a href="https://twitter.com/stackademichq"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/stackademic"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/c/stackademic"><strong>YouTube</strong></a> | <a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Discord</strong></a></li><li>Visit our other platforms: <a href="https://plainenglish.io/"><strong>In Plain English</strong></a> | <a href="https://cofeed.app/"><strong>CoFeed</strong></a> | <a href="https://differ.blog/"><strong>Differ</strong></a></li><li>More content at <a href="https://stackademic.com/"><strong>Stackademic.com</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e784526e9481" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/profiling-go-applications-in-the-right-way-with-examples-e784526e9481">Profiling Go Applications in the Right Way with Examples</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Let’s implement a basic leader election algorithm using Go with RPC ]]></title>
            <link>https://itnext.io/lets-implement-a-basic-leader-election-algorithm-using-go-with-rpc-6cd012515358?source=rss-77397c9e27------2</link>
            <guid isPermaLink="false">https://medium.com/p/6cd012515358</guid>
            <category><![CDATA[leader-election]]></category>
            <category><![CDATA[eventbus]]></category>
            <category><![CDATA[rpc]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Abdulsamet İLERİ]]></dc:creator>
            <pubDate>Mon, 07 Aug 2023 04:17:48 GMT</pubDate>
            <atom:updated>2023-08-07T12:46:19.168Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-AoVsoK4pFRGoRq0" /><figcaption>Photo by <a href="https://unsplash.com/@markusspiske?utm_source=medium&amp;utm_medium=referral">Markus Spiske</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Motivation</h3><p>When I read a book called <a href="https://www.amazon.com/Database-Internals-Deep-Distributed-Systems/dp/1492040347">Database Internals</a>, I read a chapter about leader election. During the reading, to learn more, I decided to implement the most straightforward algorithm, called the <a href="https://en.wikipedia.org/wiki/Bully_algorithm">bully algorithm</a>.</p><h4>Working Demo</h4><p><a href="https://asciinema.org/a/600162">https://asciinema.org/a/600162</a></p><h4>Source Code</h4><p><a href="https://github.com/Abdulsametileri/leader-election-bully-algorithm">https://github.com/Abdulsametileri/leader-election-bully-algorithm</a></p><h3>Introduction</h3><p>In distributed systems, a <strong>leader</strong> is a concept used to manage coordination and communication among multiple nodes or servers. Some essential functions of a leader in distributed systems include:</p><ul><li><strong><em>Consensus</em></strong>: The leader facilitates reaching an agreement (consensus) among the participating nodes about the state of the system or the data it manages.</li><li><strong><em>Coordination</em></strong>: The leader coordinates the actions of other nodes, ensuring that they all work together in harmony to achieve the system’s goals.</li><li><strong><em>Data Management</em></strong>: The leader may be responsible for managing the distribution, replication, and consistency of data across the nodes in the system.</li><li><strong><em>Failover and High Availability</em>:</strong> In the case of node failures, the leader may be responsible for detecting failures and initiating failover processes to ensure the system remains operational and highly available.</li></ul><h3>How does it work?</h3><p>In the <a href="https://en.wikipedia.org/wiki/Bully_algorithm">bully algorithm</a>, the fundamental idea is <strong>rank</strong>. It assumes that every node has a rank within the cluster, and the leader must be the highest. So it uses the node’s rank value during the election.</p><p>There are two situations for election.</p><ul><li>The system is newly initialized, so there is no leader</li><li>One of the nodes notices that the leader is down.</li></ul><p>The election is implemented as follows:</p><ol><li>The node sends <strong>ELECTION</strong> messages to the other nodes with higher ranks than their own.</li><li>The node waits for <strong>ALIVE</strong> responses.</li></ol><ul><li>If no higher-ranked node responds, it makes itself a leader.</li><li>Otherwise, it is notified of the new leader who has the highest rank.</li></ul><p>Let’s illustrate these scenarios:</p><p>We assumed that the highest rank order, like: <strong>node-04 &gt; node-03 &gt; node-02 &gt; node-01</strong></p><h4>If the system is newly initialized</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RC97nP4OuDqYjhgnRWsM5A.png" /><figcaption>Figure: Election state</figcaption></figure><p>Because node-04 is the highest-ranked node in this cluster, it didn’t get any <strong>ALIVE</strong> messages, so node-04 became a leader and broadcasted the <strong>ELECTED</strong> message to notify other nodes about the election results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CVz9hfCqW0utuIYyUzRVGw.png" /><figcaption>Figure: Election result</figcaption></figure><h4>When the leader is down</h4><p>The other nodes periodically send <strong>PING</strong> messages to notice the leader is down and wait for the leader’s <strong>PONG</strong> responses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BmeAnDCD0vnoRNtd1VxNJA.png" /><figcaption>Figure: Ping-Pong step</figcaption></figure><p>If the leader is down and the first node doesn’t get a <strong>PONG</strong> message, that node starts the election process again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HcPPpeiyAWLzBH4hPHoYZw.png" /><figcaption>Figure: Electing a new leader</figcaption></figure><h4>Problems in this algorithm</h4><ul><li>One of the apparent problems with this algorithm is that it violates the <strong>safety</strong> guarantee (that, at most, one leader can be elected at a time) in the presence of <strong>network partitions</strong>. It is easy to end up where nodes split into two or more independently functioning subsets, and each subset elects its leader. This situation is called <strong>split brain</strong>. <a href="https://www.amazon.com/Database-Internals-Deep-Distributed-Systems/dp/1492040347">[1]</a></li><li>Another problem with this algorithm is a strong preference toward high-ranked nodes, which becomes an issue if they are unstable and can lead to a permanent state of reelection. An unstable high-ranked node proposes itself as a leader, fails shortly after that, wins reelection, fails again, and the whole process repeats. <a href="https://www.amazon.com/Database-Internals-Deep-Distributed-Systems/dp/1492040347">[2]</a> But this is not specific to the bully algorithm.</li></ul><h3><strong>Implementation</strong></h3><p>I used <a href="https://github.com/Abdulsametileri/leader-election-bully-algorithm/blob/main/docker-compose.yml">docker compose</a> to simulate a couple of nodes in the cluster. All nodes are based on <a href="https://github.com/Abdulsametileri/leader-election-bully-algorithm/blob/main/Dockerfile">this docker file</a>.</p><p>To implement the algorithm, <strong>every node must be aware of each other</strong>. This requires a service discovery implementation. I didn’t want to implement it, so at the beginning, <strong>each node was aware of the network information</strong> of other nodes. <em>In the past, I implemented </em><a href="https://github.com/Abdulsametileri/simple-service-discovery"><em>a basic service discovery mechanism</em></a><em> and wrote </em><a href="https://itnext.io/lets-implement-basic-service-discovery-using-go-d91c513883f6"><em>an article</em></a><em> about it; if you are interested in the concept, you can check it out :)</em></p><p>I hardcoded all nodes as follows</p><pre>// nodeAddressByID: It includes nodes currently in cluster<br>var nodeAddressByID = map[string]string{<br> &quot;node-01&quot;: &quot;node-01:6001&quot;,<br> &quot;node-02&quot;: &quot;node-02:6002&quot;,<br> &quot;node-03&quot;: &quot;node-03:6003&quot;,<br> &quot;node-04&quot;: &quot;node-04:6004&quot;,<br>}</pre><p>at the startup based on this map; every node tries to send a PING message to the others.</p><pre>func (node *Node) ConnectToPeers() {<br> for peerID, peerAddr := range nodeAddressByID {<br>  if node.IsItself(peerID) {<br>   continue<br>  }<br><br>  rpcClient := node.connect(peerAddr)<br>  pingMessage := Message{FromPeerID: node.ID, Type: PING}<br>  reply, _ := node.CommunicateWithPeer(rpcClient, pingMessage)<br><br>  if reply.IsPongMessage() {<br>   node.Peers.Add(peerID, rpcClient)<br>  }<br> }<br>}</pre><p>After awareness of nodes, the election process is started. In this step, the node sends an <strong>ELECTION</strong> message to the other nodes with a higher rank than its own. If no higher than a node is available, it elects itself a leader and broadcasts an <strong>ELECTED</strong> message.</p><pre>func (node *Node) Elect() {<br> isHighestRankedNodeAvailable := false<br><br> peers := node.Peers.ToList()<br> for i := range peers {<br>  peer := peers[i]<br><br>  if node.IsRankHigherThan(peer.ID) {<br>   continue<br>  }<br><br>  electionMessage := Message{FromPeerID: node.ID, Type: ELECTION}<br>  reply, _ := node.CommunicateWithPeer(peer.RPCClient, electionMessage)<br><br>  if reply.IsAliveMessage() {<br>   isHighestRankedNodeAvailable = true<br>  }<br> }<br><br> if !isHighestRankedNodeAvailable {<br>  leaderID := node.ID<br>  electedMessage := Message{FromPeerID: leaderID, Type: ELECTED}<br>  node.BroadcastMessage(electedMessage)<br> }<br>}</pre><p>As soon as the leader is elected, the <a href="https://github.com/Abdulsametileri/leader-election-bully-algorithm/blob/main/node.go#L93">LeaderElected event is fired</a> via the <a href="https://github.com/Abdulsametileri/leader-election-bully-algorithm/blob/main/event/bus.go">event bus</a>, and because of <a href="https://github.com/Abdulsametileri/leader-election-bully-algorithm/blob/main/node.go#L35">the subscribe step</a>, the nodes start to send PING messages continuously.</p><pre>func (node *Node) PingLeaderContinuously(_ string, payload any) {<br> leaderID := payload.(string)<br><br>ping:<br> pingMessage := Message{FromPeerID: node.ID, Type: PING}<br> reply, err := node.CommunicateWithPeer(leader.RPCClient, pingMessage)<br> if err != nil {<br>  log.Info().Msgf(&quot;Leader is down, new election about to start!&quot;)<br>  node.Peers.Delete(leaderID)<br>  node.Elect()<br>  return<br> }<br><br> if reply.IsPongMessage() {<br>  time.Sleep(3 * time.Second)<br>  goto ping<br> }<br>}</pre><p>During this communication, if the leader is down, its connection is dropped and returns an error so that a new election process will start. You can <a href="https://asciinema.org/a/600162">check the live demo</a>.</p><blockquote>In the implementation of RPC, I used this standard library package and its very useful, for more information <a href="https://pkg.go.dev/net/rpc#pkg-overview">check it documentation out</a>.</blockquote><p>You can also run the project locally, as described <a href="https://github.com/Abdulsametileri/leader-election-bully-algorithm">in the repository</a>.</p><p>Thank you, <a href="https://medium.com/u/df5e62b48c4a">Eray Arslan</a>, for the review. 💛</p><p>Thank you for reading 💛; you can also check my other Let’s Implement articles. ☀️</p><p><a href="https://medium.com/@abdulsamet-ileri/how-was-i-build-a-version-control-system-vcs-using-pure-go-83ec8ec5d4f4">Let’s implement a version control system (VCS) using Go 🚀</a></p><p><a href="https://abdulsamet-ileri.medium.com/lets-implement-basic-service-discovery-using-go-d91c513883f6">Let’s implement basic service discovery using Go 🚀</a></p><p><a href="https://abdulsamet-ileri.medium.com/lets-implement-a-real-time-package-tracking-app-with-rabbitmq-and-web-socket-using-go-80f5a5ca5c55">Let’s implement a real-time package tracking app with RabbitMQ and Web socket using Go 🚀</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6cd012515358" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/lets-implement-a-basic-leader-election-algorithm-using-go-with-rpc-6cd012515358">Let’s implement a basic leader election algorithm using Go with RPC 🚀</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kafka Konsumer  ]]></title>
            <link>https://medium.com/trendyol-tech/kafka-konsumer-c47b4b8c1599?source=rss-77397c9e27------2</link>
            <guid isPermaLink="false">https://medium.com/p/c47b4b8c1599</guid>
            <category><![CDATA[exception-handling]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[Abdulsamet İLERİ]]></dc:creator>
            <pubDate>Fri, 07 Jul 2023 10:45:07 GMT</pubDate>
            <atom:updated>2023-10-07T17:21:34.189Z</atom:updated>
            <content:encoded><![CDATA[<h4>Easy implementation of Kafka consumer with built-in exception manager 💃</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*d83by28ZqaAZq8Gt.png" /></figure><p>As a Trendyol Indexing Team, our architecture is heavily based on an event-based system, so Kafka takes part at the center of this system. Approximately <strong>350 million events</strong> are processed daily within ~15 different Kafka consumer projects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LaWZ20Xj7Gen4kfAzHHL5w.png" /><figcaption>Figure: Daily processed events</figcaption></figure><p>In these consumer projects, we must implement</p><ul><li>boilerplate Kafka initialization codes</li><li>to handle exceptions with an eventual strategy</li><li>exposing processed event metrics, defining alerts, and more!</li></ul><p>It becomes quite difficult for many such projects after a certain period. To find a solution, we searched existing open-source projects but didn’t find anything, so we decided to develop a library <em>(</em><a href="https://github.com/Trendyol/kafka-konsumer"><em>kafka-konsumer</em></a><em>) </em><a href="https://github.com/Trendyol">under the Trendyol organization</a>,<em> </em>accommodating all needs. 👊 Thanks to this library, we start solving security problems more effortlessly, have a consistent codebase, and easily start new consumer projects and maintain old ones. 💃</p><p>We have previously developed a library <em>(</em><a href="https://github.com/Trendyol/kafka-cronsumer"><em>kafka-cronsumer</em></a><em>)</em> that implements the exception strategy these consumer projects need. You can check <a href="https://medium.com/trendyol-tech/kafka-exception-c-r-onsumer-37c459e4849d">this article</a> and review <a href="https://github.com/Trendyol/kafka-cronsumer">its repository</a> to learn how it works. This library worked in production without any problems and was very successful. 😋</p><p>We wanted to extend this behavior not only to exceptional cases but also to standard consuming processes! As a result of this, kafka-konsumer was born! 🤔</p><p>It takes Kafka connection information and a consume function; that&#39;s it. It allows us to focus only on business logic. Exception handling and exposing metrics come built-in 🚀. We will look more closely at the following.</p><h3>Features 🚀</h3><ul><li>Built-in retry/exception handling support</li><li>High-performance batch-consuming support: in the implementation phase, we are using the same strategy <a href="https://medium.com/trendyol-tech/new-winner-of-kafka-consumers-scala-to-go-journey-604c6bdd7041">mentioned in this article</a> 🚀</li><li>Distributed tracing support on consumption and produce operations 🚀</li><li>Exposed Several Metrics</li><li>Scram — Plain Text Authentication &amp; TLS Support</li><li>Kafka Producer Support</li></ul><h3>How to use it? 🛠</h3><h4>Simple Consumer</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b0eb7271580bd9d1819760e704e74b27/href">https://medium.com/media/b0eb7271580bd9d1819760e704e74b27/href</a></iframe><h4>Simple Consumer With Retry/Exception Option</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/51539055d50728b6f1eb016ecc83b80c/href">https://medium.com/media/51539055d50728b6f1eb016ecc83b80c/href</a></iframe><h4>Batch Consuming</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/baaf5b1d2bb38d4e53314c7a0b8865b4/href">https://medium.com/media/baaf5b1d2bb38d4e53314c7a0b8865b4/href</a></iframe><h4>With Grafana &amp; Prometheus</h4><p>Currently, we are exposing the following metrics:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*RzsNcLFG3S5fY3u3KrAXfA.png" /><figcaption>Figure: Exposed metrics</figcaption></figure><p>It is possible to create a Grafana dashboard and alert rules on them, for example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jl7HYS7pjYlgow0P7Oxszg.png" /><figcaption>Figure: Grafana Dashboard example</figcaption></figure><pre>groups:<br>  - name: konsumer-alerts<br>    rules:<br>      - alert: UnprocessedMessageIncreasing<br>        expr: increase(kafka_konsumer_unprocessed_messages_total{job=&quot;konsumer&quot;}[5m]) &gt; 0<br>        for: 5m<br>        labels:<br>          severity: &quot;critical&quot;<br>        annotations:<br>          summary: &quot;Kafka Konsumer unprocessed message increasing&quot;<br>          description: &quot;Kafka Konsumer unprocessed message increasing, current value: {{ $value }} &lt;http://localhost:3000/d/DlIdtG_4z/kafka-konsumer-dashboard|Grafana&gt;&quot;</pre><blockquote>You can also find several ready-to-run examples in <a href="https://github.com/Trendyol/kafka-konsumer/tree/main/examples">this directory.</a></blockquote><h3>Conclusion</h3><p>At the time of writing this article, we have been using this library in production for a few months without any problems 😌 We are also ready to write new consumer projects within 1 hour 😅 Thanks to all contributors <a href="https://medium.com/u/df34a29d57e6">Emre Odabas</a>, <a href="https://medium.com/u/f9d1098b2ee9">Anıl Mısırlıoğlu</a>, <a href="https://medium.com/u/51bc817a3d69">Mehmet Sezer</a>, <a href="https://medium.com/u/7b7e307ee151">Elif Seray Dönmez Çelik</a>, <a href="https://medium.com/u/e1bf8fd5491">Nihat Alim</a>.</p><p>We are open to hearing your voice and feedback. You can create an issue or contact me or <a href="https://medium.com/u/df34a29d57e6">Emre Odabas</a>. Thank you for reading 🎈</p><ul><li><a href="https://github.com/Trendyol/kafka-konsumer">GitHub - Trendyol/kafka-konsumer: Easy implementation of kafka consumer with built-in exception manager (kafka-cronsumer)</a></li><li><a href="https://github.com/Trendyol/kafka-cronsumer">GitHub - Trendyol/kafka-cronsumer: Cron based Kafka exception consumer with the power of auto retry &amp; concurrency</a></li></ul><p>You can also check out other articles from Trendyol Tech below</p><p><a href="https://medium.com/trendyol-tech">Trendyol Tech</a></p><p>We love to share our knowledge. If you like to “<em>share”</em> with us, you could apply for our <a href="https://jobs.lever.co/trendyol?lever-via=q48u5LZ7k7">open positions</a>.</p><p><a href="https://jobs.lever.co/trendyol">Trendyol jobs</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c47b4b8c1599" width="1" height="1" alt=""><hr><p><a href="https://medium.com/trendyol-tech/kafka-konsumer-c47b4b8c1599">Kafka Konsumer 🔥 🚀</a> was originally published in <a href="https://medium.com/trendyol-tech">Trendyol Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to use Consul-based Service Discovery in Prometheus ]]></title>
            <link>https://abdulsamet-ileri.medium.com/how-to-use-consul-based-service-discovery-in-prometheus-869ff04b357c?source=rss-77397c9e27------2</link>
            <guid isPermaLink="false">https://medium.com/p/869ff04b357c</guid>
            <category><![CDATA[service-discovery]]></category>
            <category><![CDATA[prometheus]]></category>
            <dc:creator><![CDATA[Abdulsamet İLERİ]]></dc:creator>
            <pubDate>Sun, 21 May 2023 19:29:27 GMT</pubDate>
            <atom:updated>2023-05-22T05:52:46.800Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bsWIoNQEJEcm7wiWUQuD0g.png" /><figcaption>This picture was taken from <a href="https://sysdig.com/blog/consul-monitoring/">this article</a>.</figcaption></figure><p>Hello all! In this article, I will explain why we need to integrate service discovery tools like (Consul) within <a href="https://prometheus.io/">Prometheus</a> with an example.</p><p>In this example, we’ll dive in step by step on how we achieve observing an API with Prometheus.</p><h4>Prerequisites</h4><ul><li>You can install Prometheus via <a href="https://github.com/prometheus/prometheus/releases">its GitHub repository.</a></li><li>You can install Consul <a href="https://releases.hashicorp.com/consul/1.15.2/">here</a>.</li><li>You can look at <a href="https://github.com/Abdulsametileri/service-discovery-prometheus-consul/blob/main/main.go">our API</a>. It is nothing but has /metrics endpoint. Prometheus will use this endpoint for scraping when we are configured. You can find the <a href="https://github.com/Abdulsametileri/service-discovery-prometheus-consul">source code</a> here!</li><li>If you don’t want to install Prometheus and Consul locally, you can use <a href="https://github.com/Abdulsametileri/service-discovery-prometheus-consul/blob/main/docker-compose.yml">the docker-compose file</a>. Be aware some configurations are different because of docker networking.</li></ul><p>First, before starting Prometheus, we must modify its configuration file, aka prometheus.yml. In default, this file configured as Prometheus scrapes itself for its health. I change it with my API address. Our API <a href="https://github.com/Abdulsametileri/service-discovery-prometheus-consul/blob/main/main.go#L12">runs port 8080 as default</a>. Therefore, I am adding localhost:8080 as shown below:</p><pre>scrape_configs:<br>  - job_name: &quot;api&quot;<br>    metrics_path: /metrics<br>    scheme: http<br>    static_configs:<br>      - targets: [&quot;localhost:8080&quot;]</pre><p>If we start Prometheus and go to localhost:9090/targets, we can easily see our API instance up, and Prometheus successfully starts its scrapes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*dG58aTjlOF5fpmGtCTNHrg.png" /><figcaption>Figure: Prometheus targets endpoint</figcaption></figure><p>In summary, it works as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QsDL_EXafKB80JFinDEOSQ.png" /><figcaption>Figure: Overall design</figcaption></figure><p>Okay fine! 👍 Clear and concise! What about we want to increase our API instance and want Prometheus scrapes them 🤔</p><p>No problem, we need to write the other API&#39;s address to prometheus.yml and restart Prometheus. 😄</p><pre>scrape_configs:<br>  - job_name: &quot;api&quot;<br>    metrics_path: /metrics<br>    scheme: http<br>    static_configs:<br>      - targets: [<br>        &quot;localhost:8080&quot;,<br>        &quot;localhost:8081&quot;,<br>        &quot;localhost:8082&quot;<br>      ]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1003/1*53ZqUxPYqwQEMCSBPW-uuw.png" /><figcaption>Figure: All API instances can be seen from Prometheus targets ui</figcaption></figure><p>Well! However, there is a problem here. If we run new API instances or stop older ones, we need to configure the configuration file repeatedly. In today&#39;s world, this manual work is challenging, even impossible. 😞</p><p>With the breakthrough of services, the increasing number of users, requests, and demands have made this job very difficult. Moreover, our services are constantly changing due to various situations, such as <a href="https://en.wikipedia.org/wiki/Autoscaling">autoscaling</a>, failures, and upgrades. As a result of these changes, they continuously get their new IPs. 😞</p><p>This is where service discovery enters the scene. And Prometheus has built-in support for it 😌😍. For example:</p><ul><li>Cloud and VM providers (AWS EC2, Google GCE, Microsoft Azure)</li><li>Cluster schedulers (Kubernetes)</li><li>Generic mechanisms (DNS, Consul, Zookeeper)</li><li>File-based custom service discovery.</li></ul><p>In the following, we will see how to configure consul-based discovery! 👊🚀</p><h3>Setting Up Consul-Based Discovery</h3><p><a href="https://github.com/hashicorp/consul">Consul</a> is a tool for discovering and configuring services. It provides an API that allows clients to register and discover services. And also can perform health checks to determine service availability. From our article&#39;s scope of view, we don&#39;t dive into more detail about Consul.</p><p>For simplicity, we create a service configuration file (consul.json) and give it when we start Consul. This configuration is shown below.</p><pre>{<br>  &quot;services&quot;: [<br>    {<br>      &quot;id&quot;: &quot;api1&quot;,<br>      &quot;name&quot;: &quot;api&quot;,<br>      &quot;address&quot;: &quot;127.0.0.1&quot;,<br>      &quot;port&quot;: 8080<br>    },<br>    {<br>      &quot;id&quot;: &quot;api2&quot;,<br>      &quot;name&quot;: &quot;api&quot;,<br>      &quot;address&quot;: &quot;127.0.0.1&quot;,<br>      &quot;port&quot;: 8081<br>    },<br>    {<br>      &quot;id&quot;: &quot;api3&quot;,<br>      &quot;name&quot;: &quot;api&quot;,<br>      &quot;address&quot;: &quot;127.0.0.1&quot;,<br>      &quot;port&quot;: 8082<br>    }<br>  ]<br>}</pre><p>First, start our Consul with development mode as</p><p>./consul agent -dev -config-file=consul.json</p><p>Second, configure prometheus.yml like as</p><pre>scrape_configs:<br>  - job_name: &quot;consul-example&quot;<br>    consul_sd_configs:<br>      - server: &#39;localhost:8500&#39;<br>    relabel_configs:<br>      - action: keep<br>        source_labels: [__meta_consul_service]<br>        regex: api</pre><p>That’s it. Let’s look more detail about this configuration 🤔</p><p>First of all, we are using Prometheus&#39; <a href="https://grafana.com/blog/2022/03/21/how-relabeling-in-prometheus-works/">relabelling future</a> here! Prometheus uses this feature to filter and modify targets based on some rules. And It is also useful for preparing Grafana dashboards.</p><p>In our case, we want Prometheus to scrape only come from our consul service targets. Therefore we are specifying services (targets) whose names <a href="https://github.com/Abdulsametileri/service-discovery-prometheus-consul/blob/main/consul.json#L5">are <strong>api</strong></a><strong>. </strong>Otherwise, Prometheus would also try to scrape the Consul agent, which doesn&#39;t have a <strong>metrics endpoint.</strong></p><p><em>ℹ️ Service discovery mechanisms can provide a set of labels starting with </em><strong><em>__meta_ </em></strong><em>with more metadata about the target specific to the discovery method. In our situation, we are using consul_service. If we use the k8s discovery engine, it will be </em><strong><em>__meta_kubernetes_pod_name, __meta_kubernetes_pod_ready,</em></strong><em> etc.</em></p><p>And boom 🔥</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p-ESsQV-hK3j15pRlXrCsA.png" /><figcaption>Figure: Consul UI</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dw9ZInonHLrazQm97FezCw.png" /><figcaption>Figure: Prometheus Target UI</figcaption></figure><p>We only specify the consul agent address, and no need to configure prometheus.yml again and again. Once our services register to the Consul, Prometheus will use them easily. Also, in this way, we are supporting hot reload. No need to keep track of new services or old services; the consul does this on our behalf of us.</p><p>You can see how to integrate a service discovery mechanism in Prometheus easily 💃</p><p>Thank you for reading 💛; you can also check my other articles. ☀️</p><ul><li><a href="https://abdulsamet-ileri.medium.com/boosting-your-kafka-integration-tests-using-redpanda-with-go-247e4276c61d">Boosting your Kafka integration tests using Redpanda with Go</a></li><li><a href="https://medium.com/trendyol-tech/kafka-exception-c-r-onsumer-37c459e4849d">Kafka Exception Cronsumer</a></li><li><a href="https://itnext.io/lets-implement-a-real-time-package-tracking-app-with-rabbitmq-and-web-socket-using-go-80f5a5ca5c55">Let&#39;s implement a real-time package tracking app with RabbitMQ and Web socket using Go</a></li><li><a href="https://itnext.io/lets-implement-basic-service-discovery-using-go-d91c513883f6">Let&#39;s implement basic service discovery using Go</a></li><li><a href="https://levelup.gitconnected.com/how-was-i-build-a-version-control-system-vcs-using-pure-go-83ec8ec5d4f4">How I built a version control system (VCS) using pure Go</a></li><li><a href="https://medium.com/modanisa-engineering/integrating-grafana-notifications-with-gitlab-pipeline-to-restart-debezium-tasks-using-go-1378c9eaf7b8">Integrating Grafana Notifications with GitLab Pipeline to restart Debezium tasks using Go</a></li><li><a href="https://medium.com/codex/introduction-to-keycloak-227c3902754a">Introduction to Keycloak</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=869ff04b357c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[New Winner of Kafka Consumers: Scala to Go Journey ]]></title>
            <link>https://medium.com/trendyol-tech/new-winner-of-kafka-consumers-scala-to-go-journey-604c6bdd7041?source=rss-77397c9e27------2</link>
            <guid isPermaLink="false">https://medium.com/p/604c6bdd7041</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[concurrency]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[scala]]></category>
            <dc:creator><![CDATA[Abdulsamet İLERİ]]></dc:creator>
            <pubDate>Tue, 18 Apr 2023 08:33:06 GMT</pubDate>
            <atom:updated>2023-07-10T11:55:25.835Z</atom:updated>
            <content:encoded><![CDATA[<p>350 Million Messages per Day</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mWygFruwViMjEj9G.png" /><figcaption>Picture Taken from <a href="https://blog.devgenius.io/series-intro-to-managing-golang-kafka-consumer-lag-41b36d3a14b9">this article</a></figcaption></figure><p>Co-Authors: <a href="https://medium.com/@esdonmez">Elif Seray Dönmez Çelik</a>, <a href="https://nihatalim.medium.com/">Nihat Alim</a></p><p>In this article, you will read a story about how we decreased memory usage by ~50<strong>%</strong>, increased CPU Efficiency by ~%80 and TP by ~%20 by migrating from Scala to Go.</p><h3><strong>Introduction</strong></h3><p>Trendyol has more than 170 million sellable and 200 million ready-to-sell products right now. At any time, there may be a change, or invalidation, as we call it, on these products via different events like promotion, stock, and price. By changes, we mean millions of them, approximately <strong>350 million a day</strong> if we need to be precise. As the Trendyol Indexing Team, we need to apply these changes almost near real-time because any latency could cause product data to be displayed incorrectly. We don’t even want to think about writing the price of a product wrong.</p><p>In the middle of all these changes is our application, <strong>Perses</strong>. In Greek mythology, Perses was the Greek Titan God of Destruction. Like the Destruction God, our application Perses destroys the old product data and replaces it with its new version.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*et5NvPOyYm4Gv0cfqQc1yQ.png" /><figcaption>Figure: Perses Design</figcaption></figure><p>As we can see from the daily throughput graphic of Perses below, Perses does millions of I/O operations daily. It is crucial to perform them correctly and without any latency to be able to show the users the correct product data. To achieve our purpose of having a high-performant application, Perses is designed as a multi-deployment application so that every deployment can scale independently and does not block each other invalidation process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*RxYZEXXgEjiScC5X.png" /><figcaption>Figure: Perses’ Daily Throughput</figcaption></figure><p>From all of these, we can easily say that Perses plays an important role in <a href="https://medium.com/trendyol-tech/evolution-of-our-invalidation-system-bdfee456498b">Trendyol’s invalidation process</a>.</p><p>There were several reasons why we made this migration decision.</p><ul><li>Previously, getting better results in terms of resource usage and performance by migrating our other smaller consumer projects.</li><li>Learning and maintaining Go was easier than maintaining old Scala projects for our team.</li></ul><h3>Our Implementation Steps</h3><p>We will explain our re-platforming journey in 5 steps.</p><h4><strong>1. Without monitoring, you cannot solve murders.</strong></h4><p>In Scala Perses, the consuming operation was made in batches using Akka Stream <em>(v2.12)</em>. You can see the implementation in the code block below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tUrhKpgnEHVUneV1dtyKhA.png" /></figure><p>We tested and compared these two as soon as we implemented batch consuming in Go Perses <em>(</em><a href="https://github.com/segmentio/kafka-go"><em>via kafka-go</em></a><em>)</em>, but the results shocked us. While Scala Perses was processing 13k messages in a minute per pod, Go could only process 4k. 😢</p><p>After seeing the results, we monitored the application and saw the bottleneck in the <a href="https://github.com/segmentio/kafka-go#writer">Kafka producer part</a>. When we dived into the codebase of the producer <a href="https://github.com/segmentio/kafka-go">in the Kafka library</a>, we realized that the producing operation was done synchronously. It means that we were going to the broker for every message, which was harmful to the performance.</p><h4><strong>2. Caught the killer! Wait, there are more!?</strong></h4><p>We decided to change the producer implementation from synchronous to asynchronous and to in batches. The message queue contained messages sent to two different topics at that time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-KOMxWby4L2Y98ZGJx3mXg.png" /></figure><p>After completing the implementation, we ran a load test immediately, but the results were still disappointing. Go Perses processed 9k messages in a minute per pod, but we still didn’t reach Scala Perses 13k. 😞</p><h4>3. Following the breadcrumbs</h4><p>We separated the message channel for our topics. We had only two topics that we needed to produce back then, so we created two message channels and one Goroutine per channel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*ro4qQmePfIxw7GwMoqGT9Q.png" /></figure><p>After this change, we tried different “batch size” and “batch duration” parameters to achieve the best performance. After several tries, we saw that the optimal values were 500 for batch size and 500ms for the batch duration. When we ran a load test again with the final parameters with 36 partitions and 12 pods, we processed 1.094.800 messages in 10 minutes with 136k throughput. Scala was still more performant with its 156k throughput, it could process the same number of messages in 8 minutes.🤔</p><h4>4. It’s not a dead end</h4><p>We are using <a href="https://github.com/uber-go/automaxprocs">Uber’s automaxprocs</a> library for one of our other Go projects and saw performance enhancement, so we also wanted to try it for Go Perses. Unfortunately, throughput was not changed because Perses is more i/o bound than CPU bound. 😢</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JRmeVRQ2yueYFtmj9iI3AA.png" /></figure><h4>5. The sweet smell of victory</h4><p>Because we still couldn’t get the desired results, we decided to focus on our architectural design to figure out what to do.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f3pDk2hnUzcbUcMgJGy9Mg.png" /></figure><p>In the first part of the architecture, a goroutine continuously listens to a topic and fetches new messages using an internal message queue. We think we could try <a href="https://pkg.go.dev/github.com/segmentio/kafka-go@v0.4.39#ReaderConfig.QueueCapacity">tuning the queue size here</a>, so we decided to configure it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/1*AJcZ_i14uRRm6HxOuOruPQ.png" /></figure><p>In the second part of the architecture, after the fetched messages come to the channel, the goroutines process them. Here, we thought we could tune the <a href="https://stackoverflow.com/questions/11943841/what-is-channel-buffer-size">channel buffer size</a> and the number of worker goroutines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/637/0*qi5Av6vT-3abgBBY.png" /></figure><p>We were surprised by the results after changing the configurations in the tuning process 💃💃. You can see our trials and their results in the table below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9OYqhgawTe3G4aEAV67HDg.png" /></figure><p>You can see the comparison of the most performant Scala Perses and Go Perses results below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zubCCjtAoypmOp7v.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/0*ZTsDTzrKC_eg2lNz.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/0*tajkrJ1pm_ty2mGe.png" /><figcaption>Figure: Scala Perses vs Go Perses Throughput Comparison</figcaption></figure><h3>Conclusion</h3><p>As a result of the migration process, we optimized the following resources</p><ul><li>Memory usage: ~50% Decrease (from 1.127 GB to 0.622 GB)</li><li>CPU: ~80% Efficient (from 1.72 to 0.379)</li><li>TP: ~20% Increase (from 156k to 189k)</li></ul><p>We are very pleased to make this optimization on a multiple-deployment codebase that works under high load.</p><p>What we did learn!</p><ul><li>We realized how important monitoring is when it helps us to address the kafka producer problem.</li><li>By diving into our project design, we realized some critical parameters still needed to be tuned to improve performance.</li></ul><p>Thanks for reading 🎈🎈🎈</p><p>We open-sourced our experience <a href="https://github.com/Trendyol/kafka-cronsumer">with a built-in retry manager</a> at <a href="https://github.com/Trendyol/kafka-konsumer">kafka-konsumer.</a> You can find more detail in <a href="https://medium.com/@abdulsamet-ileri/kafka-konsumer-c47b4b8c1599">its article</a>. 🚀</p><p>We are excited to get your feedback. 🙏</p><p>Thank you to <a href="https://medium.com/u/c82aee539dc2">Kutlu Araslı</a>, <a href="https://medium.com/u/df34a29d57e6">Emre Odabas</a>, and <a href="https://medium.com/u/7ebbb29834d">Mert Bulut</a> for their help 💚</p><p>If you’re interested in joining our team, you can apply for the backend developer role or any of our current <a href="https://jobs.lever.co/trendyol?lever-via=OUSPhixG90">open positions</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=604c6bdd7041" width="1" height="1" alt=""><hr><p><a href="https://medium.com/trendyol-tech/new-winner-of-kafka-consumers-scala-to-go-journey-604c6bdd7041">New Winner of Kafka Consumers: Scala to Go Journey 🚀</a> was originally published in <a href="https://medium.com/trendyol-tech">Trendyol Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Boosting your Kafka integration tests using Redpanda with Go]]></title>
            <link>https://levelup.gitconnected.com/boosting-your-kafka-integration-tests-using-redpanda-with-go-247e4276c61d?source=rss-77397c9e27------2</link>
            <guid isPermaLink="false">https://medium.com/p/247e4276c61d</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[testcontainer]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[red-panda]]></category>
            <dc:creator><![CDATA[Abdulsamet İLERİ]]></dc:creator>
            <pubDate>Thu, 17 Nov 2022 10:10:17 GMT</pubDate>
            <atom:updated>2022-11-22T01:57:40.778Z</atom:updated>
            <content:encoded><![CDATA[<h3>Boosting your Kafka integration tests using Redpanda with Go 🚀</h3><h4>With the example of testcontainers and dockertest</h4><blockquote>Integration tests are choreography tests. They do not test business rules. Rather, they test how well the assembly of components dances together. They are plumbing tests that make sure that the components are properly connected and can clearly communicate with each other. <a href="https://www.amazon.com/Clean-Coder-Conduct-Professional-Programmers/dp/0137081073">[1]</a></blockquote><p>Hello, all! In this article, I want to talk about <a href="https://redpanda.com/"><strong>Redpanda</strong></a> and how we will implement it in our integration tests with the example of two popular libraries; <a href="https://github.com/testcontainers/testcontainers-go"><strong>testcontainers</strong></a> and <a href="https://github.com/ory/dockertest"><strong>dockertest</strong></a>.</p><blockquote><strong>TL;DR</strong>: <a href="https://github.com/Abdulsametileri/integration-tests-with-redpanda">https://github.com/Abdulsametileri/integration-tests-with-redpanda</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/1*yv5Oeof8zv7wRrwWZlZHrQ.png" /><figcaption>Figure: Gophers want to learn how to implement Redpanda</figcaption></figure><h4>What is Redpanda 🤔</h4><p><strong>Redpanda</strong> is a Kafka®-compatible streaming data platform that is up to 10x faster and 6x more hardware-efficient. It is also JVM-free, ZooKeeper®-free, <a href="https://redpanda.com/blog/redpanda-official-jepsen-report-and-analysis/">Jepsen-tested</a>, and source available. <a href="https://redpanda.com/">[2]</a></p><h4>Why should I use Redpanda 🤔</h4><p>With these excellent properties of Redpanda, we can easily use it in our tests without running Kafka and Zookeeper containers. (Better startup time and lack of failure). Also, thanks to its compatibility, there is no need to change any production code in which Kafka resides.</p><p>As we already know, writing and maintaining integration tests is tough. We want to get the output of these tests as fast as possible. After replacing our integration test with Redpanda, our tests worked extremely fast compared to old Kafka tests. So I want to share with you 🔥</p><p>Before going into the implementation details, I want to inform you; I will use a beautiful testing library <a href="https://github.com/stretchr/testify">testify</a> <em>(the suite and assert packages)</em>. So if you don&#39;t have any experience, I strongly recommend you to check out.</p><p>Note: I will create a wrapper struct for each <em>(TestContainerWrapper, DockerTestWrapper)</em>and write mainly three methods we need: RunContainer , CleanUp , GetBrokerAddresses .</p><p>Note 2: I want to test two functionality to ensure that my application can produce a Kafka message and consume a Kafka message successfully.</p><p>Let&#39;s get started!</p><h4>👉 Implementation with testcontainers</h4><p>First, before our integration tests start to work, we must ensure that our setup process is done and ready to move on.</p><p>Our test suite is shown below. We created TestContainerWrapper struct and used it with Suite lifecycle methods.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5c7b9255f9b132c7778a1c778112c010/href">https://medium.com/media/5c7b9255f9b132c7778a1c778112c010/href</a></iframe><p>Firstly, we can focus on RunContainer implementation.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/56b1f7340fc8bff17660d95ff66fdcf6/href">https://medium.com/media/56b1f7340fc8bff17660d95ff66fdcf6/href</a></iframe><p>After providing the Redpanda image and version, we expose the <strong>9092</strong> port. <strong>Be careful; this exposed port number is from the container&#39;s perspective! </strong>From the host&#39;s perspective<em>,</em> Testcontainers exposes this on a random free port. This is designed to avoid port collisions that may arise with locally running software or in between parallel test runs. <a href="https://www.testcontainers.org/features/networking/">[3]</a>. So we need to get the mapped port <strong>9092</strong> to connect the brokers from our host machine.</p><p>We use default redpanda start command. There are a couple of different parameters; it depends on what you are testing, etc. Redpanda container starts with <em>— dev-container</em>. With this mode, <a href="https://github.com/redpanda-data/redpanda/blob/36902e558294680e6d89e501a80e4e09811bc815/src/go/rpk/pkg/cli/cmd/redpanda/start.go#L1062">a couple of other parameters are passed by default.</a></p><p>I passed true to the AutoRemove flag, so the container will be removed from the host when stopped.</p><p>Lastly, I used to add additional context to my errors <a href="https://go.dev/blog/go1.13-errors"><em>(%w)</em></a> to understand which flow raises an error.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dfcd2062b3cd1cf109658ee56e0a3a05/href">https://medium.com/media/dfcd2062b3cd1cf109658ee56e0a3a05/href</a></iframe><p>There is no specific logic above. After our integration tests have finished, we should terminate our container. I always try to use context.WithTimeout for these kinds of tasks. Terminate method should make it return in at most 5 seconds.</p><h4>👉 Implementation with Dockertest</h4><p>Because the suite lifecycle is the same as the testcontainer section, I skipped that part.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/144fc1a7aff11134eb4fd2384463f91b/href">https://medium.com/media/144fc1a7aff11134eb4fd2384463f91b/href</a></iframe><p>At first glance, you realize that we implement our free port implementation and integrate it with our container. Because there is no exposed random host port as testcontainer, we should do it on our own.</p><p>ExposedPorts for the container&#39;s perspective.</p><p>PortBinding for providing a container and host port mapping binding.</p><p>We also need to set <a href="https://www.confluent.io/blog/kafka-listeners-explained/">advertise-kafka-addr</a> explicitly to provide the connection from our host machine to the container. Why we didn&#39;t set it at the testcontainer section 🤔. It is because the container 9092 port is exposed to the free random host port internally. But here, if we don&#39;t provide this, we must connect as the host&#39;s 9092 port. So we don&#39;t want to use a static host port for parallel runs.</p><p>We also provide a retryFunc to understand whether our container is ready. RetryFunc tries to connect Kafka brokers, if the connection is successfully established, we can move on or try to reconnect with the adjustable backoff time.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a5aca7a731965086a3fe8e6cbffc855b/href">https://medium.com/media/a5aca7a731965086a3fe8e6cbffc855b/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dd65e9c073a1e8a0a204f810af512e0f/href">https://medium.com/media/dd65e9c073a1e8a0a204f810af512e0f/href</a></iframe><p>On the cleanUp side, we purge our container, removing the container and linked volumes from docker.</p><h4>Our tests 🚀</h4><p>I always try to separate my tests with Given-When-Then. It means in some context, (When) some action is carried out, (Then) a set of consequences should happen.</p><p>I also try to name my tests with an underscore. In doing so, it is easy to find failed functions by searching go test output.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d53855b6d0bbd0d7a34dda950e18336f/href">https://medium.com/media/d53855b6d0bbd0d7a34dda950e18336f/href</a></iframe><p><strong>Source Code:</strong> <a href="https://github.com/Abdulsametileri/integration-tests-with-redpanda">https://github.com/Abdulsametileri/integration-tests-with-redpanda</a></p><p>Thank you for reading 💛; you can also check my other articles.</p><ul><li><a href="https://medium.com/trendyol-tech/kafka-exception-c-r-onsumer-37c459e4849d">Kafka Exception Cronsumer</a></li><li><a href="https://itnext.io/lets-implement-a-real-time-package-tracking-app-with-rabbitmq-and-web-socket-using-go-80f5a5ca5c55">Let&#39;s implement a real-time package tracking app with RabbitMQ and Web socket using Go</a></li><li><a href="https://itnext.io/lets-implement-basic-service-discovery-using-go-d91c513883f6">Let&#39;s implement basic service discovery using Go</a></li><li><a href="https://levelup.gitconnected.com/how-was-i-build-a-version-control-system-vcs-using-pure-go-83ec8ec5d4f4">How I built a version control system (VCS) using pure Go</a></li><li><a href="https://medium.com/modanisa-engineering/integrating-grafana-notifications-with-gitlab-pipeline-to-restart-debezium-tasks-using-go-1378c9eaf7b8">Integrating Grafana Notifications with GitLab Pipeline to restart Debezium tasks using Go</a></li><li><a href="https://medium.com/codex/introduction-to-keycloak-227c3902754a">Introduction to Keycloak</a></li></ul><h4>References</h4><p>[1] The Clean Coder: A Code of Conduct for Professional Programmers</p><p>[2] <a href="https://redpanda.com/">https://redpanda.com</a></p><p>[3] <a href="https://www.testcontainers.org/features/networking/">https://www.testcontainers.org/features/networking/</a></p><p>[4] <a href="https://www.confluent.io/blog/kafka-listeners-explained/">https://www.confluent.io/blog/kafka-listeners-explained/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=247e4276c61d" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/boosting-your-kafka-integration-tests-using-redpanda-with-go-247e4276c61d">Boosting your Kafka integration tests using Redpanda with Go</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kafka Exception C{r}onsumer  ]]></title>
            <link>https://medium.com/trendyol-tech/kafka-exception-c-r-onsumer-37c459e4849d?source=rss-77397c9e27------2</link>
            <guid isPermaLink="false">https://medium.com/p/37c459e4849d</guid>
            <category><![CDATA[consumer]]></category>
            <category><![CDATA[exception-handling]]></category>
            <category><![CDATA[cron]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Abdulsamet İLERİ]]></dc:creator>
            <pubDate>Mon, 31 Oct 2022 12:54:15 GMT</pubDate>
            <atom:updated>2023-09-17T15:21:42.747Z</atom:updated>
            <content:encoded><![CDATA[<p>As a Trendyol Indexing Team, our architecture is heavily based on an event-based system that should be consistent, eventually. So even though this word seems to give relief, it is tough to ensure that every message in the system is successfully processed.</p><p>Our eventing system relies on <a href="https://kafka.apache.org/">Apache Kafka</a>. It gives us powerful abilities when we need performance on event streaming. Our messages carry invalidations (daily 150M+) that should apply to Trendyol contents (300M+). This invalidation could be about a stock, promotion, or one of our 30 events.</p><p>In events worlds, we need to orchestrate our messages even in some inconsistent states. Retrying and delaying messages are our key strategies in this “eventual” life cycle of events. We have 10+ consumers that should apply this strategy. Therefore, we created an open-sourced library called <a href="https://github.com/Trendyol/kafka-cronsumer">Kafka Cronsumer</a> for easy implementation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*A6ZpbmodaUk8t64F.png" /><figcaption>Figure 1: Show me the code</figcaption></figure><h3>How Kafka Cronsumer Works 💡</h3><p>As the library name suggests, it consumes events within the exception topic based on the given cron expression.</p><p>For example, we could specify a cron expression as */20 * * * * , which means to run every 20th minutes. We could also set a duration 15 minutes (15m) represents our exception consumer actively consuming events within this fixed duration.</p><ul><li>If the consumer gets an error when consuming a message, it increases the message retry count and produces the exception topic for handling the next iteration (next work time) again.</li><li>A message is discarded or moved to a dead letter<strong> </strong>if it exceeds<strong> </strong>maxRetry value<strong>.</strong></li><li>Each message is processed only once at every iteration. So if the consumer encounters a message produced after our consumer start time, it pauses and waits for the next iteration.</li></ul><p>As an overview, it works shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*8MPu0n-1BXIEMiSl.png" /><figcaption>Figure 2: Kafka Exception Cronsumer timeline overview</figcaption></figure><h3>How to use it? 👈</h3><p>First of all, we need to set the required config values. We can also specify Kafka Consumer &amp; Producer related configs, but it is not mandatory.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/28a3a1c4b31d93831591f3778279f375/href">https://medium.com/media/28a3a1c4b31d93831591f3778279f375/href</a></iframe><p>We define consumeFn function, which describes how the exception messages are consumed.</p><p>After that, we initialize Kafka cronsumer with these config values and consumeFn and run. It starts to work as shown above in the how to Kafka Cronsumer works section.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0d08194ce919f12aaa10bdef883aa5ba/href">https://medium.com/media/0d08194ce919f12aaa10bdef883aa5ba/href</a></iframe><blockquote>You can find a number of ready-to-run examples at <a href="https://github.com/Trendyol/kafka-cronsumer/blob/main/examples">this directory</a>.</blockquote><h3>When to use it? 🤔</h3><ul><li>Iteration-based back-off strategies are applicable</li><li>Messages could be processed in an eventually consistent state</li><li>Max retry exceeded messages could be ignored</li><li>To increase consumer resiliency</li><li>To increase consumer performance with concurrency</li></ul><h3>When to avoid?❗️</h3><ul><li>Messages should be processed in order</li><li>Messages should be certainly processed <em>(we discard messages if max retry is exceeded or move to dead letter topic)</em></li><li>Messages should be committed <em>(we use auto-commit interval for increasing performance)</em></li><li>Messages with TTL <em>(Time to Live)</em></li></ul><p>Thanks for reading this far. All feedbacks are welcome. We love to share our knowledge. If you like to “<em>share”</em> with us, you could apply for our <a href="https://jobs.lever.co/trendyol?lever-via=q48u5LZ7k7">open positions</a>.</p><p>Co-Authored by:</p><ul><li><a href="https://medium.com/u/df34a29d57e6">Emre Odabas</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=37c459e4849d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/trendyol-tech/kafka-exception-c-r-onsumer-37c459e4849d">Kafka Exception C{r}onsumer 🔥 🚀</a> was originally published in <a href="https://medium.com/trendyol-tech">Trendyol Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>