<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sushank Hegde on Medium]]></title>
        <description><![CDATA[Stories by Sushank Hegde on Medium]]></description>
        <link>https://medium.com/@sushankhegde10?source=rss-1d589bf6b428------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*9K3rNmdf2G2kSoaA</url>
            <title>Stories by Sushank Hegde on Medium</title>
            <link>https://medium.com/@sushankhegde10?source=rss-1d589bf6b428------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 16:18:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sushankhegde10/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Backpressure in System Design: Why It Matters and How to Implement It]]></title>
            <link>https://medium.com/@sushankhegde10/backpressure-in-system-design-why-it-matters-and-how-to-implement-it-e3043ba738e6?source=rss-1d589bf6b428------2</link>
            <guid isPermaLink="false">https://medium.com/p/e3043ba738e6</guid>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[backpressure]]></category>
            <dc:creator><![CDATA[Sushank Hegde]]></dc:creator>
            <pubDate>Sun, 28 Dec 2025 13:52:30 GMT</pubDate>
            <atom:updated>2025-12-28T13:52:30.094Z</atom:updated>
            <content:encoded><![CDATA[<p>Modern backend systems are built to scale, but many of them still fail under load.<br>The reason is rarely a lack of computing power — it’s the absence of <strong>backpressure</strong>.</p><p>Backpressure is one of those concepts that sounds theoretical, but in reality, it decides whether your system <strong>degrades gracefully</strong> or <strong>collapses spectacularly</strong> during traffic spikes.</p><p>This article explains:</p><ol><li>What backpressure really means</li><li>Why is it critical in distributed systems</li><li>How to implement it correctly, with detailed examples</li></ol><h3>What Is Backpressure?</h3><p><strong>Backpressure</strong> is a mechanism by which a system <strong>signals upstream components to slow down or stop sending requests when it is overloaded</strong>.</p><p>In simple terms:</p><p><em>Backpressure is the system’s ability to say </em><strong><em>“I can’t handle more right now.”</em></strong></p><p>Instead of accepting unlimited work and failing later, a system with backpressure <strong>limits incoming work based on its actual processing capacity</strong>.</p><h3>Why Backpressure Is Important</h3><p>Most production outages follow a familiar pattern:</p><ol><li>Traffic spikes</li><li>Requests pile up</li><li>Latency increases</li><li>Retries amplify the load</li><li>Memory or thread pools are exhausted</li><li>The system crashes</li></ol><h3>Without Backpressure</h3><ol><li>Unbounded queues grow endlessly</li><li>Memory usage increases silently</li><li>Downstream services get overwhelmed</li><li>Failures cascade across the system</li></ol><h3>With Backpressure</h3><ol><li>The system rejects excess load early</li><li>Latency remains predictable</li><li>Downstream services stay protected</li><li>The system remains stable, even under overload</li></ol><p><strong>Backpressure is not about handling peak traffic.<br>It is about surviving overload.</strong></p><h3>A Realistic Example: Order Processing System</h3><h3>System Flow</h3><pre>Client → Order API → Queue → Order Processor → Database</pre><h3>System Capacity</h3><ol><li>Incoming traffic spike: <strong>1,000 requests/sec</strong></li><li>Order processors + DB can handle: <strong>200 requests/sec</strong></li></ol><h3>How Backpressure Fixes the Same System</h3><p>Let’s apply backpressure step by step.</p><h3>1. Backpressure at the API Layer (Fail Fast)</h3><h3>What to Do</h3><p>Limit the number of <strong>in-flight requests</strong>.</p><h3>Why</h3><p>The API is the <strong>entry point</strong>. If it doesn’t apply backpressure, nothing else can save the system.</p><h3>Behavior</h3><ul><li>Allow only a fixed number of concurrent requests</li><li>Reject excess requests immediately</li></ul><h3>Result</h3><ul><li>Clients receive <strong>HTTP 429 / 503</strong></li><li>Pressure is pushed upstream</li><li>Internal components stay protected</li></ul><blockquote><em>A fast failure is better than a slow success.</em></blockquote><h3>2. Bounded Queues (Never Buffer Infinitely)</h3><h3>The Problem</h3><p>Unbounded queues delay failure instead of preventing it.</p><h3>The Fix</h3><p>Use <strong>bounded queues</strong> with explicit limits.</p><p>Example:</p><pre>Max queue size = 10,000 orders</pre><p>When the queue is full:</p><ol><li>Stop accepting new tasks</li><li>Reject or block producers intentionally</li></ol><p>This ensures:</p><ol><li>Memory usage is capped</li><li>Overload is visible, not hidden</li></ol><h3>3. Pull-Based Consumption (Workers Control the Pace)</h3><h3>Push-Based (Bad)</h3><ol><li>Producers push messages continuously</li><li>Consumers get overwhelmed</li></ol><h3>Pull-Based (Good)</h3><ol><li>Consumers request work only when ready</li></ol><p>In this model:</p><ol><li>Workers pull messages in batches</li><li>Processing speed defines system throughput</li></ol><p>This is why systems like <strong>Kafka</strong> work well:</p><ol><li>Consumers control max.poll.records</li><li>Consumption rate adapts to processing capacity</li></ol><h3>4. Database as a Backpressure Signal</h3><p>Databases naturally enforce backpressure through:</p><ol><li>Connection pool limits</li><li>Locks</li><li>Query execution time</li></ol><h3>Common Mistake</h3><p>Increasing the DB connection pool size endlessly.</p><h3>Correct Approach</h3><ol><li>Respect DB limits</li><li>Fail fast when connections are exhausted</li><li>Propagate the failure upstream</li></ol><p>If the database is slow, the system <strong>must slow down</strong>, rather than fighting it.</p><h3>5. Load Shedding (Hard Backpressure)</h3><p>When overload is severe, the system should prioritize <strong>core functionality</strong>.</p><p>Examples:</p><ol><li>Reject low-priority requests</li><li>Disable non-critical features</li><li>Serve cached or degraded responses</li></ol><p>This keeps essential operations alive while sacrificing optional ones.</p><h3>What Backpressure Is NOT</h3><p>Backpressure is often confused with other mechanisms.</p><ol><li>❌ <strong>Retries</strong> — usually amplify load</li><li>❌ <strong>Autoscaling</strong> — reacts too late</li><li>❌ <strong>Bigger machines</strong> — postpones failure</li></ol><p>Backpressure is <strong>explicit control</strong>, not reactive hope.</p><h3>How to Validate Backpressure Works</h3><p>You haven’t implemented backpressure until you test failure.</p><p>Test scenarios:</p><ol><li>Traffic spikes</li><li>Slow database</li><li>Downstream timeouts</li></ol><p>Expected behavior:</p><ol><li>Errors increase predictably</li><li>Latency plateaus instead of exploding</li><li>Memory usage remains stable</li><li>System recovers quickly</li></ol><h3>Key Takeaways</h3><ol><li>Every ingress point needs backpressure</li><li>Never accept unlimited work</li><li>Bounded queues are non-negotiable</li><li>Fail fast to protect the system</li><li>Design for overload, not average traffic</li></ol><blockquote><em>If your system design does not explain what happens when components slow down, backpressure is missing.</em></blockquote><h3>Final Thought</h3><p>Backpressure is not an optimization.<br> It is a <strong>fundamental requirement</strong> for building reliable distributed systems.</p><p>A system that rejects requests gracefully is far better than one that accepts everything and crashes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e3043ba738e6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is Your App Ready for Prime Time? A Deep Dive into Performance Testing & Fixing Bottlenecks]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-snippet">Ever used an app that crashed right when you needed it most? Maybe during a flash sale or when trying to book a last-minute ticket&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@sushankhegde10/is-your-app-ready-for-prime-time-a-deep-dive-into-performance-testing-fixing-bottlenecks-5ac045e30c11?source=rss-1d589bf6b428------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@sushankhegde10/is-your-app-ready-for-prime-time-a-deep-dive-into-performance-testing-fixing-bottlenecks-5ac045e30c11?source=rss-1d589bf6b428------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ac045e30c11</guid>
            <category><![CDATA[performance-testing]]></category>
            <category><![CDATA[design-systems]]></category>
            <category><![CDATA[load-testing]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Sushank Hegde]]></dc:creator>
            <pubDate>Tue, 14 Oct 2025 16:00:31 GMT</pubDate>
            <atom:updated>2025-10-14T16:00:31.584Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[The Unseen Dance: Understanding Apache Kafka Consumer Rebalancing]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-snippet">In the distributed world of Apache Kafka, consumers work together in consumer groups to read messages from topics. This collaborative&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@sushankhegde10/the-unseen-dance-understanding-apache-kafka-consumer-rebalancing-0e553215a179?source=rss-1d589bf6b428------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@sushankhegde10/the-unseen-dance-understanding-apache-kafka-consumer-rebalancing-0e553215a179?source=rss-1d589bf6b428------2</link>
            <guid isPermaLink="false">https://medium.com/p/0e553215a179</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[kafka-rebalancing]]></category>
            <dc:creator><![CDATA[Sushank Hegde]]></dc:creator>
            <pubDate>Wed, 08 Oct 2025 18:26:50 GMT</pubDate>
            <atom:updated>2025-10-08T18:26:50.488Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[ The End-to-End Journey of a Message in Apache Kafka]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-snippet">When people first hear about Apache Kafka, they usually think of it as a &#x201C;message queue.&#x201D; But Kafka is far more powerful than that. It&#x2019;s a&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@sushankhegde10/the-end-to-end-journey-of-a-message-in-apache-kafka-25dc124b1ab9?source=rss-1d589bf6b428------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@sushankhegde10/the-end-to-end-journey-of-a-message-in-apache-kafka-25dc124b1ab9?source=rss-1d589bf6b428------2</link>
            <guid isPermaLink="false">https://medium.com/p/25dc124b1ab9</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[backend-development]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Sushank Hegde]]></dc:creator>
            <pubDate>Wed, 24 Sep 2025 19:21:52 GMT</pubDate>
            <atom:updated>2025-09-24T19:21:52.986Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Decoding Kafka: A Beginner’s Guide to the Ultimate Messaging System ]]></title>
            <link>https://medium.com/@sushankhegde10/decoding-kafka-a-beginners-guide-to-the-ultimate-messaging-system-cf31f2d1b4b5?source=rss-1d589bf6b428------2</link>
            <guid isPermaLink="false">https://medium.com/p/cf31f2d1b4b5</guid>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[message-queue]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Sushank Hegde]]></dc:creator>
            <pubDate>Mon, 22 Sep 2025 16:20:05 GMT</pubDate>
            <atom:updated>2025-09-22T16:20:05.734Z</atom:updated>
            <content:encoded><![CDATA[<p>Ever wondered how Netflix instantly knows what you just watched to recommend a new show, or how Uber tracks thousands of cars in real-time? The magic behind many of these massive, data-driven platforms is a powerful tool called <strong>Apache Kafka</strong>.</p><p>If you’re new to the world of software engineering, terms like “messaging queue” and “event streaming” might sound intimidating. But don’t worry! By the end of this article, you’ll understand what Kafka is, why it’s so popular, and how it works, all through a simple analogy: a super-efficient postal service.</p><h3>What’s a Messaging Queue Anyway?</h3><p>Before we dive into Kafka, let’s understand the problem it solves. Imagine a busy restaurant. 👨‍🍳</p><ul><li><strong>Waiters (Producers):</strong> They take orders from customers.</li><li><strong>Chefs (Consumers):</strong> They prepare the food.</li></ul><p>Now, what if a waiter had to run to a specific chef for every single order and wait for them to be free? It would be chaotic and incredibly slow.</p><p>Instead, restaurants use an <strong>order rail</strong>. The waiter (producer) simply clips the order onto the rail (the queue). The chefs (consumers) pick up orders from the rail whenever they are free.</p><p>This rail decouples the waiters from the chefs. Waiters can take orders as fast as they want, and chefs can cook at their own pace. A messaging queue in software works exactly the same way. It’s a middleman that holds messages from a <strong>Producer</strong> (the application sending data) until a <strong>Consumer</strong> (the application processing the data) is ready for it. This prevents data loss and makes the entire system more resilient and efficient.</p><h3>Enter Kafka: The Supercharged Postal Service</h3><p>While the order rail is a good start, Kafka is more like a global, supercharged postal service. It’s designed to handle a massive volume of mail (data) with incredible speed and reliability.</p><p>Let’s break down the core components of Kafka using this analogy.</p><h3>Producers and Consumers</h3><ul><li><strong>Producers:</strong> These are like the people <strong>sending letters</strong>. They write a message, put it in an envelope, and drop it off at the post office. In the tech world, a producer could be a web application logging user clicks or an IoT sensor sending temperature data.</li><li><strong>Consumers:</strong> These are the people <strong>receiving the letters</strong>. They subscribe to receive mail addressed to them. A consumer could be a database saving the user clicks or an analytics dashboard displaying the temperature.</li></ul><h3>Brokers: The Post Offices</h3><p>A Kafka cluster is made up of one or more servers, each called a <strong>broker</strong>. Think of a <strong>broker as a post office building</strong>. It’s responsible for receiving mail from producers, storing it safely, and handing it over to consumers when they come to collect it. A cluster of multiple brokers (a network of post offices) ensures that even if one post office goes down, your mail is still safe and accessible in another.</p><h3>Topics: The Mail Sorting Rooms</h3><p>You can’t just dump all the mail for an entire country into one big pile. It needs to be sorted. In Kafka, messages are organized into <strong>topics</strong>.</p><p>A <strong>topic is like a dedicated mail sorting room for a specific category</strong>, such as “Mail for New York” or “Utility Bills.” When a producer sends a message, it specifies the topic. For example, all user activity data might go to the user-clicks topic, while all payment transactions go to the payments topic. Consumers then subscribe to the specific topics they are interested in.</p><h3>Partitions: The Sorting Desks</h3><p>Even within the “Mail for New York” room, you need more organization. Imagine a single person trying to sort millions of letters. It would be a bottleneck. To solve this, the room is divided into several sorting desks, say “Desk 1 (Zip Codes 10001–10010),” “Desk 2 (Zip Codes 10011–10020),” and so on.</p><p>These are <strong>partitions</strong>. A Kafka topic is split into multiple partitions. This allows Kafka to process messages in parallel, dramatically increasing performance. When a producer sends a message, Kafka can assign it to a specific partition within the topic. Multiple consumers can then read from different partitions of the same topic simultaneously. This is a key reason for Kafka’s incredible speed and scalability.</p><p>To recap our analogy:</p><ul><li><strong>Producer:</strong> Sender of the letter.</li><li><strong>Consumer:</strong> Receiver of the letter.</li><li><strong>Broker:</strong> The post office building.</li><li><strong>Topic:</strong> A specific room for sorting mail (e.g., “New York Mail”).</li><li><strong>Partition:</strong> A specific sorting desk within that room.</li></ul><h3>Why is Kafka So Special? ✨</h3><p>Kafka isn’t just another messaging queue. It has a few superpowers that make it the industry standard for high-performance data pipelines.</p><h3>1. Durability and Fault Tolerance</h3><p>Kafka was built to be reliable. Messages aren’t just held in memory; they are <strong>written to disk</strong>. Going back to our analogy, this is like the post office <strong>photocopying every letter</strong> before sorting it. Furthermore, Kafka can be configured to replicate data across multiple brokers (post offices). If one post office burns down (a server fails), a perfect copy of the mail exists in other locations, ensuring no data is lost.</p><h3>2. High Throughput</h3><p>Kafka can handle millions of messages per second. It achieves this by processing messages in <strong>batches</strong>. Instead of a mail carrier delivering one letter at a time, they wait until their bag is full and deliver all the letters on their route in one efficient trip. This batching process allows Kafka to handle massive volumes of data with very low latency.</p><h3>3. Scalability</h3><p>What happens when a city’s population explodes and the volume of mail skyrockets? You build more post offices and add more sorting desks. Kafka is horizontally scalable, meaning you can easily add more brokers (post offices) to your cluster to handle more data as your needs grow.</p><h3>4. It’s a Streaming Platform</h3><p>Kafka is more than just a queue; it’s an <strong>event streaming platform</strong>. This means it stores a log of messages (events) in the order they were received. Consumers can “rewind” and re-read messages from the past. This is incredibly powerful. Imagine your analytics application had a bug and processed payments incorrectly. With Kafka, you can fix the bug and simply tell the consumer to go back in time and re-process all the payment messages from the beginning, ensuring correctness.</p><h3>Getting Started with Kafka</h3><p>The best way to learn is by doing. Here’s a simplified roadmap to get your hands dirty:</p><ol><li><strong>Download Kafka:</strong> Head over to the official Apache Kafka website and download the latest version.</li><li><strong>Start the Servers:</strong> Kafka uses another component called ZooKeeper to manage the cluster. You’ll first start the ZooKeeper server, and then the Kafka broker server using the command-line scripts provided.</li><li><strong>Create a Topic:</strong> Create your first topic (e.g., my-first-topic).</li><li><strong>Send Some Messages:</strong> Open a producer console and type some messages to send to your topic.</li><li><strong>Read the Messages:</strong> Open a consumer console, which will subscribe to your topic and immediately display the messages you just sent.</li></ol><p>Playing with these simple command-line tools is a fantastic way to see the concepts of producers, consumers, and topics in action.</p><p>Happy streaming!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf31f2d1b4b5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reducing Job Duration: From 20 Hours to 3 Hours — A Software Engineering Solution for Asynchronous…]]></title>
            <link>https://medium.com/@sushankhegde10/reducing-job-duration-from-20-hours-to-3-hours-a-software-engineering-solution-for-asynchronous-ac928a2e8e96?source=rss-1d589bf6b428------2</link>
            <guid isPermaLink="false">https://medium.com/p/ac928a2e8e96</guid>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[relevance]]></category>
            <dc:creator><![CDATA[Sushank Hegde]]></dc:creator>
            <pubDate>Tue, 06 Aug 2024 13:49:16 GMT</pubDate>
            <atom:updated>2024-08-06T13:49:16.416Z</atom:updated>
            <content:encoded><![CDATA[<h3>Reducing Job Duration: From 20 Hours to 3 Hours — A Software Engineering Solution for Asynchronous Job Processing</h3><p>Let’s begin with, what are jobs and why do we have to use them in solving tech problems?</p><p>In software engineering, “jobs” typically refer to units of work or tasks that are scheduled and executed, often asynchronously, within a system. These jobs can be anything from data processing tasks, web scraping, or database maintenance, to more complex operations like machine learning model training or large-scale data migrations</p><p>Similarly, Blibli search uses one of the jobs named “Normalisation”, now the main objective of the job is to get the famous top K search terms and get the max stats which include relevance score stats and our customized boosters score stats, and store these values in our global cache and push these data to local cache so that user queries can use these stats and help in normalizing the product score and help in reranking the search results.</p><p>Let’s take a look at a job that used to take 20 hours to complete. This job is initiated when an API is triggered by a Rundeck cron job. The API could be triggered in any one of the pods/instances due to health checks and certain Kubernetes rituals. Once triggered, this single pod would process millions of search terms in a synchronous, single-threaded manner.</p><p>Running such a lengthy job on a single instance introduces a significant risk of failure. If the instance is restarted due to OS-related reasons or experiences an out-of-memory (OOM) error, the job will be terminated, and all the progress made up to that point will be lost. This creates a single point of failure, undermining the reliability and efficiency of the process.</p><p>Now, everyone might be curious about how and who solved this problem. The answer lies with our extraordinary hero, the mighty Lord Apache Kafka!</p><p>With a wave of his wand, Kafka magically transformed our colossal 20-hour job into tiny, manageable chunks. He orchestrated a symphony where all the instances worked together in perfect harmony, like a well-rehearsed choir hitting every note.</p><p>The result? A parallel-processing fiesta that wrapped up the job faster than you can say “distributed systems!” Thanks to Kafka’s wizardry, what once took nearly a day is now completed in record time, proving that with the right hero, even the toughest tech problems can turn into a piece of cake!</p><p>Let’s come back and look at how we solved the problem. Imagine we have 10 million famous search terms for which we need to run this job. Let’s also assume we have 10 pods/instances of the service to help us with this task. What did we do? We divided the search terms into 10 chunks (10 million / 10), creating each event with 1 million search terms. Knowing we had 10 instances/pods, we created an internal topic with 10 partitions and started pushing these events to our internal topic. We then had 10 consumers from the same service consuming these events and processing them.</p><p>Seems like a walk in the park, doesn’t it? But let’s take a closer look at the challenges we faced when we decided to fan out the job. Spoiler alert: it wasn’t all sunshine and rainbows!</p><ol><li><strong>Addressing the surge in throughput for downstream services caused by increased parallelism.<br></strong>Initially, when a single instance was handling the job, the throughput was around 2k requests per minute (rpm). But crank up the parallelism, and suddenly you’re pushing over 15k rpm to downstream services. This surge can wreak havoc on job performance and data quality. The fix? Simple: limit the parallelism to what your system can handle without falling apart.</li><li><strong>Once we fan out, it’s difficult to keep track of what’s happening<br></strong>Once the data is pushed to Kafka, it’s processed asynchronously, leaving us in the dark about when the job will finish. To tackle this, we came up with a solution: a charming little counter in a common data store like Redis. Here’s the magic: we set the counter to the number of events we create before pushing them to Kafka. As each event finishes its job, it gives the counter a gentle nudge by decrementing it. When the counter finally hits zero, Tadow! The job is complete. Simple, elegant, and the beauty of Redis.</li><li><strong>Unequal distribution of events across the consumers<br></strong>Whenever events are pushed to Kafka, Kafka’s default partitioning strategy — whether it’s sticky or some other default method — can lead to an uneven distribution of events across partitions. To fix this and ensure an even distribution among consumers, you can take a more hands-on approach.<br>Here’s a neat trick: assign each event a partition number manually. This way, you can control exactly which partition the event lands in. Use a modulus operation to determine the partition number: for each event, calculate its position in the list and take the modulus with the total number of partitions for that topic. Essentially, this turns your distribution strategy into a round-robin approach, ensuring that events are spread out more evenly across all partitions and consumers. Easy!</li><li><strong>Kafka rebalancing issue<br></strong>Once an event is consumed by a Kafka consumer, the offset isn’t committed until the Kafka thread finishes processing the event. The consumer acknowledges completion and requests the consumer group to advance the offset. However, since some events might take hours or minutes to process, the consumer group might mistakenly assume the consumer has failed and enter a rebalancing state.<br>To address this, we implemented a solution where the event, once consumed, is handed off to a separate thread from our customized thread pool. This approach allows us to process events asynchronously and avoid the rebalancing issue, ensuring that the consumer group remains stable and responsive.</li></ol><p>In essence, the solutions we’ve explored not only improved efficiency but also demonstrated the power of thoughtful problem-solving in software engineering which was possible due to the search tech team. With these strategies in place, we’ve set the stage for a robust, scalable, and reliable system capable of handling complex jobs with ease.</p><p>As we continue to refine and adapt these approaches, the goal remains clear: to harness the full potential of our tools and technologies, turning intricate challenges into streamlined successes.</p><p>We could still see the scope for improvement, but feel free to let us know</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac928a2e8e96" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>