<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mohsan Abbas on Medium]]></title>
        <description><![CDATA[Stories by Mohsan Abbas on Medium]]></description>
        <link>https://medium.com/@moabbas.ch?source=rss-1990bd72e256------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*yw7CObh78BilPiMEtGB8uw@2x.jpeg</url>
            <title>Stories by Mohsan Abbas on Medium</title>
            <link>https://medium.com/@moabbas.ch?source=rss-1990bd72e256------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 10:50:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@moabbas.ch/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Effective Kafka Consumption in Golang: A Comprehensive Guide]]></title>
            <link>https://medium.com/@moabbas.ch/effective-kafka-consumption-in-golang-a-comprehensive-guide-aac54b5b79f0?source=rss-1990bd72e256------2</link>
            <guid isPermaLink="false">https://medium.com/p/aac54b5b79f0</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[kafka-consumer]]></category>
            <category><![CDATA[sarama]]></category>
            <dc:creator><![CDATA[Mohsan Abbas]]></dc:creator>
            <pubDate>Mon, 15 May 2023 20:21:56 GMT</pubDate>
            <atom:updated>2023-05-15T20:21:56.863Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Introduction:</strong></p><p>Kafka is a popular distributed streaming platform that offers reliable, scalable, and high-performance data processing. When consuming data from Kafka, there are several best practices to consider to ensure efficiency and reliable processing. This article provides an overview of these best practices and demonstrates their implementation using the Sarama library in Go.</p><p>Here are some high-level best practices for efficient Kafka consumption.</p><ol><li><strong>Choose an appropriate committing strategy</strong>: Kafka offers two committing strategies, automatic and manual. While automatic is easy to use, it may lead to data loss or duplication. Manual committing offers a higher level of control, ensuring that messages are processed at least once or exactly once, depending on the use case.</li><li><strong>Minimize round trips to Kafka</strong>: Reading messages in larger batches can significantly improve throughput. This can be achieved by tuning parameters such as fetch.min.bytes and fetch.max.wait.ms.</li><li><strong>Use a consumer group when possible</strong>: Kafka allows multiple consumers to form a consumer group to consume data in parallel. This enables Kafka to distribute data across all the consumers in a group, leading to efficient data consumption.</li><li><strong>Tune consumer buffer sizes</strong>: Buffer sizes for consumers, like receive.buffer.bytes and max.partition.fetch.bytes, can be tuned according to the expected size of the messages and the memory capacity of the consumer. This can improve the consumer’s performance.</li><li><strong>Handle rebalancing</strong>: When a new consumer joins a consumer group, or an existing consumer leaves, Kafka triggers a rebalancing process to redistribute the load. During this process, consumers stop consuming data. Therefore, handling rebalancing quickly and efficiently can increase overall throughput.</li><li><strong>Monitor your consumers</strong>: Use Kafka’s consumer metrics to monitor the consumer’s performance. Regular monitoring can help identify performance bottlenecks and tune the consumer’s configuration.</li></ol><h3>Choose an appropriate committing strategy</h3><ol><li><strong>Automatic Committing</strong></li></ol><p>Sarama’s ConsumerGroup automatically commits offsets by default. This means it periodically commits the offsets of messages that have been successfully consumed, which allows consumers to pick up where they left off in case of a restart or failure.</p><p>Here’s a simple example of a consumer with automatic committing:</p><pre>config := sarama.NewConfig()<br>config.Version = sarama.V2_0_0_0 // specify appropriate Kafka version<br>config.Consumer.Offsets.AutoCommit.Enable = true<br>config.Consumer.Offsets.AutoCommit.Interval = 1 * time.Second<br><br>consumerGroup, err := sarama.NewConsumerGroup(brokers, groupID, config)<br>if err != nil {<br>    log.Panicf(&quot;Error creating consumer group client: %v&quot;, err)<br>}<br><br>consumer := Consumer{}<br>ctx := context.Background()<br><br>for {<br>    err := consumerGroup.Consume(ctx, []string{topic}, consumer)<br>    if err != nil {<br>        log.Panicf(&quot;Error from consumer: %v&quot;, err)<br>    }<br>}</pre><p>In this example, the offsets are automatically committed every second.</p><p><strong>2. Manual Committing</strong></p><p>Manual committing gives you more control over when the message offsets are committed. This can be helpful when you need stronger delivery guarantees or when your message processing is complex.</p><p>Here’s a simple example of a consumer with manual committing:</p><pre>config := sarama.NewConfig()<br>config.Version = sarama.V2_0_0_0 // specify appropriate Kafka version<br>config.Consumer.Offsets.AutoCommit.Enable = false // disable auto-commit<br><br>consumerGroup, err := sarama.NewConsumerGroup(brokers, groupID, config)<br>if err != nil {<br>    log.Panicf(&quot;Error creating consumer group client: %v&quot;, err)<br>}<br><br>consumer := Consumer{}<br>ctx := context.Background()<br><br>for {<br>    err := consumerGroup.Consume(ctx, []string{topic}, consumer)<br>    if err != nil {<br>        log.Panicf(&quot;Error from consumer: %v&quot;, err)<br>    }<br>}<br><br>// define the consumer<br>type Consumer struct{}<br><br>func (consumer Consumer) Setup(_ sarama.ConsumerGroupSession) error   { return nil }<br>func (consumer Consumer) Cleanup(_ sarama.ConsumerGroupSession) error { return nil }<br>func (consumer Consumer) ConsumeClaim(sess sarama.ConsumerGroupSession, claim sarama.ConsumerGroupClaim) error {<br>    for msg := range claim.Messages() {<br>        // process the message<br>        fmt.Printf(&quot;Message topic:%q partition:%d offset:%d\n&quot;, msg.Topic, msg.Partition, msg.Offset)<br><br>        // after processing the message, mark the offset<br>        sess.MarkMessage(msg, &quot;&quot;)<br>    }<br>    return nil<br>}</pre><p>In this example, sess.MarkMessage(msg, &quot;&quot;) is used to manually mark a message as processed, which will eventually be committed as per Consumer.Offsets.CommitInterval configuration.</p><p><em>Please remember to handle the errors properly in your production code. The error handling is omitted here for the sake of brevity.</em></p><h3>Minimize round trips to Kafka</h3><p>Reducing round trips to Kafka is all about optimizing the way you read from and write to Kafka. Here’s how you can do it:</p><ol><li><strong>Increase batch size:</strong> Kafka works best when you send messages in batches. The larger the batch, the more efficiently Kafka can send the data. However, there’s a trade-off between latency and throughput. Larger batches mean better throughput but increased latency because you need to wait longer to fill the batch.</li></ol><p>In Go, you can set the batch size when producing messages using the sarama package as follows:</p><pre>config := sarama.NewConfig()<br>config.Producer.Flush.Bytes = 1024 * 1024 // 1MB</pre><p><em>This tells the producer to accumulate 1MB of data (or whatever you set it to) before sending it to Kafka.</em></p><p>2.<strong> Increase fetch size</strong>: When consuming messages from Kafka, you can fetch more data at once. This reduces the number of round trips needed to fetch the same amount of data. However, fetching more data consumes more memory, so you need to balance these two concerns.</p><pre>config := sarama.NewConfig()<br>config.Consumer.Fetch.Default = 1024 * 1024 // 1MB</pre><p><em>This tells the consumer to fetch 1MB of data at a time.</em></p><p>3. <strong>Use long polling:</strong> Long polling means the consumer will wait for data to arrive if there’s no data in Kafka when it polls. This reduces the number of round trips because the consumer doesn’t need to keep asking for data when there’s no data.</p><pre>config := sarama.NewConfig()<br>config.Consumer.MaxWaitTime = 500 * time.Millisecond</pre><p><em>This tells the consumer to wait up to 500 milliseconds for data to arrive before returning.</em></p><p>4.<strong> Consume multiple partitions concurrently</strong>: If a Kafka topic has multiple partitions, you can consume them concurrently. This increases the total amount of data you can consume at once.</p><pre>config := sarama.NewConfig()<br>consumer, err := sarama.NewConsumer(brokers, config)<br>if err != nil {<br>    // handle error<br>}<br>partitions, err := consumer.Partitions(topic)<br>if err != nil {<br>    // handle error<br>}<br>for _, partition := range partitions {<br>    pc, err := consumer.ConsumePartition(topic, partition, offset)<br>    if err != nil {<br>        // handle error<br>    }<br>    go func(pc sarama.PartitionConsumer) {<br>        for message := range pc.Messages() {<br>            // process message<br>        }<br>    }(pc)<br>}</pre><p><em>This creates a goroutine for each partition and consumes the partitions concurrently.</em></p><h3>Use a consumer group when possible</h3><p>A consumer group is a group of consumers that work together to consume messages from a Kafka topic. Consumer groups allow you to distribute the processing of messages among multiple consumers, making it possible to scale out your processing capacity as needed. When using a consumer group, Kafka takes care of assigning partitions to the consumers in the group and ensuring that each partition is processed by only one consumer at a time.</p><p>Here’s how to use a consumer group with the sarama package in Go:</p><ol><li>Create a sarama.Config with the required configuration:</li></ol><pre>config := sarama.NewConfig()<br>config.Version = sarama.V2_6_0_0 // Set the appropriate Kafka version<br>config.Consumer.Return.Errors = true</pre><p>2. Implement a ConsumerGroupHandler interface:</p><p>This interface has three methods: Setup, Cleanup, and ConsumeClaim. You need to implement these methods to handle the initialization, processing, and cleanup of consuming messages from Kafka.</p><pre>type exampleConsumerGroupHandler struct {<br>}<br><br>func (h *exampleConsumerGroupHandler) Setup(_ sarama.ConsumerGroupSession) error {<br>    // Perform any necessary setup tasks here.<br>    return nil<br>}<br><br>func (h *exampleConsumerGroupHandler) Cleanup(_ sarama.ConsumerGroupSession) error {<br>    // Perform any necessary cleanup tasks here.<br>    return nil<br>}<br><br>func (h *exampleConsumerGroupHandler) ConsumeClaim(session sarama.ConsumerGroupSession, claim sarama.ConsumerGroupClaim) error {<br>    for message := range claim.Messages() {<br>        // Process the message here.<br>        fmt.Printf(&quot;Message: %s\n&quot;, string(message.Value))<br><br>        // Mark the message as processed in the session.<br>        session.MarkMessage(message, &quot;&quot;)<br>    }<br>    return nil<br>}</pre><p>3. Create a sarama.ConsumerGroup and start consuming:</p><pre>brokers := []string{&quot;localhost:9092&quot;} // List of Kafka brokers<br>topic := &quot;example_topic&quot;<br>groupID := &quot;example_consumer_group&quot;<br><br>// Create a new consumer group<br>consumerGroup, err := sarama.NewConsumerGroup(brokers, groupID, config)<br>if err != nil {<br>    log.Fatalf(&quot;Error creating consumer group: %v&quot;, err)<br>}<br>defer consumerGroup.Close()<br><br>// Create an instance of your ConsumerGroupHandler implementation<br>handler := &amp;exampleConsumerGroupHandler{}<br><br>// Consume messages using the consumer group<br>for {<br>    err := consumerGroup.Consume(context.Background(), []string{topic}, handler)<br>    if err != nil {<br>        log.Printf(&quot;Error consuming messages: %v&quot;, err)<br>    }<br>}</pre><p>This example sets up a consumer group that consumes messages from the specified topic. The consumer group automatically scales the message processing across multiple consumers in the group, making it possible to increase processing capacity by adding more consumers.</p><p>When using a consumer group, always remember to handle rebalancing events and errors that may occur during consumption.</p><h3>Tune consumer buffer sizes</h3><p>In Sarama, you can tune the consumer buffer size to adjust how many messages the consumer can hold in memory before processing them. This can be done by adjusting the ChannelBufferSize field of the Config structure.</p><p>By default, the buffer size is set to 256. This means that Sarama will hold up to 256 messages in memory before starting to process them. If your consumer is slow, increasing the buffer size might help to improve the throughput. However, keep in mind that a larger buffer will also consume more memory.</p><p>Here’s an example of how you can increase the buffer size:</p><pre>config := sarama.NewConfig()<br>config.Consumer.Return.Errors = true<br>config.Version = sarama.V2_1_0_0<br>config.Consumer.Offsets.Initial = sarama.OffsetOldest<br>// Increase buffer size<br>config.ChannelBufferSize = 500<br><br>// Create a new consumer group<br>group, err := sarama.NewConsumerGroup([]string{broker}, groupID, config)<br>if err != nil {<br>    panic(err)<br>}<br><br>// Consume messages<br>ctx := context.Background()<br>for {<br>    topics := []string{topic}<br>    handler := exampleConsumerGroupHandler{}<br><br>    err := group.Consume(ctx, topics, &amp;handler)<br>    if err != nil {<br>        panic(err)<br>    }<br>}</pre><p>In the above example, we’ve set config.ChannelBufferSize to 500, which allows Sarama to buffer up to 500 messages in memory before processing them. This might help to improve the throughput if the consumer is slow.</p><p>However, keep in mind that this is a trade-off: a larger buffer size will consume more memory, but it might help to improve the throughput if the consumer is slow. It’s important to monitor the memory usage of your application and adjust the buffer size accordingly.</p><h3>Handle rebalancing</h3><p>When a new consumer is added to a consumer group or an existing consumer leaves (due to a crash or being shutdown), Kafka rebalances the consumers in the group. Rebalancing is the process by which Kafka ensures that all the consumers in a consumer group are consuming from unique partitions.</p><p>In Sarama, handling rebalancing is done by implementing the Setup and Cleanup functions of the ConsumerGroupHandler interface. The Setup function is called at the beginning of a new session, when the consumer group is being rebalanced. The Cleanup function is called at the end of a session, just before the next rebalance occurs.</p><p>Here’s an example of how to handle rebalancing events:</p><pre>type exampleConsumerGroupHandler struct{}<br><br>func (exampleConsumerGroupHandler) Setup(_ sarama.ConsumerGroupSession) error {<br>    // Mark the beginning of a new session. This is called when the consumer group is being rebalanced.<br>    log.Println(&quot;Consumer group is being rebalanced&quot;)<br>    return nil<br>}<br><br>func (exampleConsumerGroupHandler) Cleanup(_ sarama.ConsumerGroupSession) error {<br>    // Mark the end of the current session. This is called just before the next rebalance happens.<br>    log.Println(&quot;Rebalancing will happen soon, current session will end&quot;)<br>    return nil<br>}<br><br>func (h exampleConsumerGroupHandler) ConsumeClaim(sess sarama.ConsumerGroupSession, claim sarama.ConsumerGroupClaim) error {<br>    // This is where you put your message handling logic<br>    for msg := range claim.Messages() {<br>        log.Printf(&quot;Message topic:%q partition:%d offset:%d\n&quot;, msg.Topic, msg.Partition, msg.Offset)<br>        sess.MarkMessage(msg, &quot;&quot;)<br>    }<br>    return nil<br>}<br><br>// ...<br><br>// Create a new consumer group<br>group, err := sarama.NewConsumerGroup([]string{broker}, groupID, config)<br>if err != nil {<br>    panic(err)<br>}<br><br>// Consume messages<br>ctx := context.Background()<br>for {<br>    topics := []string{topic}<br>    handler := exampleConsumerGroupHandler{}<br><br>    err := group.Consume(ctx, topics, handler)<br>    if err != nil {<br>        panic(err)<br>    }<br>}</pre><p>In the above example, when a rebalance starts, “Consumer group is being rebalanced” will be printed to the console, and when a rebalance is about to happen, “Rebalancing will happen soon, current session will end” will be printed.</p><p>By properly handling rebalancing events, you can ensure that your application gracefully handles changes to the consumer group, such as consumers leaving or joining, and that no messages are lost or processed twice during these events.</p><h3>Monitor your consumers</h3><p>Monitoring your Kafka consumers is critical for ensuring the health and performance of your system. You want to keep an eye on metrics such as lag, processing time, and error rates.</p><p>Golang does not have built-in support for Kafka monitoring, but there are several libraries and tools available that can help you. Let’s take a look at a few of these:</p><ol><li><strong>Sarama’s Metrics</strong>: Sarama provides a metrics registry which reports various metrics that can be helpful to monitor, such as the number of requests, responses, the size of requests and responses, etc. These metrics can be collected and monitored using a monitoring system like Prometheus.</li><li><strong>JMX Exporter:</strong> If you’re running Kafka on JVM, you can use the JMX Exporter to expose Kafka’s MBeans for Prometheus.</li><li><strong>Kafka Exporter:</strong> Kafka Exporter is a third-party tool that can provide more detailed metrics about Kafka. It can provide consumer group lag, which is a key metric to monitor when consuming from Kafka.</li><li><strong>Jaeger or OpenTelemetry:</strong> These tools can be used for distributed tracing, which can be helpful to track how messages are flowing through your system and where bottlenecks may be occurring.</li><li><strong>Logs:</strong> Keep an eye on your application logs. Logging any errors or unexpected behavior in your consumer can help diagnose issues.</li><li><strong>Consumer group command:</strong> You can use the kafka-consumer-groups command line tool that comes with Kafka to describe the state of a consumer group, including its offsets and lag.</li></ol><p>Remember to not only track these metrics but also set up alerts for any conditions that need your attention, such as high lag or error rates. This way, you can quickly respond to any potential issues before they become serious problems.</p><p>Overall, monitoring Kafka consumers involves tracking key performance indicators (KPIs), setting up alerts for abnormal conditions, and using logs and tracing to understand the behavior of your consumers. This can help ensure your Kafka-based applications are robust, reliable, and efficient.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aac54b5b79f0" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>