<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Misba Sheikh on Medium]]></title>
        <description><![CDATA[Stories by Misba Sheikh on Medium]]></description>
        <link>https://medium.com/@misbas2313?source=rss-b9f2b18ed347------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*wU48wAVnf97AWxkb</url>
            <title>Stories by Misba Sheikh on Medium</title>
            <link>https://medium.com/@misbas2313?source=rss-b9f2b18ed347------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 17:28:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@misbas2313/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Kafka cluster, Message producer, Topic Setup]]></title>
            <link>https://medium.com/@misbas2313/kafka-cluster-message-producer-topic-setup-3024dc0a3c56?source=rss-b9f2b18ed347------2</link>
            <guid isPermaLink="false">https://medium.com/p/3024dc0a3c56</guid>
            <category><![CDATA[kafka-consumer]]></category>
            <category><![CDATA[setup-guide]]></category>
            <category><![CDATA[kafka-topic]]></category>
            <category><![CDATA[kafka-cluster]]></category>
            <category><![CDATA[kafka-producer]]></category>
            <dc:creator><![CDATA[Misba Sheikh]]></dc:creator>
            <pubDate>Fri, 15 Nov 2024 09:45:00 GMT</pubDate>
            <atom:updated>2024-11-15T09:45:00.919Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>What is kafka?</strong></h3><p><strong>Apache Kafka</strong> is an open-source distributed event streaming platform. It’s a distributed publish-subscribe messaging system and a robust queue that can handle a high volume of data and enables you to pass messages from one endpoint to another. Kafka is suitable for both offline and online message consumption. Kafka messages are persisted on the disk and replicated within the cluster to prevent data loss. Kafka is built on top of the Zookeeper synchronization service. It integrates very well with Apache Storm and Spark for real-time streaming data analysis.</p><p>Kafka is a unified platform for handling all the real-time data feeds. Kafka supports low latency message delivery and gives guarantee for fault tolerance in the presence of machine failures. It can handle many diverse consumers. Kafka is very fast, performs 2 million writes/sec. Kafka persists all data to the disk, which essentially means that all the writes go to the page cache of the OS (RAM). This makes it very efficient to transfer data from page cache to a network socket. Before moving deep into the Kafka, you must be aware of the main terminologies such as topics, brokers, producers and consumers.</p><p>• <strong>Topics: </strong>A stream of messages belonging to a particular category is called atopic. Data is stored in topics. Topics are split into partitions. For each topic, Kafka keeps a minimum of one partition. Each such partition contains messages in an immutable ordered sequence. A partition is implemented as a set of segment files of equal sizes.</p><p>• <strong>Partition: </strong>Topics may have many partitions, so it can handle an arbitrary amount of data.</p><p>• <strong>Offset: </strong>Sequence ID given to messages as they arrive in a position. It is immutable and stores messages in order of arrival. If you want to locate a message, you need its Topic name →Partition →Offset.</p><p>• <strong>Brokers: </strong>Brokers are simple system responsible for maintaining the published data. Each broker may have zero or more partitions per topic. Assume, if there are N partitions in a topic and N number of brokers, each broker will have one partition.</p><p>• <strong>Kafka Cluster: </strong>Kafka’s having more than one broker are called as Kafka cluster. A Kafka cluster can be expanded without downtime. These clusters are used to manage the persistence and replication of message data.</p><p>• <strong>Producers: </strong>Producers are the publisher of messages to one or more Kafka topics. Producers send data to Kafka brokers. Every time a producer publishes a message to a broker, the broker simply appends the message to the last segment file. The message will be appended to a partition. Producer can also send messages to a partition of their choice.</p><p>• <strong>Consumers: </strong>Consumers read data from brokers. Consumers subscribes to one or more topics and consume published messages by pulling data from the brokers.</p><p>• <strong>Zookeeper: </strong>Zookeeper is an open-source server which enables highly reliable distributed coordination. Before we start the server on which the broker is running, we need to start zookeeper’s server.</p><h3><strong>Kafka workflow</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m1_5tHFyLwEgNIVEqbbrEg.png" /><figcaption><strong>Fig. 1</strong></figcaption></figure><p>• <strong>Producers</strong> send message to a topic at regular intervals.</p><p>• <strong>Kafka broker</strong> stores all messages in the partitions configured for that topic. It ensures the messages are equally shared between partitions. If the producer sends two messages and there are two partitions, Kafka will store one message in the first partition and the second message in the second partition.</p><p>• <strong>Consumer </strong>subscribes to a specific topic.</p><p>• Once the consumer subscribes to a topic, Kafka will provide the current offset of the topic to the consumer and saves the offset in the Zookeeper ensemble.</p><p>• Consumer will request the Kafka in a regular interval (like 100 Ms) for new messages.</p><p>• Once Kafka receives the messages from producers, it forwards these messages to the consumers.</p><p>• Consumer will receive the message and process it.</p><p>• Once the messages are processed, consumer will send an acknowledgement to the Kafka broker.</p><p>• Once Kafka receives an acknowledgement, it changes the offset to the new value and updates it in the Zookeeper. Since offsets are maintained in the <strong>Zookeeper</strong>, the consumer can read next message correctly even during server outrages.</p><p>• This above flow will repeat until the consumer stops the request.</p><p>• Consumer has the option to rewind/skip to the desired offset of a topic at any time and read all the subsequent messages.</p><h3><strong>How to setup single node Kafka cluster with docker compose file?</strong></h3><p>A single node Kafka broker setup would meet most of the local development needs, so let’s start by learning this simple setup.</p><p><strong>Docker-compose.yml configuration</strong></p><p>To start an Apache Kafka server, we’d first need to start a Zookeeper server. We can configure this dependency in a<strong> docker-compose.yml</strong> file, which will ensure that the Zookeeper server always starts before the Kafka server and stops after it.</p><blockquote>Let’s create a simple docker-compose.yml<em> </em>file with two services, namely zookeeper and Kafka:</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xZHTWWd9W9hN46db6gTflQ.png" /></figure><blockquote>Let’s start the Kafka server by spinning up the containers using the <em>docker-Compose</em> command:<strong> docker compose up -d</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZE_VFNLcYRLblpu-xkdTWw.png" /><figcaption><strong>Fig. 2</strong></figcaption></figure><p>In this setup, our Zookeeper server is listening on port=22181 for the Kafka service, which is defined within the same container setup. However, for any client running on the host, it’ll be exposed on port 22181.</p><p>Similarly, the Kafka service is exposed to the host applications through port 29092, but it is advertised on port 9092 within the container environment configured by the KAFKA_ADVERTISED_LISTENERS property.</p><h3><strong>Connection Using Offset Explorer</strong></h3><p>Offset Explorer (formerly Kafka Tool) is <strong>a GUI application for managing and using Apache Kafka clusters</strong>. It provides an intuitive UI that allows one to quickly view objects within a Kafka cluster as well as the messages stored in the topics of the cluster.</p><p>Offset Explorer is <strong>free for personal use</strong> only. Any non-personal use, including commercial, educational and non-profit work is not permitted without purchasing a license. Non-personal use is allowed for evaluation purposes for 30 days following the download of Offset Explorer, after which you must purchase a valid license or remove the software.</p><p>Offset Explorer supports Apache Kafka version 0.8.1 and above.</p><p>• <strong>Setup Offset Explorer</strong></p><p>To download the Offset Explorer (formerly Kafka Tool) for your operating system, use the links below: <strong>https://www.kafkatool.com/download.html</strong></p><p>All versions of Offset Explorer come with a bundled JRE except for the Linux version. For Linux, you must have Java 8 installed on your operating system before using Offset Explorer.</p><p>• <strong>Connection</strong></p><ol><li>In order to view data in your Kafka cluster you must first create a connection to it. This can be done using the ‘Add Cluster’ / ‘Add New Connection’ menu item in the File-menu. In the ‘Add Cluster’ dialog you must provide the following values under the General section:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4wzdl6cFrosjS1Up-_C71w.png" /><figcaption><strong>Fig. 3</strong></figcaption></figure><blockquote><strong>Cluster Name: </strong>name you want to give the cluster you’re trying to connect.</blockquote><blockquote><strong>Version: </strong>version of the cluster.</blockquote><blockquote><strong>Zookeeper Host: </strong>hostname or IP address of the zookeeper host in the cluster.</blockquote><blockquote><strong>Zookeeper Port: </strong>port of the zookeeper host.</blockquote><blockquote><strong>Chroot path: </strong>path where the Kafka cluster data appears in Zookeeper. The default value is correct in most cases.</blockquote><p>2. After adding the cluster, you will be able to interact with your Kafka cluster brokers, topics and consumers. Below you can see the properties of your Kafka cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-y2Re7VFOviDSco0p0rLpg.png" /><figcaption><strong>Fig. 4</strong></figcaption></figure><blockquote>After adding the cluster, you will be able to interact with all the brokers, host, ID and port will be listed.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3JSHxq1NKJ-uI3ZneL-IgQ.png" /><figcaption><strong>Fig. 5</strong></figcaption></figure><p>3. List of all the topics present inside your cluster and you can even create new topic from offset explorer with help of plus logo option available in the below screenshot. After creating the topic, you can investigate the topic for information like offset, partition, key, value and timestamp. Similarly, you can also check number of consumers available inside your cluster and investigate it for information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*9QPo-Gv3bQESUGO9T93NXQ.png" /><figcaption><strong>Fig. 6</strong></figcaption></figure><h3><strong>Kafka publisher using spring boot</strong></h3><p>Up till now we have discussed about Kafka and its workflow, how to setup single node Kafka cluster with docker compose file and setup connection Using Offset Explorer.</p><p>It’s time to build a producer for Kafka, as discussed earlier Kafka Producers are the publisher of messages to one or more Kafka topics.</p><p>We have a single node cluster up and running and a connection tool to interact withthe cluster. Let’s implement a message producer so that we can have a look at how data gets published into a topic.</p><blockquote><strong>Step 1</strong>: We need to create Producer Factory object and pass Producer configurations in it.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4K9va5enGKybfVs7KDOfoA.png" /></figure><blockquote><strong>Step 2</strong>: We need to create Listenable Future object and pass configurations in it.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wCH3KRE9xNWsBo5WRH2FBA.png" /></figure><blockquote><strong>Step 3</strong>: At last, create Kafka Template object and pass Producer Factory’s object as an argument.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SEDTKFXmY-ivQoLz-wFdgA.png" /></figure><blockquote>KafkaProducerConfig.java</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oIFDVcka72dVS4d9tgcqlQ.png" /></figure><blockquote>MessageProducer.java</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MqZjQdQfdS09odw6I3_dKw.png" /></figure><blockquote>KafkaApplication.java</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ej7osagBuRo815_ELLOp3Q.png" /></figure><blockquote>application.properties</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OzNGN8Yp0TcpSx4O2zD4pQ.png" /></figure><p>To understand the code better and what is the function of each class in this publisher, below is the flow-chart specifying the flow of code.</p><p>1. <strong>KafkaApplication.java</strong>: Contains main method.</p><p>2. <strong>MessageProducer.java</strong>: Reads messages from csv file and publishes them to Kafka topic which is scheduled every 15 seconds.</p><p>3. <strong>KafkaProducerConfig.java</strong>: Contains producer configuration.</p><p>4. <strong>application.properties</strong>: Contains values for Ip: port and topic name.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/938/1*aAvGWRMeE5tr_VyoidC1_Q.png" /><figcaption><strong>Fig. 7</strong></figcaption></figure><h3><strong>Conclusion</strong></h3><p>After the last step your data is published into a topic in form of messages and each row is a message for Kafka. It is processed one at a time in a data stream.</p><p>From here, consumers can fetch the data for processing from the topic. There they can do various operations on the data as per their requirement.</p><p>I hope this tutorial was helpful for you in understanding Kafka and its resources.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3024dc0a3c56" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>