<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Raghav on Medium]]></title>
        <description><![CDATA[Stories by Raghav on Medium]]></description>
        <link>https://medium.com/@buildthingsuseful?source=rss-5078f30a490b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*8sRTPmVg6aGUCMKD4CrBEg.jpeg</url>
            <title>Stories by Raghav on Medium</title>
            <link>https://medium.com/@buildthingsuseful?source=rss-5078f30a490b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 12:17:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@buildthingsuseful/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Stage 8: Build Your Own Kafka — Test the System]]></title>
            <link>https://buildthingsuseful.medium.com/stage-8-build-your-own-kafka-test-the-system-39ca1c1dca5e?source=rss-5078f30a490b------2</link>
            <guid isPermaLink="false">https://medium.com/p/39ca1c1dca5e</guid>
            <category><![CDATA[kafka-consumer]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[message-queue]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[kafka-producer]]></category>
            <dc:creator><![CDATA[Raghav]]></dc:creator>
            <pubDate>Thu, 13 Mar 2025 05:48:12 GMT</pubDate>
            <atom:updated>2025-03-13T05:48:12.863Z</atom:updated>
            <content:encoded><![CDATA[<h3>Stage 8: Build Your Own Kafka — Test the System</h3><h3>Test the System</h3><p>Testing a distributed system requires running multiple components and ensuring they interact correctly. This step-by-step approach allows you to verify each component’s functionality before moving on.</p><h3>1. Compile the project using Maven:</h3><pre>mvn clean package</pre><p>This will create a JAR file in the target directory.</p><h3>2: Start ZooKeeper</h3><p>ZooKeeper is required for Build Your Own Kafka to function. Start it with:</p><pre># Start ZooKeeper with default configuration<br>zkServer start<br><br># If that doesn&#39;t work, try:<br>zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties<br># Or create a simple config file:<br>echo &quot;tickTime=2000&quot; &gt; zk.cfg<br>echo &quot;dataDir=/tmp/zookeeper&quot; &gt;&gt; zk.cfg<br>echo &quot;clientPort=2181&quot; &gt;&gt; zk.cfg<br>zookeeper-server-start zk.cfg</pre><h3>3: Start Multiple Broker Instances</h3><p>Open three terminal windows and start three brokers:</p><pre># Terminal 1 - Broker 1<br>java -cp target/simple-kafka-1.0-SNAPSHOT.jar com.simplekafka.broker.SimpleKafkaBroker 1 localhost 9091 2181<br><br># Terminal 2 - Broker 2<br>java -cp target/simple-kafka-1.0-SNAPSHOT.jar com.simplekafka.broker.SimpleKafkaBroker 2 localhost 9092 2181<br><br># Terminal 3 - Broker 3<br>java -cp target/simple-kafka-1.0-SNAPSHOT.jar com.simplekafka.broker.SimpleKafkaBroker 3 localhost 9093 2181</pre><p>In the terminal outputs, you should see:</p><ul><li>Each broker registering with ZooKeeper</li><li>One broker being elected as the controller</li><li>Logs about broker metadata being updated</li></ul><h3>4: Produce Messages</h3><p>Open a new terminal to produce messages:</p><pre>java -cp target/simple-kafka-1.0-SNAPSHOT.jar com.simplekafka.client.SimpleKafkaProducer localhost 9091 test-topic</pre><p>The producer will:</p><ol><li>Create the topic if it doesn’t exist</li><li>Send 10 test messages</li><li>Print the offset for each message</li></ol><h3>5: Consume Messages</h3><p>Open another terminal to consume messages:</p><pre>java -cp target/simple-kafka-1.0-SNAPSHOT.jar com.simplekafka.client.SimpleKafkaConsumer localhost 9091 test-topic 0</pre><p>The consumer will:</p><ol><li>Connect to the topic</li><li>Continuously poll for messages</li><li>Print each received message</li></ol><h3>6: Test Fault Tolerance</h3><p>Once you’ve verified that messages flow correctly, test the system’s fault tolerance:</p><ol><li>Terminate one of the broker processes (e.g., broker 1 — the likely controller)</li><li>Observe the logs of the remaining brokers — one should become the new controller</li><li>Produce more messages using a different broker:</li></ol><pre>java -cp target/simple-kafka-1.0-SNAPSHOT.jar com.simplekafka.client.SimpleKafkaProducer localhost 9092 test-topic</pre><p>4. Consume messages to verify they’re still accessible:</p><pre>java -cp target/simple-kafka-1.0-SNAPSHOT.jar com.simplekafka.client.SimpleKafkaConsumer localhost 9092 test-topic 0</pre><h3>7: Test Broker Recovery</h3><ol><li>Restart the terminated broker:</li></ol><pre>java -cp target/simple-kafka-1.0-SNAPSHOT.jar com.simplekafka.broker.SimpleKafkaBroker 1 localhost 9091 2181</pre><ol><li>Observe in the logs how the broker rejoins the cluster</li><li>Check that it loads existing topics</li></ol><h3>8: Examine the Data Files</h3><p>Look at the data directory structure to understand how SimpleKafka stores data:</p><pre>ls -la data/</pre><p>You should see:</p><ul><li>Directories for each broker ID</li><li>Topic directories within each broker directory</li><li>Partition directories within each topic directory</li><li>Log and index files within each partition directory</li></ul><h3>Key Concepts to Focus On During Testing</h3><p>As you test your Build Your Own Kafka implementation, pay special attention to these key Kafka concepts:</p><p><strong>1. Topic Partitioning and Replication</strong></p><p>Watch how a topic gets divided into partitions and how those partitions are distributed across brokers. In the logs, you’ll see statements about partition creation and leader assignment. This partitioning is what allows Kafka to scale horizontally.</p><p><strong>2. Leader and Follower Mechanics</strong></p><p>For each partition, observe:</p><ul><li>How one broker becomes the leader</li><li>How followers replicate data from the leader</li><li>What happens when a leader fails and a new leader is elected</li></ul><p><strong>3. The Controller Role</strong></p><p>One broker serves as the controller for the cluster. Watch how:</p><ul><li>The controller is elected through ZooKeeper</li><li>It manages partition assignments</li><li>It handles broker failures</li><li>A new controller is elected if the current one fails</li></ul><p><strong>4. Message Persistence</strong></p><p>Examine the data files on disk to understand:</p><ul><li>The log segment structure</li><li>How messages are appended sequentially</li><li>How indices map offsets to file positions</li></ul><p><strong>5. Client-Broker Protocol</strong></p><p>While testing, use tools like Wireshark to observe the network traffic between clients and brokers. This will help you understand:</p><ul><li>The binary protocol format</li><li>Request/response patterns</li><li>How clients discover and connect to the right brokers</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39ca1c1dca5e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stage 7: Build Your Own Kafka — Building Higher-Level Producer and Consumer APIs]]></title>
            <link>https://buildthingsuseful.medium.com/stage-7-build-your-own-kafka-building-higher-level-producer-and-consumer-apis-2fba12ecf062?source=rss-5078f30a490b------2</link>
            <guid isPermaLink="false">https://medium.com/p/2fba12ecf062</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[kafka-consumer]]></category>
            <category><![CDATA[message-queue]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[kafka-producer]]></category>
            <dc:creator><![CDATA[Raghav]]></dc:creator>
            <pubDate>Thu, 13 Mar 2025 05:44:10 GMT</pubDate>
            <atom:updated>2025-03-13T05:54:10.419Z</atom:updated>
            <content:encoded><![CDATA[<h3>Stage 7: Build Your Own Kafka — Building Higher-Level Producer and Consumer APIs</h3><h3>Building Higher-Level Producer and Consumer APIs</h3><p>The producer and consumer APIs provide a more user-friendly interface on top of the core client, handling details like serialization and offset management. This layered approach is similar to Kafka’s client library design.</p><h3>Step 9: Create the Producer API</h3><pre>package com.simplekafka.client;<br><br>import java.io.IOException;<br>import java.nio.charset.StandardCharsets;<br>public class SimpleKafkaProducer {<br>    private final SimpleKafkaClient client;<br>    private final String topic;<br>    public SimpleKafkaProducer(String bootstrapBroker, int bootstrapPort, String topic) {<br>        this.client = new SimpleKafkaClient(bootstrapBroker, bootstrapPort);<br>        this.topic = topic;<br>    }<br>    public void initialize() throws IOException {<br>        client.initialize();<br>    }<br>    public long send(String message) throws IOException {<br>        byte[] data = message.getBytes(StandardCharsets.UTF_8);<br>        return client.send(topic, 0, data); // Default to partition 0<br>    }<br>    // Additional methods<br>}</pre><p><strong>Send Methods</strong></p><p><strong>Send to Random Partition</strong></p><pre>public long send(String message) throws IOException</pre><p><strong>Step-by-step process:</strong></p><ol><li>Retrieve metadata for the topic</li><li>Verify the topic exists, throwing an IOException if not</li><li>Calculate the number of partitions available</li><li>Select a random partition</li><li>Call the partition-specific send method</li></ol><p><strong>Send to Specific Partition</strong></p><pre>public long send(String message, int partition) throws IOException</pre><p><strong>Step-by-step process:</strong></p><ol><li>Convert the message string to UTF-8 bytes</li><li>Use the client to send the data to the specified topic and partition</li><li>Return the offset where the message was written</li></ol><p>Both send methods return a long value representing the offset where the message was stored in the partition.</p><p><strong>Close Method</strong></p><pre>public void close()</pre><p>This method is designed to clean up resources, though in this implementation it doesn’t do anything specific. In a more complete implementation, this would close connections and release resources.</p><h3>Step 10: Create the Consumer API</h3><p>The SimpleKafkaConsumer class is designed to consume messages from a Kafka topic. Let’s look at its essential components:</p><pre>package com.simplekafka.client;<br><br>import java.io.IOException;<br>import java.nio.charset.StandardCharsets;<br>import java.util.List;<br>public class SimpleKafkaConsumer {<br>    private final SimpleKafkaClient client;<br>    private final String topic;<br>    private final int partition;<br>    private long currentOffset;<br>    public SimpleKafkaConsumer(String bootstrapBroker, int bootstrapPort, String topic, int partition) {<br>        this.client = new SimpleKafkaClient(bootstrapBroker, bootstrapPort);<br>        this.topic = topic;<br>        this.partition = partition;<br>        this.currentOffset = 0;<br>    }<br>    public void initialize() throws IOException {<br>        client.initialize();<br>    }<br>    public List&lt;byte[]&gt; poll() throws IOException {<br>        List&lt;byte[]&gt; messages = client.fetch(topic, partition, currentOffset, 1024 * 1024);<br>        if (!messages.isEmpty()) {<br>            currentOffset += messages.size();<br>        }<br>        return messages;<br>    }<br>    // Additional methods<br>}</pre><p><strong>Message Consumption Methods</strong></p><p><strong>Basic Polling Method</strong></p><pre>public List&lt;byte[]&gt; poll() throws IOException {<br>    List&lt;byte[]&gt; messages = client.fetch(topic, partition, currentOffset, MAX_BYTES);<br>    if (!messages.isEmpty()) {<br>        currentOffset += messages.size();<br>    }<br>    return messages;<br>}</pre><p>This core method:</p><ol><li>Fetches messages from the specified topic and partition</li><li>Starts from the current offset</li><li>Limits the fetch size to MAX_BYTES (1MB)</li><li>Advances the offset counter when messages are received</li><li>Returns the retrieved messages</li></ol><p><strong>Continuous Consumption Loop</strong></p><pre>public void startConsuming(MessageHandler handler) {<br>    if (running.compareAndSet(false, true)) {<br>        consumerThread = new Thread(() -&gt; {<br>            try {<br>                while (running.get()) {<br>                    List&lt;byte[]&gt; messages = poll();<br><br>            for (byte[] message : messages) {<br>                                    handler.handle(message, currentOffset - messages.size() + messages.indexOf(message));<br>                                }<br>                                // If no messages, wait a bit before polling again<br>                                if (messages.isEmpty()) {<br>                                    Thread.sleep(POLL_INTERVAL_MS);<br>                                }<br>                            }<br>                        } catch (Exception e) {<br>                            if (running.get()) {<br>                                LOGGER.log(Level.SEVERE, &quot;Error in consumer loop&quot;, e);<br>                            }<br>                            running.set(false);<br>                        }<br>                    });<br>                    consumerThread.setDaemon(true);<br>                    consumerThread.start();<br>                    LOGGER.info(&quot;Started consuming from topic: &quot; + topic + &quot;, partition: &quot; + partition);<br>                }<br>            }</pre><p>This method implements continuous consumption by:</p><ol><li>Using atomic compareAndSet to ensure only one consumption loop starts</li><li>Creating a dedicated thread for polling</li><li>Continuously polling for new messages in a loop</li><li>Processing each message through the provided MessageHandler</li><li>Calculating the correct offset for each message</li><li>Implementing a backoff delay when no messages are available</li><li>Handling errors and logging them</li><li>Setting the thread as a daemon so it doesn’t prevent JVM shutdown</li></ol><p><strong>Stopping Consumption</strong></p><pre>public void stopConsuming() {<br>    if (running.compareAndSet(true, false)) {<br>        if (consumerThread != null) {<br>            try {<br>                consumerThread.interrupt();<br>                consumerThread.join(1000);<br>            } catch (InterruptedException e) {<br>                Thread.currentThread().interrupt();<br>            }<br>        }<br><br>LOGGER.info(&quot;Stopped consuming from topic: &quot; + topic + &quot;, partition: &quot; + partition);<br>    }<br>}</pre><p>This method safely stops the consumption loop by:</p><ol><li>Using atomic compareAndSet to ensure the stop operation happens only once</li><li>Interrupting the consumer thread</li><li>Waiting for the thread to terminate (with a timeout)</li><li>Preserving the interrupted status if an InterruptedException occurs</li></ol><p><strong>Offset Management</strong></p><pre>public void seek(long offset) {<br>    this.currentOffset = offset;<br>}<br><br>public long getCurrentOffset() {<br>    return currentOffset;<br>}</pre><p>These methods provide control over the consumption position:</p><ul><li>seek() allows jumping to a specific offset</li><li>getCurrentOffset() retrieves the current position</li></ul><p><a href="https://medium.com/@buildthingsuseful/stage-8-build-your-own-kafka-test-the-system-39ca1c1dca5e">Next Article — Stage 8</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2fba12ecf062" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stage 6: Build Your Own Kafka — Developing the Client Library]]></title>
            <link>https://buildthingsuseful.medium.com/stage-6-build-your-own-kafka-developing-the-client-library-96ecc0218089?source=rss-5078f30a490b------2</link>
            <guid isPermaLink="false">https://medium.com/p/96ecc0218089</guid>
            <category><![CDATA[message-queue]]></category>
            <category><![CDATA[kafka-producer]]></category>
            <category><![CDATA[kafka-consumer]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Raghav]]></dc:creator>
            <pubDate>Thu, 13 Mar 2025 05:41:15 GMT</pubDate>
            <atom:updated>2025-03-13T05:53:30.446Z</atom:updated>
            <content:encoded><![CDATA[<h3>Stage 6: Build Your Own Kafka — Developing the Client Library</h3><h3>Developing the Client Library</h3><h3>Step 8: Create the Client API</h3><p>The client library abstracts away the complexity of the protocol and broker discovery, providing a simple API for producing and consuming messages. It handles details like finding the right broker for a specific partition</p><p>Now let’s create the client side of our Build Your Own Kafka:</p><p>Create a SimpleKafkaClient.java file in the client package:</p><pre>package com.simplekafka.client;<br><br>import com.simplekafka.broker.Protocol;<br>import java.io.IOException;<br>import java.net.InetSocketAddress;<br>import java.nio.ByteBuffer;<br>import java.nio.channels.SocketChannel;<br>import java.util.ArrayList;<br>import java.util.List;<br>import java.util.Map;<br>import java.util.concurrent.ConcurrentHashMap;<br>public class SimpleKafkaClient {<br>    private final String bootstrapBroker;<br>    private final int bootstrapPort;<br>    private final Map&lt;String, TopicMetadata&gt; topicMetadata;<br>    public SimpleKafkaClient(String bootstrapBroker, int bootstrapPort) {<br>        this.bootstrapBroker = bootstrapBroker;<br>        this.bootstrapPort = bootstrapPort;<br>        this.topicMetadata = new ConcurrentHashMap&lt;&gt;();<br>    }<br>    public void initialize() throws IOException {<br>        refreshMetadata();<br>    }<br>    public void refreshMetadata() throws IOException {<br>        // Fetch cluster metadata from broker<br>    }<br>    public long send(String topic, int partition, byte[] message) throws IOException {<br>        // Send a message to a specific topic-partition<br>        return -1; // Placeholder<br>    }<br>    public List&lt;byte[]&gt; fetch(String topic, int partition, long offset, int maxBytes) throws IOException {<br>        // Fetch messages from a topic-partition<br>        return new ArrayList&lt;&gt;(); // Placeholder<br>    }<br>    // Additional methods and inner classes for metadata<br>}</pre><p><a href="https://medium.com/@buildthingsuseful/stage-7-build-your-own-kafka-building-higher-level-producer-and-consumer-apis-2fba12ecf062">Next Article — Stage 7</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=96ecc0218089" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stage 5: Build Your Own Kafka — Building the Broker]]></title>
            <link>https://buildthingsuseful.medium.com/stage-5-build-your-own-kafka-building-the-broker-e516c4757356?source=rss-5078f30a490b------2</link>
            <guid isPermaLink="false">https://medium.com/p/e516c4757356</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[message-queue]]></category>
            <category><![CDATA[kafka-producer]]></category>
            <category><![CDATA[kafka-consumer]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Raghav]]></dc:creator>
            <pubDate>Thu, 13 Mar 2025 05:39:26 GMT</pubDate>
            <atom:updated>2025-03-13T05:52:58.787Z</atom:updated>
            <content:encoded><![CDATA[<h3>Stage 5: Build Your Own Kafka — Building the Broker</h3><h3>Building the Broker</h3><p>The broker is the central component of Kafka, responsible for handling client connections, storing messages, and coordinating with other brokers. It uses non-blocking I/O for efficiency and maintains metadata about topics and partitions.</p><h3>Overview of SimpleKafkaBroker Architecture</h3><p>The SimpleKafkaBroker implements a distributed messaging system similar to Apache Kafka with:</p><ul><li>Multiple broker nodes that form a cluster</li><li>Topic-based message organization with partitioned storage</li><li>Leader-follower replication for fault tolerance</li><li>ZooKeeper for cluster coordination and metadata management</li></ul><h3>Step 7: Implement the Broker Server</h3><p>Create a SimpleKafkaBroker.java file in the broker package:</p><ol><li><strong>Core Components</strong></li></ol><p><strong>Broker Initialization and Configuration</strong></p><pre>public SimpleKafkaBroker(int brokerId, String host, int port, int zkPort) throws IOException {<br>    this.brokerId = brokerId;<br>    this.brokerHost = host;<br>    this.brokerPort = port;<br>    this.topics = new ConcurrentHashMap&lt;&gt;();<br>    this.executor = Executors.newFixedThreadPool(10);<br>    this.serverChannel = ServerSocketChannel.open();<br>    this.isRunning = new AtomicBoolean(false);<br>    this.isController = new AtomicBoolean(false);<br>    this.clusterMetadata = new ConcurrentHashMap&lt;&gt;();<br>    <br>    // Initialize data directory<br>    File dataDir = new File(DATA_DIR + File.separator + brokerId);<br>    if (!dataDir.exists()) {<br>        dataDir.mkdirs();<br>    }<br>    // Initialize ZooKeeper client<br>    this.zkClient = new ZookeeperClient(&quot;localhost&quot;, zkPort);<br>}</pre><p>The constructor initializes:</p><ul><li>Broker ID, host, and port for identification and networking</li><li>Topics map to store topic-partition information</li><li>Thread pool for handling client requests</li><li>Server socket channel for network communication</li><li>Flags for broker status (running, controller)</li><li>Cluster metadata to track other brokers</li><li>Local data directory for message storage</li><li>ZooKeeper client for coordination</li></ul><p><strong>2. Starting and Stopping the Broker</strong></p><p><strong>Starting the Broker</strong></p><pre>public void start() throws IOException { .... }</pre><p>This method:</p><ul><li>Sets up the network socket</li><li>Registers the broker with ZooKeeper</li><li>Participates in controller election</li><li>Loads existing topic metadata</li><li>Starts accepting client connections</li></ul><p><strong>Stopping the Broker</strong></p><pre>public void stop() { ... }</pre><p>This method gracefully shuts down the broker by:</p><ul><li>Closing the network server socket</li><li>Closing all partition log files</li><li>Shutting down the thread pool</li><li>Closing the ZooKeeper connection</li></ul><p><strong>3. ZooKeeper Integration for Cluster Management</strong></p><p><strong>Registering with ZooKeeper</strong></p><pre>private void registerWithZookeeper() { ... }</pre><p>This method:</p><ul><li>Connects to ZooKeeper</li><li>Creates an ephemeral node for this broker (ephemeral nodes disappear when the connection is lost)</li><li>Adds the broker’s information to local metadata</li><li>Sets up a watch on the broker registry to detect cluster changes</li></ul><p><strong>Handling Broker Changes</strong></p><pre>private void onBrokersChanged(List&lt;String&gt; brokerIds) {<br>    // Update cluster metadata<br>    // Remove brokers that have disappeared<br>    // Re-elect controller if needed<br>        // As controller, rebalance partitions due to cluster changes<br>        // Re-attempt controller election<br>}</pre><p>This method handles changes in the broker cluster:</p><ul><li>Updates local metadata with new brokers</li><li>Removes brokers that are no longer available</li><li>Manages controller responsibilities when cluster changes</li><li>Triggers partition rebalancing if needed</li></ul><p><strong>4. Controller Election and Management</strong></p><pre>private void electController() {<br>    try {<br>        String controllerPath = &quot;/controller&quot;;<br>        <br>        // Check if node exists and is valid<br>        boolean nodeExists = zkClient.exists(controllerPath);<br>        if (nodeExists) {<br>            String existingData = zkClient.getData(controllerPath);<br>            if (existingData == null || existingData.trim().isEmpty()) {<br>                zkClient.deleteNode(controllerPath);<br>                nodeExists = false;<br>            }<br>        }<br>        // Try to become controller<br>        boolean becameController = false;<br>        if (!nodeExists) {<br>            becameController = zkClient.createEphemeralNode(controllerPath, String.valueOf(brokerId));<br>        }<br>        if (becameController) {<br>            isController.set(true);<br>            rebalancePartitions();<br>        } else {<br>            // Watch for controller changes<br>            zkClient.watchNode(controllerPath, this::onControllerChange);<br>        }<br>    } catch (Exception e) {<br>        LOGGER.log(Level.SEVERE, &quot;Controller election failed&quot;, e);<br>        // Retry election after delay<br>        new Thread(() -&gt; {<br>            try {<br>                Thread.sleep(2000);<br>                electController();<br>            } catch (InterruptedException ie) {<br>                Thread.currentThread().interrupt();<br>            }<br>        }).start();<br>    }<br>}</pre><p>This method implements a controller election process:</p><ul><li>Checks if a controller already exists in ZooKeeper</li><li>Attempts to create a controller node if none exists</li><li>If successful, becomes the controller and rebalances partitions</li><li>If not, watches for controller changes</li><li>Retries after failures with a delay</li></ul><p><strong>5. Topic and Partition Management</strong></p><p><strong>Creating Topics</strong></p><pre>private void createTopic(String topic, int numPartitions, short replicationFactor) {<br>        // Create topic directory<br>        // Create topic in ZooKeeper<br>        // Create partitions<br>            // Select leader and followers<br>            // Create partition<br>            // Store partition metadata in ZooKeeper<br>        // Add topic to broker&#39;s metadata<br>        // Notify all brokers to load the topic<br>}</pre><p>This method handles topic creation:</p><ul><li>Ensures only the controller can create topics</li><li>Creates local directories for topic storage</li><li>Creates topic metadata in ZooKeeper</li><li>Assigns partitions with leaders and followers</li><li>Stores partition metadata in ZooKeeper</li><li>Updates local topic metadata</li><li>Notifies other brokers about the new topic</li></ul><p><strong>Loading Topics</strong></p><pre>private void loadTopic(String topic) throws Exception { ... }</pre><p>This method loads existing topic metadata:</p><ul><li>Checks if topic already exists locally</li><li>Verifies topic exists in ZooKeeper</li><li>Creates local directory structure for the topic</li><li>Reads partition metadata from ZooKeeper</li><li>Creates partition objects with leader and follower information</li><li>Adds topic to local metadata</li></ul><p><strong>Rebalancing Partitions</strong></p><pre>private void rebalancePartitions() { ... }</pre><p>This method redistributes partition leaders and followers:</p><ul><li>Ensures only the controller performs rebalancing</li><li>Checks each partition for valid leaders</li><li>Assigns new leaders for partitions with missing leaders</li><li>Updates follower assignments</li><li>Updates partition metadata in ZooKeeper</li></ul><p><strong>6. Message Handling</strong></p><p><strong>Producing Messages</strong></p><pre>private void handleProduceRequest(SocketChannel clientChannel, ByteBuffer buffer) throws IOException {<br>    // Parse request data<br>    // Check if topic exists<br>    // Find the partition<br>    // Check if this broker is the leader for the partition<br>        // Forward to leader<br>    // Append message to log<br>    // Replicate to followers<br>    // Send acknowledgment to client<br>}</pre><p>This method handles message production:</p><ul><li>Parses the produce request (topic, partition, message)</li><li>Verifies topic and partition exist</li><li>Checks if this broker is the leader for the partition</li><li>If not leader, forwards the request to the leader</li><li>If leader, appends the message to the local log</li><li>Replicates the message to follower brokers</li><li>Sends acknowledgment with the message offset</li></ul><p><strong>Consuming Messages</strong></p><pre>private void handleFetchRequest(SocketChannel clientChannel, ByteBuffer buffer) throws IOException {<br>    // Parse request data<br>    // Check if topic exists<br>    // Find the partition<br>    // Check if the offset is valid<br>        // No messages available at this offset<br>    // Read messages from log<br>    // Send response<br>			// 1 byte for response type, 4 bytes for message count<br>      // 8 bytes for offset, 4 bytes for length, plus message bytes<br>}</pre><p>This method handles message consumption:</p><ul><li>Parses the fetch request (topic, partition, offset, max bytes)</li><li>Verifies topic and partition exist</li><li>Checks if the requested offset is valid</li><li>Reads messages from the local log</li><li>Constructs a response with the messages</li><li>Sends the response to the client</li></ul><p><strong>Message Replication</strong></p><pre>private void replicateToFollowers(String topic, Partition partition, byte[] message, long offset) {<br>                // Prepare replication request<br>                // Send request to follower<br>                // Read acknowledgment<br>}</pre><p>This method replicates messages to followers:</p><ul><li>Identifies follower brokers for a partition</li><li>For each follower, creates a separate thread to handle replication</li><li>Opens a connection to the follower</li><li>Sends the message data with metadata</li><li>Waits for acknowledgment from the follower</li><li>Logs replication status</li></ul><p><strong>Handling Replication Requests</strong></p><pre>private void handleReplicateRequest(SocketChannel clientChannel, ByteBuffer buffer) throws IOException {<br>    // Parse request data<br>    // Verify topic and partition<br>    // Find the partition<br>    // Append message to log (as follower)<br>    // Send acknowledgment<br>}</pre><p>This method handles replication requests from leader brokers:</p><ul><li>Parses the replication request (topic, partition, offset, message)</li><li>Verifies topic and partition exist locally</li><li>Appends the message to the local log</li><li>Sends acknowledgment to the leader</li></ul><p><strong>7. Client Connection Handling</strong></p><pre>private void acceptConnections() {<br>    while (isRunning.get()) {<br>        try {<br>            SocketChannel clientChannel = serverChannel.accept();<br>            if (clientChannel != null) {<br>                clientChannel.configureBlocking(false);<br><br>                // Handle client connection in a separate thread<br>                                executor.submit(() -&gt; handleClient(clientChannel));<br>                            }<br>                            Thread.sleep(100); // Small pause to prevent CPU spin<br>                        } catch (Exception e) {<br>                            if (isRunning.get()) {<br>                                LOGGER.log(Level.SEVERE, &quot;Error accepting connection&quot;, e);<br>                            }<br>                        }<br>                    }<br>                }<br>                private void handleClient(SocketChannel clientChannel) {<br>                    try {<br>                        ByteBuffer buffer = ByteBuffer.allocate(1024);<br>                        while (clientChannel.isOpen() &amp;&amp; isRunning.get()) {<br>                            buffer.clear();<br>                            int bytesRead = clientChannel.read(buffer);<br>                            if (bytesRead &gt; 0) {<br>                                buffer.flip();<br>                                // Process the message based on protocol<br>                                processClientMessage(clientChannel, buffer);<br>                            } else if (bytesRead &lt; 0) {<br>                                // Connection closed by client<br>                                clientChannel.close();<br>                                break;<br>                            }<br>                            Thread.sleep(50); // Small pause to prevent CPU spin<br>                        }<br>                    } catch (Exception e) {<br>                        // Handle errors<br>                    } finally {<br>                        try {<br>                            if (clientChannel.isOpen()) {<br>                                clientChannel.close();<br>                            }<br>                        } catch (IOException e) {<br>                            LOGGER.log(Level.WARNING, &quot;Error closing client channel&quot;, e);<br>                        }<br>                    }<br>                }<br>                private void processClientMessage(SocketChannel clientChannel, ByteBuffer buffer) throws IOException {<br>                    byte messageType = buffer.get();<br>                    switch (messageType) {<br>                        case Protocol.PRODUCE:<br>                            handleProduceRequest(clientChannel, buffer);<br>                            break;<br>                        case Protocol.FETCH:<br>                            handleFetchRequest(clientChannel, buffer);<br>                            break;<br>                        case Protocol.METADATA:<br>                            handleMetadataRequest(clientChannel, buffer);<br>                            break;<br>                        case Protocol.CREATE_TOPIC:<br>                            handleCreateTopicRequest(clientChannel, buffer);<br>                            break;<br>                        case Protocol.REPLICATE:<br>                            handleReplicateRequest(clientChannel, buffer);<br>                            break;<br>                        case Protocol.TOPIC_NOTIFICATION:<br>                            handleTopicNotification(clientChannel, buffer);<br>                            break;<br>                        default:<br>                            LOGGER.warning(&quot;Unknown message type: &quot; + messageType);<br>                            Protocol.sendErrorResponse(clientChannel, &quot;Unknown message type&quot;);<br>                    }<br>                }</pre><p>These methods handle client connections:</p><ul><li>Accept incoming connections on the server socket</li><li>Create a new thread for each client</li><li>Read requests using non-blocking I/O</li><li>Process requests based on the message type</li><li>Dispatch to specific handlers for different operations</li><li>Clean up resources when the connection closes</li></ul><h3>Main Execution Flow</h3><p>When implementing build your own Kafka-like system, follow this general execution flow:</p><ol><li>Initialize broker with configuration</li><li>Start network server</li><li>Register with coordination service (ZooKeeper)</li><li>Participate in controller election</li><li>Load existing topic metadata</li><li>Accept and process client requests</li><li>Handle message production, consumption, and replication</li><li>Respond to cluster changes by rebalancing partitions</li><li>Gracefully shut down when requested</li></ol><p><a href="https://medium.com/@buildthingsuseful/stage-6-build-your-own-kafka-developing-the-client-library-96ecc0218089">Next Article — Stage 6</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e516c4757356" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stage 4: Build Your Own Kafka — Building the Storage Layer]]></title>
            <link>https://buildthingsuseful.medium.com/stage-4-build-your-own-kafka-building-the-storage-layer-7a8497a614f3?source=rss-5078f30a490b------2</link>
            <guid isPermaLink="false">https://medium.com/p/7a8497a614f3</guid>
            <category><![CDATA[message-queue]]></category>
            <category><![CDATA[kafka-consumer]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[kafka-producer]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Raghav]]></dc:creator>
            <pubDate>Thu, 13 Mar 2025 05:34:57 GMT</pubDate>
            <atom:updated>2025-03-13T05:52:04.419Z</atom:updated>
            <content:encoded><![CDATA[<h3>Stage 4: Build Your Own Kafka — Building the Storage Layer</h3><h3>Building the Storage Layer</h3><h3>Step 6: Create the Partition Class</h3><p>The Partition class represents a fundamental unit of data storage in Kafka. The Partition class is responsible for storing messages on disk using log segments. Kafka stores messages in append-only log segments on disk. Each partition has its own set of log segments, and messages are written sequentially. This design allows for efficient reads and writes, as well as easy cleanup of old data.</p><p>Create a Partition.java file in the broker package:</p><p><strong>1. Core Concepts and Structure</strong></p><p>The Partition class represents a fundamental unit of data storage in Kafka. Let&#39;s break down its key components:</p><pre>private final int id;                     // Unique partition identifier<br>private int leader;                       // Leader broker ID<br>private List&lt;Integer&gt; followers;          // Follower broker IDs for replication<br>private final String baseDir;             // Directory for log storage<br>private final AtomicLong nextOffset;      // Next available message offset<br>private final ReadWriteLock lock;         // Concurrency control mechanism<br>private RandomAccessFile activeLogFile;   // Currently active log file<br>private FileChannel activeLogChannel;     // Channel for file operations<br>private final List&lt;SegmentInfo&gt; segments; // List of segments in the partition</pre><p>Key Design Elements:</p><ul><li><strong>Partition ID</strong>: Each partition has a unique identifier</li><li><strong>Leader-Follower Model</strong>: Implements Kafka’s replication mechanism</li><li><strong>Segmented Storage</strong>: Data is divided into manageable segments</li><li><strong>Thread Safety</strong>: Uses ReadWriteLock for concurrent access</li></ul><p><strong>2. Initialization Process</strong></p><p>The initialize() method sets up the partition:</p><pre>private void initialize() {<br>    // Create directory if needed<br>    // Load existing segments<br>    // Sort segments by offset<br>    // Determine next available offset<br>    // Create a new segment if none exists<br>    // Open the last segment for appending<br>}</pre><p>Implementation Details:</p><ul><li>Creates the directory structure if it doesn’t exist</li><li>Discovers existing log and index files by scanning the directory</li><li>Parses file names to extract base offsets (e.g., 00000000000000000000.log)</li><li>Sorts segments chronologically by base offset</li><li>Calculates the next available offset based on existing data</li><li>Opens the most recent segment for writing or creates a new one</li></ul><p><strong>3. Message Append Operation</strong></p><p>The append() method handles writing new messages:</p><pre>public long append(byte[] message) {<br>    // Acquire write lock<br>    // Check if new segment needed<br>    // Write message size and data<br>    // Force write to disk<br>    // Update index<br>    // Increment offset counter<br>    // Return assigned offset<br>}</pre><p>Implementation Details:</p><ul><li>Acquires a write lock to prevent concurrent modifications</li><li>Checks if current segment is full (exceeds 1MB) and creates a new one if needed</li><li>Formats message with a 4-byte length prefix</li><li>Persists data using FileChannel for efficient I/O</li><li>Forces data to disk to ensure durability</li><li>Updates the index file with the new offset-position mapping</li><li>Increments and returns the message offset</li></ul><p><strong>4. Message Reading Operation</strong></p><p>The readMessages() method retrieves stored messages:</p><pre>public List&lt;byte[]&gt; readMessages(long offset, int maxBytes) {<br>    // Acquire read lock<br>    // Find segment containing offset<br>    // Find file position using index<br>    // Read messages until maxBytes limit<br>    // Handle segment boundaries<br>    // Return message list<br>}</pre><p>Implementation Details:</p><ul><li>Acquires a read lock (allowing concurrent reads)</li><li>Locates the segment containing the requested offset</li><li>Uses the index to find the exact file position for efficient access</li><li>Reads messages sequentially up to the byte limit</li><li>Handles crossing segment boundaries seamlessly</li><li>Returns messages as byte arrays</li></ul><p><strong>5. Segment Lifecycle Management</strong></p><p>The segment lifecycle follows these patterns:</p><p><strong>Creation</strong>: Segments are created when:</p><ul><li>A partition is initialized and has no existing segments</li><li>The active segment reaches the size limit (1MB)</li><li>The system is recovering from a crash</li></ul><p><strong>Activation</strong>: Only one segment is active for writing at a time:</p><ul><li>The active segment has open file handles</li><li>All other segments are closed until needed for reading</li></ul><p><strong>Retirement</strong>: Segments become inactive when:</p><ul><li>They reach the size limit</li><li>A new segment is created</li></ul><p><strong>Lookup</strong>: Segment lookup happens during:</p><ul><li>Read operations to find which segment contains a message</li><li>Write operations to check if a new segment is needed</li></ul><p>Several methods handle segment operations:</p><pre>private void createNewSegment(long baseOffset)<br>private void openSegmentForAppend(SegmentInfo segment)<br>private SegmentInfo findSegmentForOffset(long offset)<br>private long findPositionForOffset(SegmentInfo segment, long offset)</pre><p>Implementation Details:</p><p><strong>Segment Creation</strong>:</p><ul><li><strong>File naming convention</strong>: Uses a standardized 20-digit format for base offsets, ensuring proper sorting and consistency</li><li><strong>Dual file creation</strong>: Creates both log (.log) and index (.index) files for each segment</li><li><strong>Segment tracking</strong>: Adds the new segment to the in-memory segments list</li><li><strong>Immediate activation</strong>: Opens the new segment for immediate use</li></ul><p><strong>Segment Opening</strong>:</p><ul><li><strong>Resource management</strong>: Properly closes any previously active file handles</li><li><strong>File access mode</strong>: Opens files in “rw” (read-write) mode using RandomAccessFile</li><li><strong>Append positioning</strong>: Positions the file pointer at the end to ensure append-only behavior</li><li><strong>Channel-based I/O</strong>: Uses FileChannel for efficient I/O operations</li></ul><p><strong>Segment Finding</strong>:</p><ul><li><strong>Range validation</strong>: Checks if the offset is within valid bounds</li><li><strong>Binary search algorithm</strong>: Uses O(log n) search to efficiently find the correct segment</li><li><strong>Segment boundary logic</strong>: Determines if an offset falls between the current segment’s base offset and the next segment’s base offset</li><li><strong>Special case handling</strong>: Has specific logic for the last segment (which doesn’t have an upper bound)</li></ul><p><strong>Position Finding</strong>:</p><ul><li><strong>Resource management</strong>: Uses try-with-resources to ensure file handles are closed</li><li><strong>Message format understanding</strong>: Leverages knowledge of the message format (4-byte size prefix)</li><li><strong>Efficient scanning</strong>: Reads only the size field and skips message content</li><li><strong>Position manipulation</strong>: Uses direct position control rather than reading all data</li></ul><p><strong>6. Index Structure and Operation Management</strong></p><p>The index maintains a sparse mapping of offsets to file positions:</p><p><strong>Structure</strong>:</p><ul><li>Each entry is 16 bytes (8 bytes offset + 8 bytes position)</li><li>Entries are stored in ascending offset order</li><li>Not every message is indexed (sparse indexing)</li></ul><p><strong>Write Process</strong>:</p><ul><li>When a message is written, its offset and position are appended to the index</li><li>The index grows as more messages are added</li></ul><p><strong>Read Process</strong>:</p><ul><li>The relative offset is used to calculate a specific entry in the index</li><li>If the exact offset isn’t indexed, the closest previous entry is used</li><li>The position from the index is used to seek directly to the message</li></ul><p><strong>Optimization</strong>:</p><ul><li>Sparse indexing reduces index size while maintaining performance</li><li>Direct position calculation avoids linear scans</li><li>Binary search for segment location minimizes lookup time</li></ul><p><strong>7. Index Management</strong></p><p>The coordination between segments and indexes enables efficient operations:</p><p><strong>Direct Mapping</strong>: Each segment has exactly one index file</p><ul><li>Naming convention ensures proper pairing (same base name, different suffix)</li><li>Both files are created and managed together</li></ul><p><strong>Offset Translation</strong>:</p><ul><li>Global offsets are translated to segment-relative offsets for index lookup</li><li>Base offset in segment name provides the translation reference</li></ul><p><strong>Recovery Process</strong>:</p><ul><li>During startup, both segment and index files are scanned</li><li>Message counts are determined by reading segment files</li><li>Next available offset is calculated from the base offset and message count</li></ul><p><strong>Boundary Handling</strong>:</p><ul><li>When reading crosses segment boundaries, the system seamlessly opens the next segment</li><li>File handles are properly managed during transitions</li></ul><p>The system maintains index files alongside log files:</p><pre>private void updateIndex(long offset, long position)<br>private long findPositionForOffset(SegmentInfo segment, long offset)</pre><p>Implementation Details:</p><p>Updating Indexing:</p><ul><li><strong>Index record format</strong>: Stores pairs of 8-byte values (offset + position = 16 bytes per entry)</li><li><strong>Append-only index</strong>: Always positions at the end of the index file, maintaining chronological order</li><li><strong>Durability</strong>: Forces the index to disk with force(true) to ensure persistence</li><li><strong>Current segment focus</strong>: Always updates the index of the most recent segment</li></ul><p>Find Position For Offset:</p><ul><li><strong>Relative offset calculation</strong>: Translates global offsets to segment-relative offsets</li><li><strong>Direct index lookup</strong>: Uses the relative offset to calculate the exact byte position in the index file</li><li><strong>Special case handling</strong>: Handles empty indexes and out-of-range offsets</li><li><strong>Efficient random access</strong>: Uses direct file positioning rather than sequential scanning</li></ul><h3>Implementation Considerations</h3><p><strong>Recovery and Fault Tolerance</strong>:</p><ul><li>The system recovers state from disk on startup</li><li>Reconstructs offsets and segment information</li><li>Leader-follower model supports replication</li></ul><p><strong>Memory Management</strong>:</p><ul><li>File operations use fixed-size buffers</li><li>Avoids loading entire segments into memory</li><li>Controls memory pressure during high throughput</li></ul><p><strong>Performance Optimizations</strong>:</p><ul><li>Binary search for segment location</li><li>Indexed lookups for position finding</li><li>Sequential writes for better disk performance</li></ul><p><strong>Consistency Considerations</strong>:</p><ul><li>Ordering guarantees within a partition</li><li>Atomic append operations</li><li>Replication support via leader-follower design</li></ul><h3>Key Design Principles</h3><p><strong>Segmented Storage Architecture</strong>:</p><ul><li>Breaks logs into manageable chunks (1MB segments)</li><li>Enables efficient log retention policies and cleanup</li><li>Allows faster recovery and reduces I/O during startup</li></ul><p><strong>Efficient Random Access</strong>:</p><ul><li>Uses index files to enable O(log n) lookup times</li><li>Avoids scanning entire log files to find messages</li><li>Binary search optimization for segment location</li></ul><p><strong>Concurrency Model</strong>:</p><ul><li>ReadWriteLock allows multiple concurrent readers</li><li>Ensures thread safety without excessive blocking</li><li>Separates read and write locks for better throughput</li></ul><p><strong>Durability Guarantees</strong>:</p><ul><li>Forces writes to disk (force(true)) for persistence</li><li>Maintains separate index files for recovery</li><li>Updates indexes after successful writes</li></ul><p><strong>Zero-Copy Design</strong>:</p><ul><li>Uses Java NIO’s FileChannel for efficient I/O</li><li>Minimizes unnecessary data copying between buffers</li><li>Direct memory access for higher throughput</li></ul><p><strong>Stateless Processing</strong>:</p><ul><li>Offsets are client-managed, enabling flexible consumption</li><li>Consumers can start reading from any point</li><li>Supports multiple consumer strategies (exactly-once, at-least-once)</li></ul><h3>Key Takeaways</h3><ol><li><strong>Segment-Based Log Design</strong>: Implement a segmented log storage system for scalability and performance.</li><li><strong>Index Files</strong>: Maintain separate index files for efficient offset-to-position mapping.</li><li><strong>Concurrency Control</strong>: Use appropriate locking mechanisms (like ReadWriteLock) to handle concurrent operations.</li><li><strong>Offset Management</strong>: Implement a reliable offset tracking system that persists across restarts.</li><li><strong>File Formats</strong>: Design simple, robust file formats with clear separators between messages.</li><li><strong>Replication Strategy</strong>: Implement a leader-follower model for fault tolerance.</li><li><strong>Efficient I/O</strong>: Use modern I/O APIs (like NIO) for better performance.</li><li><strong>State Recovery</strong>: Ensure the system can rebuild its state after failures.</li></ol><p><a href="https://medium.com/@buildthingsuseful/stage-5-build-your-own-kafka-building-the-broker-e516c4757356">Next Article — Stage 5</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a8497a614f3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stage 3: Build Your Own Kafka — Implementing ZooKeeper Integration]]></title>
            <link>https://buildthingsuseful.medium.com/stage-3-build-your-own-kafka-3baaf4c873de?source=rss-5078f30a490b------2</link>
            <guid isPermaLink="false">https://medium.com/p/3baaf4c873de</guid>
            <category><![CDATA[kafka-producer]]></category>
            <category><![CDATA[message-queue]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[kafka-consumer]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Raghav]]></dc:creator>
            <pubDate>Thu, 13 Mar 2025 05:27:34 GMT</pubDate>
            <atom:updated>2025-03-13T05:51:21.360Z</atom:updated>
            <content:encoded><![CDATA[<h3>Implementing ZooKeeper Integration</h3><h3>Step 5: Create the ZooKeeper Client</h3><p>ZooKeeper provides distributed coordination services that Kafka uses for broker discovery, topic configuration, and leader election. This separation of concerns allows Kafka to focus on message handling while delegating coordination to ZooKeeper. The ZookeeperClient class serves as an interface between your Kafka implementation and ZooKeeper, handling:</p><ul><li>Connection management to ZooKeeper</li><li>Creation and management of ZooKeeper paths/nodes</li><li>Watching for changes in the ZooKeeper tree</li><li>Maintaining ephemeral nodes for broker liveness</li></ul><p>Create a ZookeeperClient.java file in the broker package:</p><p><strong>Connection Management</strong></p><pre>public void connect() throws IOException, InterruptedException {<br>    zooKeeper = new ZooKeeper(getConnectString(), SESSION_TIMEOUT, this);<br>    connectedSignal.await();<br>    <br>    // Create required paths if they don&#39;t exist<br>    createPath(&quot;/brokers&quot;);<br>    createPath(&quot;/topics&quot;);<br>    createPath(&quot;/controller&quot;);<br>}</pre><p>This method:</p><ol><li>Establishes a connection to ZooKeeper using the configured host and port</li><li>Waits for the connection to be established using a CountDownLatch</li><li>Creates the essential base paths needed for a Kafka system:</li></ol><ul><li>/brokers - Tracks active brokers</li><li>/topics - Stores topic configurations and partitions</li><li>/controller - Manages the controller broker information</li></ul><p><strong>Node Management</strong></p><p><strong>Creating Persistent Nodes</strong></p><pre>public void createPersistentNode(String path, String data) throws KeeperException, InterruptedException {<br>    Stat stat = zooKeeper.exists(path, false);<br>    if (stat == null) {<br>        zooKeeper.create(path, data.getBytes(), ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);<br>        LOGGER.info(&quot;Created persistent node: &quot; + path);<br>    } else {<br>        zooKeeper.setData(path, data.getBytes(), -1);<br>        LOGGER.info(&quot;Updated persistent node: &quot; + path);<br>    }<br>}</pre><p>Use this method for:</p><ul><li>Storing topic configurations (number of partitions, replication factor)</li><li>Saving partition assignments</li><li>Persisting consumer group offsets</li></ul><p><strong>Creating Ephemeral Nodes</strong></p><pre>public boolean createEphemeralNode(String path, String data) throws KeeperException, InterruptedException {<br>    Stat stat = zooKeeper.exists(path, false);<br>    if (stat == null) {<br>        zooKeeper.create(path, data.getBytes(), ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL);<br>        LOGGER.info(&quot;Created ephemeral node: &quot; + path);<br>        return true;<br>    } else {<br>        LOGGER.info(&quot;Ephemeral node already exists: &quot; + path);<br>        return false;<br>    }<br>}</pre><p>Ephemeral nodes are critical for broker registration:</p><ul><li>They exist only while the ZooKeeper session is active</li><li>If a broker crashes or disconnects, its ephemeral node disappears</li><li>This is how Kafka detects broker failures</li></ul><p><strong>Watches and Callbacks</strong></p><p><strong>Watching Child Nodes</strong></p><pre>public void watchChildren(String path, ChildrenCallback callback) {<br>    try {<br>        List&lt;String&gt; children = zooKeeper.getChildren(path, event -&gt; {<br>            if (event.getType() == Watcher.Event.EventType.NodeChildrenChanged) {<br>                try {<br>                    List&lt;String&gt; newChildren = zooKeeper.getChildren(path, event2 -&gt; {<br>                        if (event2.getType() == Watcher.Event.EventType.NodeChildrenChanged) {<br>                            watchChildren(path, callback);<br>                        }<br>                    });<br>                    callback.onChildrenChanged(newChildren);<br>                } catch (Exception e) {<br>                    LOGGER.log(Level.SEVERE, &quot;Error processing children changed event&quot;, e);<br>                }<br>            }<br>        });<br>        callback.onChildrenChanged(children);<br>    } catch (Exception e) {<br>        LOGGER.log(Level.SEVERE, &quot;Failed to watch children for path: &quot; + path, e);<br>    }<br>}</pre><p>This is used for:</p><ul><li>Detecting new brokers joining the cluster (watch /brokers)</li><li>Monitoring changes in topic configurations (watch /topics)</li><li>Tracking consumer group membership (watch /consumers/[group]/ids)</li></ul><p><strong>Watching Individual Nodes</strong></p><pre>public void watchNode(String path, NodeCallback callback) {<br>    try {<br>        zooKeeper.exists(path, event -&gt; {<br>            if (event.getType() == Watcher.Event.EventType.NodeDeleted ||<br>                event.getType() == Watcher.Event.EventType.NodeDataChanged ||<br>                event.getType() == Watcher.Event.EventType.NodeCreated) {<br>                callback.onNodeChanged();<br>            }<br>        });<br>    } catch (Exception e) {<br>        LOGGER.log(Level.SEVERE, &quot;Failed to watch node: &quot; + path, e);<br>    }<br>}</pre><p>Use this to:</p><ul><li>Monitor controller changes</li><li>Watch partition leader elections</li><li>Track changes to topic configurations</li></ul><p><strong>Session Management</strong></p><p>The process method handles ZooKeeper session events:</p><pre>@Override<br>public void process(WatchedEvent event) {<br>    if (event.getState() == Event.KeeperState.SyncConnected) {<br>        connectedSignal.countDown();<br>        LOGGER.info(&quot;Connected to ZooKeeper&quot;);<br>    } else if (event.getState() == Event.KeeperState.Disconnected) {<br>        LOGGER.warning(&quot;Disconnected from ZooKeeper&quot;);<br>    } else if (event.getState() == Event.KeeperState.Expired) {<br>        LOGGER.warning(&quot;ZooKeeper session expired, reconnecting...&quot;);<br>        try {<br>            if (zooKeeper != null) {<br>                zooKeeper.close();<br>            }<br>            connectedSignal = new CountDownLatch(1);<br>            zooKeeper = new ZooKeeper(getConnectString(), SESSION_TIMEOUT, this);<br>            connectedSignal.await();<br>            LOGGER.info(&quot;Reconnected to ZooKeeper after session expiry&quot;);<br>        } catch (Exception e) {<br>            LOGGER.log(Level.SEVERE, &quot;Failed to reconnect to ZooKeeper&quot;, e);<br>        }<br>    }<br>}</pre><p>This handles:</p><ul><li>Initial connection establishment</li><li>Reconnection after disconnects</li><li>Session expiry (critical for ephemeral nodes)</li></ul><h3>Implementation Guide for Build Your Own Kafka-like System</h3><p><strong>1: Broker Registration</strong></p><p>When a broker starts:</p><ol><li>Connect to ZooKeeper using the connect() method</li><li>Register the broker by creating an ephemeral node:</li></ol><pre>String brokerData = &quot;{\\&quot;host\\&quot;:\\&quot;localhost\\&quot;,\\&quot;port\\&quot;:9092,\\&quot;endpoints\\&quot;:[]}&quot;; createEphemeralNode(&quot;/brokers/ids/1&quot;, brokerData);</pre><p><strong>2: Topic Management</strong></p><p>For creating topics:</p><ol><li>Create a persistent node for the topic:</li></ol><pre>String topicConfig = &quot;{\\&quot;partitions\\&quot;:3,\\&quot;replication_factor\\&quot;:2}&quot;; createPersistentNode(&quot;/topics/my-topic&quot;, topicConfig);</pre><ol><li>For each partition, create nodes for replica assignment:</li></ol><pre>// For partition 0 with replicas on brokers 1, 2 <br>createPersistentNode(&quot;/topics/my-topic/partitions/0/state&quot;,                      &quot;{\\&quot;leader\\&quot;:1,\\&quot;isr\\&quot;:[1,2],\\&quot;replicas\\&quot;:[1,2]}&quot;);</pre><p><strong>3: Controller Election</strong></p><p>For controller election:</p><ol><li>When a broker wants to become a controller, try to create an ephemeral node:</li></ol><pre>boolean becameController = createEphemeralNode(&quot;/controller&quot;,                                              &quot;{\\&quot;broker\\&quot;:1,\\&quot;timestamp\\&quot;:123456789}&quot;);</pre><ol><li>If successful, the broker is now the controller</li><li>Other brokers should watch this node for changes:</li></ol><pre>watchNode(&quot;/controller&quot;, () -&gt; handleControllerChange());</pre><p><strong>4: Broker Failure Detection</strong></p><p>To detect broker failures:</p><ol><li>Watch for changes in the broker list:</li></ol><pre>watchChildren(&quot;/brokers/ids&quot;, children -&gt; {     // Handle broker list changes     // If previously known broker missing, initiate leader elections });</pre><p><strong>5: Partition Leadership</strong></p><p>For partition leadership:</p><ol><li>The controller assigns leaders and ISRs (In-Sync Replicas)</li><li>Brokers watch for leadership changes:</li></ol><pre>watchNode(&quot;/topics/my-topic/partitions/0/state&quot;, () -&gt; {     String state = getData(&quot;/topics/my-topic/partitions/0/state&quot;);     // Parse state JSON and check if this broker is the leader });</pre><h3>Key Patterns to Understand</h3><ol><li><strong>Ephemeral Nodes for Liveness</strong>: Kafka uses ephemeral nodes to track active brokers</li><li><strong>Watch-Based Notifications</strong>: Changes propagate via watches rather than polling</li><li><strong>Hierarchical Data Structure</strong>: ZooKeeper’s path-based structure organizes Kafka metadata</li><li><strong>Session Expiry Handling</strong>: Critical for proper broker failure detection</li></ol><p><a href="https://medium.com/@buildthingsuseful/stage-4-build-your-own-kafka-building-the-storage-layer-7a8497a614f3">Next Article — Stage 4</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3baaf4c873de" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stage 2: Building Your Own Kafka — Creating the Core Protocol Layer]]></title>
            <link>https://buildthingsuseful.medium.com/stage-2-building-your-own-kafka-creating-the-core-protocol-layer-5838c1b2364c?source=rss-5078f30a490b------2</link>
            <guid isPermaLink="false">https://medium.com/p/5838c1b2364c</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[kafka-producer]]></category>
            <category><![CDATA[kafka-consumer]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[message-queue]]></category>
            <dc:creator><![CDATA[Raghav]]></dc:creator>
            <pubDate>Thu, 13 Mar 2025 05:09:20 GMT</pubDate>
            <atom:updated>2025-03-13T05:57:19.467Z</atom:updated>
            <content:encoded><![CDATA[<h3>Stage 2: Building Your Own Kafka — Creating the Core Protocol Layer</h3><h3>Creating the Core Protocol Layer</h3><h4>Step 3: Create the BrokerInfo Class</h4><p>This small class holds information about each broker in the cluster</p><pre>package com.simplekafka.broker;<br><br>/**<br> * Holds information about a broker in the SimpleKafka cluster<br> */<br>public class BrokerInfo {<br>    private final int id;<br>    private final String host;<br>    private final int port;<br>    <br>    public BrokerInfo(int id, String host, int port) {<br>        this.id = id;<br>        this.host = host;<br>        this.port = port;<br>    }<br>    <br>    public int getId() {<br>        return id;<br>    }<br>    <br>    public String getHost() {<br>        return host;<br>    }<br>    <br>    public int getPort() {<br>        return port;<br>    }<br>    <br>    @Override<br>    public String toString() {<br>        return &quot;BrokerInfo{id=&quot; + id + &quot;, host=&#39;&quot; + host + &quot;&#39;, port=&quot; + port + &quot;}&quot;;<br>    }<br>    <br>    @Override<br>    public boolean equals(Object obj) {<br>        if (this == obj) return true;<br>        if (obj == null || getClass() != obj.getClass()) return false;<br>        <br>        BrokerInfo other = (BrokerInfo) obj;<br>        return id == other.id;<br>    }<br>    <br>    @Override<br>    public int hashCode() {<br>        return Integer.hashCode(id);<br>    }<br>}</pre><h4>Step 4: Define the Wire Protocol</h4><p>The wire protocol is the binary format that brokers and clients use to communicate. Create a Protocol.java file in the broker package. The Protocol class is the foundation of Build Your Own Kafka&#39;s wire protocol implementation, defining how clients and brokers communicate over the network. I will explain the key components and methods that are crucial for understanding how to build a Kafka-like system that supports such wire protocol.</p><p>Kafka uses a binary protocol for efficiency. Each message type is identified by a unique byte code, followed by the message payload. This approach is more efficient than text-based protocols like HTTP for high-throughput messaging.</p><p><strong>Message Types and Constants</strong></p><pre>// Client request types<br>public static final byte PRODUCE = 0x01;<br>public static final byte FETCH = 0x02;<br>public static final byte METADATA = 0x03;<br>public static final byte CREATE_TOPIC = 0x04;<br>    <br>// Broker response types<br>public static final byte PRODUCE_RESPONSE = 0x11;<br>public static final byte FETCH_RESPONSE = 0x12;<br>// ...and others</pre><p>These constants define the message types for different operations in the system. Each operation (produce, fetch, metadata, create topic, etc.) has a unique byte identifier that helps the receiver identify the type of request or response.</p><p><strong>Request Encoding Methods</strong></p><ol><li><strong>Produce Request</strong></li></ol><pre>public static ByteBuffer encodeProduceRequest(String topic, int partition, byte[] message)</pre><p><strong>Purpose</strong>: Encodes a request to produce (write) a message to a specific topic and partition</p><p><strong>Key components</strong>:</p><ul><li>Request type byte (PRODUCE)</li><li>Topic name length and bytes</li><li>Partition ID</li><li>Message length and content</li></ul><p><strong>Structure</strong>:</p><ol><li>First byte identifies request type (0x01)</li><li>Next 2 bytes contain topic string length</li><li>Next N bytes contain the topic name</li><li>Next 4 bytes contain partition ID</li><li>Next 4 bytes contain message length</li><li>Remaining bytes contain the message</li></ol><p><strong>2. Fetch Request</strong></p><pre>public static ByteBuffer encodeFetchRequest(String topic, int partition, long offset, int maxBytes)</pre><p><strong>Purpose</strong>: Encodes a request to fetch (read) messages from a specific topic, partition, starting from a given offset</p><p><strong>Key parameters</strong>: Topic, partition, starting offset, and maximum bytes to fetch</p><p><strong>Structure</strong>: Similar to produce request but includes offset and maxBytes parameters</p><p><strong>3. Metadata Request</strong></p><pre>public static ByteBuffer encodeMetadataRequest()</pre><p><strong>Purpose</strong>: Encodes a request to retrieve metadata about brokers and topics in the cluster</p><p><strong>Structure</strong>: Simplest request — just the request type byte (0x03)</p><p><strong>4. Create Topic Request</strong></p><pre>public static ByteBuffer encodeCreateTopicRequest(String topic, int numPartitions, short replicationFactor)</pre><p><strong>Purpose</strong>: Encodes a request to create a new topic with specified partitions and replication</p><p><strong>Key parameters</strong>: Topic name, number of partitions, and replication factor</p><p><strong>Response Decoding Methods</strong></p><ol><li><strong>Decode Produce Response</strong></li></ol><pre>public static ProduceResult decodeProduceResponse(ByteBuffer buffer)</pre><p><strong>Purpose</strong>: Decodes a response from a produce request</p><p><strong>Return value</strong>: A ProduceResult containing:</p><ul><li>Offset where the message was written</li><li>Error information (if any)</li></ul><p><strong>Process</strong>:</p><ol><li>Checks if response type matches PRODUCE_RESPONSE</li><li>Extracts offset and status information</li><li>Handles error responses</li></ol><p><strong>2. Decode Fetch Response</strong></p><pre>public static FetchResult decodeFetchResponse(ByteBuffer buffer)</pre><p><strong>Purpose</strong>: Decodes a response from a fetch request</p><p><strong>Return value</strong>: A FetchResult containing:</p><ul><li>Array of message byte arrays</li><li>Error information (if any)</li></ul><p><strong>Process</strong>:</p><ol><li>Verifies correct response type</li><li>Extracts message count</li><li>Processes each message with its offset and size</li></ol><p><strong>Decode Metadata Response</strong></p><pre>public static MetadataResult decodeMetadataResponse(ByteBuffer buffer)</pre><p><strong>Purpose</strong>: Decodes a response containing cluster metadata</p><p><strong>Return value</strong>: A MetadataResult containing:</p><ul><li>List of broker information</li><li>List of topic metadata with partition information</li><li>Error information (if any)</li></ul><p><strong>Process</strong>:</p><ol><li>Processes broker information (IDs, hosts, ports)</li><li>Processes topic metadata (partitions, leaders, replicas)</li></ol><p><strong>Replication and Internal Communication</strong></p><pre>public static ByteBuffer encodeReplicateRequest(String topic, int partition, long offset, byte[] message)<br>public static ByteBuffer encodeTopicNotification(String topic)</pre><p><strong>Purpose</strong>: Support broker-to-broker communication for replication and topic updates</p><p><strong>Key features</strong>:</p><ul><li>Similar encoding pattern to client-facing requests</li><li>Contains necessary metadata for replication</li></ul><p><strong>Result Classes</strong></p><p>The Protocol class includes nested result classes that encapsulate response data:</p><p><strong>ProduceResult</strong>: Contains offset and error information for produce operations</p><p><strong>FetchResult</strong>: Contains arrays of messages and error information for fetch operations</p><p><strong>MetadataResult</strong>: Contains broker and topic metadata along with error information</p><p><strong>TopicMetadata</strong>: Contains topic name and partition metadata</p><p><strong>PartitionMetadata</strong>: Contains partition ID, leader broker ID, and replica broker IDs</p><p><strong>Error Handling</strong></p><pre>public static void sendErrorResponse(SocketChannel channel, String errorMessage)</pre><p><strong>Purpose</strong>: Provides a standardized way to send error responses to clients</p><p><strong>Process</strong>:</p><ol><li>Encodes error response with ERROR_RESPONSE type byte</li><li>Includes error message length and text</li><li>Writes to socket channel</li></ol><p><strong>Key Implementation Considerations</strong></p><ol><li><strong>Binary Protocol</strong>: The entire protocol is binary-based for efficiency, using ByteBuffer for serialization.</li><li><strong>Length Prefixing</strong>: Strings and arrays are prefixed with their length for proper deserialization.</li><li><strong>Type Identification</strong>: Each message begins with a type byte for easy identification.</li><li><strong>Error Handling</strong>: Each response includes error checking mechanisms and can return specific error messages.</li><li><strong>Consistent Patterns</strong>: The encoding/decoding methods follow consistent patterns, making the protocol extensible.</li></ol><p><strong>Why Kafka Uses a Binary Protocol</strong></p><p>Kafka’s choice of a binary protocol over alternatives like JSON, XML, or REST APIs offers several significant benefits for a distributed messaging system:</p><ol><li><strong>Performance and Efficiency</strong></li></ol><p><strong>Reduced Size</strong>: Binary representations are significantly more compact than text-based formats like JSON or XML. A 4-byte integer is always 4 bytes in binary, but could be 1–10 characters in a text format.</p><p><strong>Parsing Efficiency</strong>: Binary protocols require less CPU processing to encode and decode, as they don’t need complex parsing logic like text-based formats do.</p><p><strong>Network Efficiency</strong>: Smaller message sizes mean less network bandwidth usage and faster transmission times, crucial for a high-throughput system like Kafka.</p><p><strong>2. Precise Type Control</strong></p><p><strong>Type Safety</strong>: Binary protocols maintain exact data types (integers, floats, strings) without conversion ambiguities that can occur in text formats.</p><p><strong>Fixed Schemas</strong>: With defined message structures, binary protocols ensure data consistency and prevent type mismatches or interpretation errors.</p><p><strong>3. Performance at Scale</strong></p><p><strong>Throughput Optimization</strong>: Kafka is designed to handle millions of messages per second. Even small inefficiencies in a text-based protocol would multiply into significant overhead at this scale.</p><p><strong>Reduced Garbage Collection</strong>: In JVM-based systems like Kafka, working with raw bytes reduces object creation and garbage collection pressure compared to parsing JSON/XML.</p><p><strong>4. Versioning and Compatibility</strong></p><p><strong>Schema Evolution</strong>: Well-designed binary protocols can support forward and backward compatibility, allowing gradual system updates without breaking existing clients.</p><p><strong>Field Addition/Removal</strong>: The Build Your Own Kafka protocol shown in your example defines clear message structures that can be extended while maintaining compatibility with older clients.</p><p><strong>5. Low-level Control</strong></p><p><strong>Direct Memory Management</strong>: Binary protocols allow direct manipulation of byte buffers (as shown in the code with ByteBuffer), enabling zero-copy optimizations.</p><p><strong>Network-Level Optimizations</strong>: Working directly with bytes allows for better integration with low-level networking APIs.</p><p><a href="https://buildthingsuseful.medium.com/stage-3-build-your-own-kafka-3baaf4c873de">Next Article</a> — Stage 3</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5838c1b2364c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stage 1: Building Your Own Kafka-like System From Scratch: A Step-by-Step Guide]]></title>
            <link>https://buildthingsuseful.medium.com/building-your-own-kafka-like-system-from-scratch-a-step-by-step-guide-d3c5f0a303c0?source=rss-5078f30a490b------2</link>
            <guid isPermaLink="false">https://medium.com/p/d3c5f0a303c0</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[step-by-step-guide]]></category>
            <category><![CDATA[message-broker]]></category>
            <category><![CDATA[message-queue]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Raghav]]></dc:creator>
            <pubDate>Thu, 13 Mar 2025 05:07:04 GMT</pubDate>
            <atom:updated>2025-03-13T05:56:43.664Z</atom:updated>
            <content:encoded><![CDATA[<p>If you’re looking to truly understand Kafka’s architecture by implementing a simplified version yourself, you’ve come to the right place. Rather than just copying code, we’ll build SimpleKafka incrementally, understanding each component as we go. This approach will give you a deep understanding of distributed messaging systems. Follow along with this multi-part series, starting with Stage 1.</p><h3>Workflow Summary</h3><p>This incremental approach allows you to build and understand each component of a Kafka-like system:</p><ol><li><strong>Set up the project structure</strong></li><li><strong>Create the core protocol layer</strong></li><li><strong>Implementing Zookeeper Integration</strong></li><li><strong>Building the storage layer</strong></li><li><strong>Build the broker</strong></li><li><strong>Develop the client library</strong></li><li><strong>Building higher level producer and consumer APIs</strong></li><li><strong>Test the system</strong></li></ol><p>By building each component yourself, you’ll gain a deep understanding of Kafka’s architecture and the design decisions behind it. This knowledge will be invaluable when working with the real Kafka or designing your own distributed systems.</p><p>Remember to take it one step at a time and test each component thoroughly before moving on. Distributed systems are complex, but breaking them down into manageable pieces makes them much easier to understand.</p><p>Happy coding, and enjoy your journey into the world of distributed messaging systems!</p><h3>Stage 1: Setting Up the Project</h3><h4>Step 1: Create the Project Structure</h4><p>Start with setting up a basic Maven project structure in your IDE:</p><pre>build-your-own-kafka/<br>├── src/<br>│   ├── main/<br>│   │   ├── java/<br>│   │   │   └── com/<br>│   │   │       └── simplekafka/<br>│   │   │           ├── broker/<br>│   │   │           └── client/<br>│   ├── test/<br>│   │   └── java/<br>└── pom.xml</pre><h4>Step 2: Set Up the Maven Configuration</h4><p>Create a basic pom.xml file:</p><pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;<br>&lt;project xmlns=&quot;&lt;http://maven.apache.org/POM/4.0.0&gt;&quot;<br>         xmlns:xsi=&quot;&lt;http://www.w3.org/2001/XMLSchema-instance&gt;&quot;<br>         xsi:schemaLocation=&quot;&lt;http://maven.apache.org/POM/4.0.0&gt; &lt;http://maven.apache.org/xsd/maven-4.0.0.xsd&gt;&quot;&gt;<br>    &lt;modelVersion&gt;4.0.0&lt;/modelVersion&gt;</pre><pre>    &lt;groupId&gt;com.simplekafka&lt;/groupId&gt;<br>    &lt;artifactId&gt;simple-kafka&lt;/artifactId&gt;<br>    &lt;version&gt;1.0-SNAPSHOT&lt;/version&gt;</pre><pre>    &lt;properties&gt;<br>        &lt;maven.compiler.source&gt;11&lt;/maven.compiler.source&gt;<br>        &lt;maven.compiler.target&gt;11&lt;/maven.compiler.target&gt;<br>    &lt;/properties&gt;</pre><pre>    &lt;dependencies&gt;<br>        &lt;dependency&gt;<br>            &lt;groupId&gt;org.apache.zookeeper&lt;/groupId&gt;<br>            &lt;artifactId&gt;zookeeper&lt;/artifactId&gt;<br>            &lt;version&gt;3.8.1&lt;/version&gt;<br>        &lt;/dependency&gt;<br>    &lt;/dependencies&gt;<br>    <br>    &lt;build&gt;<br>    &lt;plugins&gt;<br>      &lt;plugin&gt;<br>        &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;<br>        &lt;artifactId&gt;maven-shade-plugin&lt;/artifactId&gt;<br>        &lt;version&gt;3.4.1&lt;/version&gt;<br>        &lt;executions&gt;<br>          &lt;execution&gt;<br>            &lt;phase&gt;package&lt;/phase&gt;<br>            &lt;goals&gt;<br>              &lt;goal&gt;shade&lt;/goal&gt;<br>            &lt;/goals&gt;<br>            &lt;configuration&gt;<br>              &lt;transformers&gt;<br>                &lt;transformer<br>                  implementation=&quot;org.apache.maven.plugins.shade.resource.ManifestResourceTransformer&quot;&gt;<br>                  &lt;mainClass&gt;com.simplekafka.broker.SimpleKafkaBroker&lt;/mainClass&gt;<br>                &lt;/transformer&gt;<br>                &lt;transformer<br>                  implementation=&quot;org.apache.maven.plugins.shade.resource.ServicesResourceTransformer&quot; /&gt;<br>              &lt;/transformers&gt;<br>            &lt;/configuration&gt;<br>          &lt;/execution&gt;<br>        &lt;/executions&gt;<br>      &lt;/plugin&gt;<br>    &lt;/plugins&gt;<br>  &lt;/build&gt;<br>&lt;/project&gt;</pre><p>Project organization is crucial for maintaining a clear separation of broker and client code, which reflects Kafka’s architecture of separate server and client components.</p><p><a href="https://medium.com/@buildthingsuseful/stage-2-building-your-own-kafka-creating-the-core-protocol-layer-5838c1b2364c">Next Article — Stage 2</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d3c5f0a303c0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Flow — Book Notes — 1]]></title>
            <link>https://buildthingsuseful.medium.com/flow-book-notes-1-df98ed2dd3a8?source=rss-5078f30a490b------2</link>
            <guid isPermaLink="false">https://medium.com/p/df98ed2dd3a8</guid>
            <category><![CDATA[consciousness]]></category>
            <category><![CDATA[philosophy]]></category>
            <category><![CDATA[happiness]]></category>
            <category><![CDATA[focus]]></category>
            <category><![CDATA[achieving-goals]]></category>
            <dc:creator><![CDATA[Raghav]]></dc:creator>
            <pubDate>Mon, 26 Oct 2020 17:37:12 GMT</pubDate>
            <atom:updated>2020-10-26T17:37:12.341Z</atom:updated>
            <content:encoded><![CDATA[<h3>Flow — Book Notes — 1</h3><p><strong>Flow: The Psychology Of Optimal Experience</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/328/0*I8QpJRxOhln30RKp.jpg" /></figure><p>Whenever we come across the word <em>Psychology </em>people tend to think about an experience that deals with the mind and behavioral attitude of a person in general. Dealing in such situations could lead to pleasant or unpleasant experiences. The book talks about the experience in the field of psychology in a completely different form. <em>In some form, it connects the western discovery with yogic system.</em> The author mentions <strong>You<em> </em></strong>being the <em>Protagonist</em> of your own experiences of your life. The experiences that you build upon will manifest to the outer world in various forms. Some of them could be good fortune, affluent lifestyles, power or misery. The ultimate experience is not about the materialistic world that we live in, our desires, something to acquire, or suffer. Instead it is in every moment in your life that you enjoy becomes a natural way to happiness.</p><p>But wait. Every person in this world would have heard someone saying <em>enjoy every moment of your life </em>without mentioning how to do that at every step of your life? This is exactly what this book talks about. While I make the notes for the book, I am also capturing my own life experiences in topics that have resonated on me.</p><p>The subject itself is complex not because it is difficult to follow or intricate naturally, instead it is extremely hard to explain to others. The author has taken a bold step in changing life perspective for all sort of people in the world. No matter where you live, your religion or identity the framework mentioned in the book helps to determine <em>Optimal Experience</em>.</p><p><strong>Introduction</strong></p><p>Happiness is not something that you missed in the past or supposed to happen in the future. It is an experience that you prepare, cultivate and defend individually by you. People who learn to control their inner experience will be able to achieve utmost gratification that is close enough to be exhilaration.</p><p>How do we achieve it? It requires a <em>commitment</em> that begins with achieving control over the <em>contents of our consciousness</em>. The book will delve into more details on consciousness and how it relates to the happiness of our lives.</p><p>One important thing to remember is that albeit it seems the practice is simple but it is not easy. It requires courage and commitment in practicing every minute of the day for the next 10–15 years. Only then you could call yourself that you have achieved some sense of mastery.</p><p>What we are today are shaped by our experiences in the past which could be good or bad. Most of the forces that shape our experiences are outside of our control. In some cases, instead of being buffeted by various forces we do feel in control of our actions, emotions, feelings and thoughts. On rare occasions that happens we feel a sense of exhilaration, a deep sense of joy that is long cherished and that becomes a landmark in our memory and what life should be like. This is called <strong>Optimal Experience</strong>.</p><p>The best moments in life is when a person’s body or mind is unleashed to its full potential in a voluntary effort to achieve something difficult and worthwhile. Optimal experience is thus to make it happen.</p><p>Getting control and adding conscious content in your life is never easy and it is mostly a painful experience. But in the long run the optimal experiences add up to a sense of mastery — or perhaps better, a sense of participation in determining the content of your life.</p><p>Optimal experience is the state of the mind. <strong>Flow</strong> is the concept in which we achieve the optimal experience.</p><p>So then what is Flow? Author says “<em>the state in which people are involved in an activity that nothing else seems to matter; the experience itself is so enjoyable that people will do it even at great cost, for the sheer sake of doing it</em>”.</p><p>I could relate one of my real life example when I discovered <a href="https://en.wikipedia.org/wiki/Jim_Rohn">Jim Rohn </a>podcast on leading an inspired life. It was resonating on my life experiences that I decided to drove 200 miles up north of California to complete the three hours podcast in a stretch. After completion, I felt a sense of ecstasy and landmark in my mind that enlightened personal development.</p><p><em>Remember without some intellectual effort, a commitment to reflect and think hard about your own experience, you will not gain much from the book.</em></p><p>As we delve deep into the subject, we shall zoom into each of those following in details.</p><ul><li>How subjective states are shaped in mind? How consciousness works and how it is controlled? Everything we experience — joy or pain, interest or boredom — is represented in the mind as information. If we are able to control this information, only then we can decide what our lives will be like.</li><li>“The optimal state of inner experience is one in which there is <em>order in consciousness. </em>This happens when <em>psychic energy — or attention — is invested in realistic goals</em>, and when <em>skills match the opportunities for action</em>. <em>The pursuit of a goal brings order in awareness</em> because a person must concentrate attention on the task at hand and momentarily forget everything else.” These periods of struggling to overcome challenges are what people find to be the most enjoyable times of their lives.</li><li><em>Conditions of the flow</em> — why some things we do are more enjoyable than others. “Flow” is the way people describe their state of mind when consciousness is harmoniously ordered, and they want to pursue whatever they are doing for its own sake.</li><li>To achieve control over what happens in the mind one can draw upon infinite range of opportunities for enjoyment — through the use of <em>physical and sensory skills</em>. E.g. Sports, Yoga or through the development of <em>symbolic skills </em>such as poetry, philosophy, or mathematics.</li><li><em>Transform jobs into flow-producing activities.</em></li><li><em>Relations with parents, spouses, children and friends more enjoyable</em></li><li>Many lives are disrupted by tragic accidents, and even the most fortunate are subjected to stresses of various kinds. Yet such blows do not necessarily diminish happiness. It is how people respond to stress that determines whether they will profit from misfortune or be miserable.</li><li>And finally, how people manage to <em>join all experience into a meaningful pattern. </em>When that is accomplished, and a person feels in control of life and feels that it makes sense, there is nothing left to desire.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=df98ed2dd3a8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setting up site-to-site VPN on AWS — Part 2]]></title>
            <link>https://buildthingsuseful.medium.com/setting-up-site-to-site-vpn-on-aws-part-2-ca468df079db?source=rss-5078f30a490b------2</link>
            <guid isPermaLink="false">https://medium.com/p/ca468df079db</guid>
            <category><![CDATA[aws-vpn]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[vpn]]></category>
            <category><![CDATA[ipsec]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Raghav]]></dc:creator>
            <pubDate>Mon, 26 Aug 2019 03:06:00 GMT</pubDate>
            <atom:updated>2019-08-29T06:04:02.496Z</atom:updated>
            <content:encoded><![CDATA[<h3>Setting up site-to-site VPN on AWS — Part 2</h3><p>This post is part 2 of a two-part series about site-to-site VPN on AWS. In part 1, we covered the basics of the VPN. This post covers in detail the set up of site-to-site VPN on AWS.</p><p>To begin let us first walk through the AWS site-to-site VPN workflow and terminologies. The below picture depicts the setup of AWS site-to-site VPN with a corporate network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d34vW3js7LDMdYkv9NHkng.png" /><figcaption>Typical AWS site-to-site VPN connectivity with corporate network</figcaption></figure><p>Above we have AWS VPC with two subnets. Public subnet CIDR 10.0.0.0/24 having a route to internet gateway (igw). Private subnet CIDR 10.0.1.0/24 having a route to Virtual Private Gateway (vgw) connecting to customer gateway (cgw).</p><p><strong>Internet Gateway (igw)</strong> — It allows traffic from/to the Internet. Any subnet that has a route to igw will be accessible from the Internet. It is a highly available component within VPC.</p><p><strong>Virtual Private Gateway (vgw)</strong> — It is the component for setting up the VPN connection to other networks. It is a highly available component within VPC.</p><p><strong>Customer Gateway (cgw)</strong> — It is an integration point for the connectivity between your network and the AWS VPC network.</p><p>Because we don’t have an actual corp network to set up a site-to-site VPN. We can simulate a similar network with another VPC in a different region. We will be running VPN software on an EC2 instance. Remember VPN gateway can be within a router, or dedicated VPN gateway or software running on a machine. The below picture shows the simulated site-to-site VPN network between two regions within AWS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M5ggxw7BndjKAR9JYl3irQ.png" /><figcaption>Site-to-site VPN network between two AWS regions (Simulation)</figcaption></figure><p>As you follow along, I will describe the details for each step for setting the VPN connection. <strong>Note for inter-VPC network connectivity we have a concept called VPC peering. Don’t inter-relate VPC peering here. It is completely a different concept. Here we are only interested in setting up VPN connections between networks. We are using this as the setup for demo. In reality for connecting between VPCs, we will be using VPC peering.</strong></p><p>Few things to mention before we get started.</p><p>We need to make sure the IPs don’t overlap, hence in the above example, we have two different CIDRs for each network. Both of them are from <a href="https://tools.ietf.org/html/rfc1918">RFC 1918</a> range.</p><p>You can choose any AWS region to setup. I will be following as per the picture above.</p><p><strong>US-West (Oregon) Region</strong></p><p><strong>Step 1</strong>: Create a VPC with CIDR <strong>10.0.0.0/16</strong> in US-West (Oregon) Region. Immediately after creating a VPC you will find a route table created by default. Notice the route table has an indicator ‘Main’ — Yes. This means any subnet that you create will have the default association with this route table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/913/1*BD5aDrhYqjILy9fB51R2Bg.png" /><figcaption>VPC Creation</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a7GgxUgtJEvVxd6ZCVlfiw.png" /><figcaption>Default Route Table</figcaption></figure><p><strong>Step 2</strong>: Create a public subnet with CIDR <strong>10.0.0.0/24</strong>. To create a public subnet we need to create an internet gateway and attach it to the VPC. Create a separate routing table. Attach the public subnet to the newly created routing table. Add routes (0.0.0.0/0) to the internet in the newly created routing table. (Following the <a href="https://en.wikipedia.org/wiki/Principle_of_least_privilege">Principle of least privilege</a>). This ensures only associated subnets will have a route to the internet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ukkOFTX5Y6dFCD_mHG1dCg.png" /><figcaption>Internet Gateway Attached To The VPC</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qb0vmXuRt6EMnD5pjvNshQ.png" /><figcaption>Newly created route table for internet access from public subnets</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Aea2Y77RCMzFFQZjt_N4Q.png" /><figcaption>Create public subnet</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*38VlaroKv_XlPKIsU2UCDw.png" /><figcaption>Explicitly associate a public subnet with a newly created route table</figcaption></figure><p><strong>Step 3</strong>: Create a private subnet with CIDR <strong>10.0.1.0/24</strong>. Create a virtual private gateway (vgw) and attach it to the VPC. In the main route table select “Yes” for route propagation associated with the vgw. (Route propagation is used so that you don’t have to manually type the routes for vgw in the route table. When vgw changes its IPs it will be automatically updated in the route table). This will ensure all packets destined for 0.0.0.0/0 will be routed to vgw in the private subnets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vJH3mY-oi3SNvvzf5vciJA.png" /><figcaption>Private subnet</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b5UvCjvVICIp8Lh4e61zcQ.png" /><figcaption>Create Virtual Private Gateway (vgw)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EZAQ8WPcFU1dxUqSydk73w.png" /><figcaption>Route Propagation Select as Yes in the Main Route Table</figcaption></figure><p><em>Quick summary: So far in this section, we have created a VPC with two subnets public and private. Additionally, we have created an internet gateway, virtual private gateway, and a new route table. In the new route table, we added an entry for internet gateway and associated it with the public subnet. In the main route table, we added an entry for the virtual private gateway.</em></p><p><strong>US-East (N Virginia)</strong></p><p><strong>Step 3: </strong>Similar to Step 1 and Step 2 of the US-West (Oregon) region. Create a VPC with CIDR <strong>172.16.0.0/16 </strong>in US-East (N Virginia) region. Follow the steps and create a public subnet with <strong>CIDR 172.16.0.0/24</strong>. Create an internet gateway. Create a new route table, add a new entry for internet gateway with destination 0.0.0.0/0. Associate the public subnet with this new route table.</p><p><strong>Step 4: </strong>Create an EC2 instance in the public subnet. I am using Amazon Linux 2 AMI image for the EC2 instance. The instance type is t2.micro. Ensure you have selected the Auto-assign Public IP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9cPiVL9gF2Mtdofw5gm2Cw.png" /><figcaption>EC2 Instance in Public Subnet — US-East (N Virginia)</figcaption></figure><p><strong>Step 5: </strong>Create a new security group. Add rules for SSH access from your IP only. Add rules for All ICMP — IPv4. Add rules for All TCP. In the source section select custom by adding US-West (Oregon) region VPC CIDR 10.0.0.0/16 to both of them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*dyuMez6XTagvW0vYnkbBbQ.png" /><figcaption>EC2 Instance Security Group — US-East (N Virginia)</figcaption></figure><p><strong>Step 6: </strong>SSH to the EC2 instance using public IP. Run the following commands with sudo access. <a href="https://www.openswan.org/">Openswan</a> is the VPN software used in Linux.</p><pre>yum -y update</pre><pre>yum -y install openswan</pre><p><em>Quick summary: In this section, we have created a VPC with one public subnet. Additionally, we have created an internet gateway and a new route table. In the new route table, we added an entry for internet gateway and associated it with the public subnet. We created an EC2 instance by attaching security groups for SSH, ICMP and TCP traffic. Installed openswan on the EC2 instance.</em></p><p><strong>US-West (Oregon) Region</strong></p><p><strong>Step 7: </strong>Create an EC2 instance in the public subnet of the US-West (Oregon) region. Ensure you have selected the Auto-assign Public IP. Create a security group with only SSH access from your IP. This instance is mainly used for SSH into private instances from a public instance (similar to bastion host).</p><p><strong>Step 8: </strong>Create an EC2 instance in the private subnet of the US-West (Oregon) region. Create a new security group. Add rules for SSH access from your IP only. Add rules for All ICMP — IPv4. Add rules for All TCP. In the source section select custom by adding US-West (Oregon) region VPC CIDR 172.16.0.0/16 to both of them.</p><p><strong>Step 9: </strong>Create a customer gateway. Select static routing. Add the public IP address of the US-East (N Virginia) region in the IP Address field.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6_oob2cn570lNuAuLljXSQ.png" /><figcaption>Customer Gateway (cgw) — US West (Oregon)</figcaption></figure><p><strong>Step 10: </strong>Navigate to Site-to-Site VPN connections. Create a VPN connection. Select the VPN gateway, customer gateway and choose static routing. In the static IP address field add the CIDR 172.16.0.0/16 which is US-East (N Virginia) region. Leave the rest of the field&#39;s default. Once the VPN connection is available you should see two tunnels created. Currently, those two tunnels show the status has DOWN.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sBvwEDDfeAfqyjHLH4ZbWQ.png" /><figcaption>Create a site-to-site VPN connection — US West (Oregon) region</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E31OVAVqSJoKwo0-3NBclA.png" /><figcaption>VPN connection — US West (Oregon) region</figcaption></figure><p><strong>Step 11: </strong>While it takes some time for the VPN connection to be available, let us move forward with other steps. Next click on the download configuration on the VPN connection page. In the vendor drop-down select Openswan. Save the configuration in a file locally. This file contains the configuration for the Openswan setup of VPN.</p><p><em>Quick summary: In this section, we have created a VPC connection using site-to-site VPN. Configured the customer gateway, virtual private gateway and downloaded the appropriate configuration for openswan.</em></p><p><strong>US-East (N Virginia) Region</strong></p><p><strong>Step 12: </strong>Let us set up the openswan configuration by following the steps in the downloaded file. SSH into the public EC2 instance. The first step is to make the VPN gateway server (EC2 instance) act as a router for all other hosts in the network.</p><pre>vi /etc/sysctl.conf</pre><pre>net.ipv4.ip_forward = 1<br>net.ipv4.conf.default.rp_filter = 0<br>net.ipv4.conf.default.accept_source_route = 0</pre><p>Enable the changes by running the following command</p><pre>sysctl -p /etc/sysctl.conf</pre><p>Open <strong>/etc/ipsec.conf</strong> and look for the line below. Ensure that the # in front of the line has been removed, then save and exit the file.</p><pre>include /etc/ipsec.d/*.conf</pre><p>Create a new file at <strong>/etc/ipsec.d/aws.conf</strong> if it doesn’t already exist, and then open it. Append the following configuration to the end in the file:</p><p><em>Note — I have removed the “auth=esp” value from the configuration</em></p><pre>conn Tunnel1<br> authby=secret<br> auto=start<br> left=%defaultroute<br> leftid=34.229.15.228<br> right=52.10.50.36<br> type=tunnel<br> ikelifetime=8h<br> keylife=1h<br> phase2alg=aes128-sha1;modp1024<br> ike=aes128-sha1;modp1024<br> keyingtries=%forever<br> keyexchange=ike<br> leftsubnet=172.16.0.0/16<br> rightsubnet=10.0.0.0/16<br> dpddelay=10<br> dpdtimeout=30<br> dpdaction=restart_by_peer</pre><p>#leftsubnet= is the local network behind your openswan server, and you will need to replace the &lt;LOCAL NETWORK&gt; below with this value (don’t include the brackets).</p><p>#rightsubnet= is the remote network on the other side of your VPN tunnel that you wish to have connectivity with, and you will need to replace &lt;REMOTE NETWORK&gt; with this value (don’t include brackets).</p><p>Create a new file at <strong>/etc/ipsec.d/aws.secrets</strong> if it doesn’t already exist, and append this line to the file (be mindful of the spacing!):</p><pre>34.229.15.228 52.10.50.36: PSK “mg87mGPvxRsgvk_ulnTq6NV6BVmFWdv0”</pre><p>For this tutorial, we are using a pre-shared key approach.</p><p>Next, start the IP sec service on the instance and ensure it is running.</p><pre>systemctl start ipsec</pre><pre>systemctl status ipsec</pre><p>For troubleshooting, use</p><pre>journalctl -xe</pre><p><em>Quick summary: In this section, we have configured openswan with the connection to tunnel 1 only. For production use, add tunnel 2 information into the configuration file.</em></p><p><strong>US-West (Oregon) Region</strong></p><p><strong>Step 13: </strong>In the VPN connection section, you should see one of the Tunnel is UP and running.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z2bbKlnlHno_HaCmvRUHIA.png" /><figcaption>VPN connection Tunnel 1 status UP — US West (Oregon) region</figcaption></figure><p><strong>Step 14: </strong>If you have survived up until now, then you should be ready to test. Check if ping works through private IPs. SSH into EC2 private subnet instance. Ping the private IP of US-East (N Virginia) region EC2 instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*4EI3nu_FbrxKg25lsGmKVg.png" /><figcaption>Testing ping from private EC2 instance of US-West (Oregon) To US-East (N Virginia) region using private IPs.</figcaption></figure><p>Before we wrap up. Few facts about VPN connections.</p><ol><li>The customer VPN gateway will always establish the initial connection.</li><li>The bandwidth of VPN gateway is 1.25 Gbps</li><li>Although we have two tunnels, only one tunnel will send traffic to the customer network at any given time.</li><li>Here we have used static routes. For production setups, you would be using dynamic routing using BGP.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca468df079db" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>