<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nimesh Khandelwal on Medium]]></title>
        <description><![CDATA[Stories by Nimesh Khandelwal on Medium]]></description>
        <link>https://medium.com/@nimeshkhandelwa?source=rss-163f6694997e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*SdVxXo8C_3AFRRZJWO309w.jpeg</url>
            <title>Stories by Nimesh Khandelwal on Medium</title>
            <link>https://medium.com/@nimeshkhandelwa?source=rss-163f6694997e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 15:57:58 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nimeshkhandelwa/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Understanding Kafka — A Distributed Streaming Platform]]></title>
            <link>https://medium.com/swlh/understanding-kafka-a-distributed-streaming-platform-9a0360b99de8?source=rss-163f6694997e------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a0360b99de8</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Nimesh Khandelwal]]></dc:creator>
            <pubDate>Tue, 02 Jul 2019 19:09:38 GMT</pubDate>
            <atom:updated>2019-07-04T22:09:21.958Z</atom:updated>
            <content:encoded><![CDATA[<h3>Understanding Kafka — A Distributed Streaming Platform</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kQXkMQTrMrG4VJ3KZehaqA.png" /></figure><p>Overview of Industry most popular streaming platform</p><h3>Introduction</h3><p>Kafka is a new buzzword in the industry nowadays. Almost every leading tech company is using Kafka on its platform. But the question is what it is and why so much hype?</p><p>Kafka is a distributed, horizontally scalable streaming platform. It is an open-source stream processing platform. Kafka originated at LinkedIn and later became an open-source Apache project in 2011, then a first-class Apache project in 2012. <em>Kafka is written in Scala and Java</em>. It aims at providing a high-throughput, low-latency platform for handling real-time data feeds.</p><h3>How Kafka works</h3><p>Kafka is based on the pub/sub model. It’s similar to any messaging system. Applications (<strong>producers</strong>) send messages (<strong>records</strong>) to a Kafka node (<strong>broker</strong>) and said messages are processed by other applications called <strong>consumers</strong>. Messages get stored in a <strong>topic </strong>and consumers can subscribe to the topic and listen to those messages. Messages can be anything like sensors measurements, meter readings, user actions, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/815/1*-0Jm_e9KV1KJribxEQW80Q.png" /><figcaption>Kafka Producer and Consumer Model</figcaption></figure><p>A <strong>Topic</strong> can be thought of as a category/feed name to which messages are stored and published.</p><h3>Partitions</h3><p>Kafka topics can be very big in size so it might not be possible to store all data of a topic in a single node, so data need to be divided into multiple partitions. <em>Partitions allow us to parallelize a topic by splitting the data in a particular topic across </em><strong><em>multiple brokers(Kafka nodes)</em></strong> i.e each partition can be placed on a separate machine to allow for multiple consumers to read from a topic in parallel. Partitioning of a topic can be based on anything, like on order they came in, hashing, id, etc.</p><p>To increase the availability of partitions, each partition also has replicas. To understand it better let’s have a Kafka cluster consisting of 3 nodes/broker.</p><p>Now a topic is split into 3 partitions and each broker has a copy of each partition. Among these copy of partition, one is elected as leader, while others just passively make themselves in sync with Leader.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j6iSTOjq094Hc--0Y5SjPA.png" /><figcaption>There are 3 partitions of a topic named as Partition 0, Partition 1 and Partition 2</figcaption></figure><p>All writes and reads to a topic go through the respective partition leader and the leader coordinates updating replicas with new data. If a leader fails, a replica takes over as the new leader.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FG3H9BD7hHVSQ8F5vpD62g.png" /><figcaption>Write is only through leader and replicas get updated asynchronously.</figcaption></figure><p>For a producer/consumer to write/read from a partition, they need to know its leader, right? This information needs to be available from somewhere.<br><em>Kafka stores such metadata in a service called </em><a href="https://zookeeper.apache.org/"><em>Zookeeper</em></a><em>.</em></p><h3>Log Anatomy</h3><p><strong>The key to Kafka performance and scalability is log</strong>. Developers often get confused when first hearing about this “log,” because we’re used to understanding “logs” in terms of application logs. What we’re talking about here, however, is the log data structure. <em>Log is a persistent ordered data structure which only supports appends.</em> You cannot modify nor delete records from it. It is read from left to right and guarantees item ordering.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*nzyZl3p_eZi5xBjQ0AaOOw.png" /><figcaption>Log structure</figcaption></figure><p>A data source writes messages to the log and one or more consumers reads from the log at the point in time they choose.</p><p><strong>Each entry in this log is uniquely identified by offset</strong>, which is nothing but a sequential index number just like an array.</p><p>As this sequence/offset can only be maintained on a particular node/broker and can’t be maintained in the whole cluster, thus <em>Kafka only guarantees to order of data per partition.</em> If your use-case requires complete ordering of data, then you can define the partition key accordingly.</p><h3>Persistence of Data in Kafka</h3><p>Kafka actually stores all of its messages to disk and not in RAM and having them ordered in the log structure lets it take advantage of sequential disk reads and writes.</p><p>It&#39;s a common how can it be a good choice to store data on a hard disk and how is it still performant, well there are numerous reasons behind it</p><ol><li>Kafka has a protocol that groups messages together. This allows network requests to group messages together and reduces network overhead, the server, in turn, persist chunk of messages in one go and consumer fetch large linear chunks at once, thus reducing disk operations</li><li>Kafka relies heavily on OS pagecache for data storage i.e using free RAM on machine effectively.</li><li>Kafka stores messages in a standardized binary format unmodified throughout the whole flow (producer-&gt;broker-&gt;consumer), it can make use of the zero-copy optimization. That is when the OS copies data from the pagecache directly to a socket, effectively bypassing the Kafka broker application entirely.</li><li>Linear reads/writes on a disk are fast. The concept that modern disks are slow is because of numerous disk seeks. Kafka does linear read and writes, thus making it performant.</li></ol><h3>Consumer and Consumer Groups</h3><p>Consumers read from any single partition, allowing you to scale throughput of message consumption in a similar fashion to message production. <em>Consumers can also be organized into consumer groups for a given topic — each consumer within the group reads from a unique partition to avoid two processes reading the same message twice</em> <em>and the group as a whole consumes all messages from the entire topic</em>. If you have <strong>consumers &gt; partitions then some consumers will be idle</strong> because they have no partitions to read from. If you have <strong>partitions &gt; consumers then consumers will receive messages from multiple partitions</strong>. If you have <strong>consumers = partitions, each consumer reads messages in order from exactly one partition</strong>.</p><p>It can further be elaborated through an image from <a href="https://kafka.apache.org/intro">Kafka documentation</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EJ5ATktsMQFqP4hZj1mF1w.png" /></figure><p>Server 1 holds partitions 0 and 3 and server 2 holds partitions 1 and 2. We have two consumer groups, A and B. A is made up of two consumers and B is made up of four consumers. Consumer Group A has two consumers so each consumer reads from two partitions. Consumer Group B, on the other hand, has the same number of consumers as partitions so each consumer reads from exactly one partition.</p><p>Kafka follows the principle of a dumb broker and smart consumer. This means that Kafka does not keep track of what records are read by the consumer and thus unaware of consumer behavior and retains messages for a configurable period of time and it is up to the consumers to adjust their behavior accordingly. Consumers themselves poll Kafka for new messages and say what records they want to read. This allows them to increment/decrement the offset they’re at as they wish, thus being able to replay and reprocess events in case of an accident.</p><p>For instance, if Kafka is configured to keep messages for a day and a consumer is down for a period of longer than a day, the consumer will lose messages. However, if the consumer is down for an hour it can begin to read messages again starting from its last known offset.</p><h3>Role of Zookeeper</h3><p><a href="https://zookeeper.apache.org/">Zookeeper</a> is a distributed key-value store. It is highly-optimized for reads but writes are slower. Kafka uses Zookeeper to do the leadership election of Kafka Broker and Topic Partition. Zookeeper is also extremely fault-tolerant and it ought to be, as Kafka heavily depends on it.</p><p>It is used for storing all sort of metadata, to mention some:</p><ul><li>The consumer group‘s offset per partition (although modern clients store offsets in a separate Kafka topic)</li><li>ACL (Access Control Lists) — used for limiting access/authorization</li><li>Producer &amp; Consumer Quotas — maximum message/sec boundaries</li><li>Partition Leaders and their health</li></ul><p>Producer and consumers don’t directly interact with Zookeeper to know about the leader of a partition and other metadata, instead, they do metadata requests to Kafka Broker which in turn interact with Zookeeper and give metadata response and then producer and consumer get informed to which partition they need to write and read form respectively.</p><h3>Conclusion</h3><p>Kafka is quickly becoming the backbone of any organization’s data pipelines — and with good reason. Kafka allows you to have a huge amount of messages go through a centralized medium and store them without worrying about things like performance or data loss.</p><p>Kafka can be the centerpiece of event-driven architecture and allows you to truly decouple applications from one another.</p><h3>Further Readings for deep dive</h3><ol><li><a href="https://kafka.apache.org/documentation/">Kafka Documentation</a> — A great and detailed guide for Kafka</li><li><a href="https://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple/">Kafka Streams made simple</a> — A brief introduction for Kafka stream API</li><li><a href="https://docs.confluent.io/current/connect/index.html">Kafka Connect </a>— To learn further about Kafka Connect API</li></ol><p>Thank you for taking the time to read this.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a0360b99de8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/understanding-kafka-a-distributed-streaming-platform-9a0360b99de8">Understanding Kafka — A Distributed Streaming Platform</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Distributed Systems — A Many-Faced God -]]></title>
            <link>https://medium.com/analytics-vidhya/many-face-god-distributed-system-f05737ec978d?source=rss-163f6694997e------2</link>
            <guid isPermaLink="false">https://medium.com/p/f05737ec978d</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[computer-science]]></category>
            <dc:creator><![CDATA[Nimesh Khandelwal]]></dc:creator>
            <pubDate>Sat, 08 Jun 2019 11:40:50 GMT</pubDate>
            <atom:updated>2019-07-12T06:02:51.759Z</atom:updated>
            <content:encoded><![CDATA[<h3>Distributed Systems — A Many-Faced God</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4Arifbkft2mLD1hRlJ6ozw.jpeg" /></figure><h4>A brief introduction to the realm of Distributed Systems</h4><p>With the ever-growing technological expansion, distributed systems are more widely used and are worshipped by all big tech companies. Most of the applications we interact with nowadays are actually served by the system of computing elements acting like one.<br>Various questions come in one’s mind while discussing distributed systems, like what are distributed systems, why use them and much more.<br>To understand the distributed system we need to first know what was before that and why was there any need for systems like that.</p><p><strong>The time before distributed systems — the lone wolf</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/243/1*ms9Wg_5AHt5F4U3OZPZZBw.gif" /><figcaption>Multiple Client Hitting one server</figcaption></figure><p>Before the distributed system came into the picture, there used to be a single system that did <em>not</em> communicate with others and functioned on its own. It meant that when you requested a resource from the server, it would always reach a particular node/system to serve that request.</p><p>It can be easily understood that such a system won’t survive as when many requests start coming to the server, its CPU usage will increase and the system will crash. There is a famous saying — <strong>When the snows fall and the white winds blow, the lone wolf dies but the pack survives.</strong></p><p>So people start realizing that a single system can’t be used to serve all requests coming to their servers, they need something that will scale to cater to their load on servers.</p><p>But a genuine question might come to one’s mind, that if your CPU Load is increasing or RAM is becoming too low , why not work towards increasing capacity of our machine (commonly known as vertical scaling) and take load of managing a distributed system(which is a lot tricky and comes with its own problems)?</p><p><strong>Why prefer horizontal scaling(distributed way) over Vertical scaling(upgrading system)</strong></p><p>It&#39;s true that managing distributed systems are a complex topic chock-full of pitfalls and landmines. It is a headache to deploy, maintain, synchronize and debug distributed systems, so why go there at all?</p><p><strong>Scaling vertically</strong> is good while you can, yet after a specific point, you will see that even the best hardware isn’t adequate for enough traffic, not to mention impractical to host.</p><p><strong>Scaling horizontally</strong> simply means adding more computers rather than upgrading the hardware of a single one, therefore, <strong>it theoretically has an infinite potential to scale.</strong></p><p><strong>Even cost is a great factor why systems can’t scale vertically after a certain threshold.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*aN3mffhsckWeLLy4r3hOOQ.png" /><figcaption>Vertical scaling become lot expensive after a certain point</figcaption></figure><p><strong>Fault Tolerance- </strong>A good reason why a distributed system is more preferable. A cluster of 10 nodes is always more fault tolerant than a single machine of any configuration.</p><p><strong>Low Latency- </strong>With systems like <a href="https://www.cloudflare.com/learning/cdn/what-is-a-cdn/">CDN</a>, content get served from your nearest node, thus reducing time to get resource from servers.</p><p><strong>How Distributed System Works</strong></p><p>In the current world, almost all big companies are running their services through distributed systems. If you’ve ever played a multiplayer game online, booked a cab, posted on Facebook, streamed a Netflix show, or bought a book on Amazon — you relied on a distributed system to do it.</p><p>A <strong><em>distributed system</em></strong> is nothing more than multiple nodes that talk to one another in some way, while also performing their operations. A node can be anything ranging from a full-fledged system to a small sensor device as long it is autonomous and has a way to communicate with other nodes.</p><p>For a distributed system to work, though, you need the software running on those nodes to be specifically designed for running on multiple computers at the same time and handling the problems that come along with it. This turns out to be no easy feat. We will talk about these softwares in detail in other upcoming posts.</p><p>Let&#39;s understand the distributed system functioning through an example. Consider a website that uses a database system that currently consists of one node whose capacity to handle a request is X. Now load on your website double-fold, which in turn increases database load and our current single system can’t handle it. Our application would immediately start to decline in performance and this would get noticed by our users.</p><p>Now let&#39;s solve this problem through distributed systems. Initially, it might seem intuitive that, all we need to do is just add a node in our cluster and boom requests get handled by both servers and problem solved. But as easy it might seem, it&#39;s tricky and needs to handle various cases.</p><p>In a typical web application, normally reading information is much more frequent than insert or modify an old one.</p><p>There is a way in distributed systems to increase read performance and that is the so-called <strong>Master-Slave </strong>strategy. Here, you create two new database servers that sync up with the main one. These new slaves will <strong>only read</strong> from these new instances.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*dPcuDvPEkDU4nEWZDSG45A.png" /><figcaption>Write only on master and read from all</figcaption></figure><p>So now whenever new information is inserted, the request will be always redirected to a master database and master will asynchronously sync up with all slaves to update data.</p><p>Congratulations, now we can execute 3x as much read queries! Isn’t that great?</p><p><strong>But wait, here is where you enter a distributed system pitfall</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*CFbbvK2oUQFFusKgRKnrbA.jpeg" /></figure><p>We immediately lost the <strong><em>C</em></strong> in our relational database’s <strong>ACID</strong> guarantees, <strong>which stands for Consistency</strong>.</p><p>When we insert a record in the master database, there is a possibility that during a time when slave syncs up, someone request read from that slave and will be informed with stale data. This is a trade-off you take while improving your performance. You can solve this problem by delaying your write operation and make users wait to entertain any request till the databases syncs up, but as stated it will impact performance. It&#39;s quite common in a distributed system to work along with such trade-offs. There is a famous theorem which talks about trade-off in distributed systems called the<strong> CAP theorem</strong>. I will try to cover it in detail in subsequent posts.</p><p>Now we have scaled our system for reads, but if we want to scale it for write operations, we can do so by using multiple masters, but that will surely create a lot of problems such as creating conflicts(e.g insert two records with the same ID).</p><p>As you might have noticed that even though in the background it seems like there are multiple systems handling client requests, but to a client, it will always seem to be handled by one and that&#39;s the beauty of distributed systems.</p><p>This is just one application of Distributed Systems, there are a lot more other ways of applying distributed systems in the real world like HDFS, Queue Systems, Map Reduce, etc. You should read about these to get more clarity on how these works.</p><p>By now, you should have a very clear idea of how a simple distributed system looks like, and why to go on with it.</p><p>I know this article is quite short, but there is a lot into the world of a distributed system to cover in just one article, thus will try to accommodate further in subsequent articles. Don’t forget to follow to get updates on further articles.</p><p>Valar Morghulis !!</p><p>Thank you for reading the article and please provide your feedback and comments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f05737ec978d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/many-face-god-distributed-system-f05737ec978d">Distributed Systems — A Many-Faced God -</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>