<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Suman Pattnaik on Medium]]></title>
        <description><![CDATA[Stories by Suman Pattnaik on Medium]]></description>
        <link>https://medium.com/@sumaninvincible?source=rss-e7f58b457ebe------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*KZwoGmw8nyJIywhL.jpg</url>
            <title>Stories by Suman Pattnaik on Medium</title>
            <link>https://medium.com/@sumaninvincible?source=rss-e7f58b457ebe------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 02:11:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sumaninvincible/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Rendezvous with Kafka : A simple guide to get started]]></title>
            <link>https://medium.com/walmartglobaltech/rendezvous-with-kafka-a-simple-guide-to-get-started-48db3b921cc?source=rss-e7f58b457ebe------2</link>
            <guid isPermaLink="false">https://medium.com/p/48db3b921cc</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[messaging]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Suman Pattnaik]]></dc:creator>
            <pubDate>Sun, 17 Feb 2019 06:33:13 GMT</pubDate>
            <atom:updated>2019-03-27T05:12:33.109Z</atom:updated>
            <content:encoded><![CDATA[<h3>Rendezvous with Kafka : A simple guide to get started</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*o8BSFWJwGbn_1kE0BgQRuA.png" /></figure><p><strong>What is Kafka?</strong></p><p>The last decade saw Technology evolving in every field that humans could imagine. From nanotechnology to high speed Gaming leveraging GPUs, technology really has expanded and changed at a rate faster than ever. The most common attribute that grew with all the technological advancements was Data. There was some extremely huge growth in digital data that enterprises created. Far greater were the insights that enterprises expected to get out of the data, be it static or streaming. That posed a great challenge for companies which historically relied on Messaging Queues for transferring data from one system to another. For example data emanating from point of sales potentially could be used by finance , by accounts and also to create customer insights through analytics. If we have to do with queues, it would need multiple queues each containing the same exact data. The complexity continues to increase with other data points like order data, Website data etc. So there is also a greater challenge to maintain that infrastructure and ensure resiliency of the system and reliability and consistency of the data. The engineering Solution that came out to help with the situation and simplify the architecture is termed “Kafka”.</p><p>Kafka is a streaming Publish-subscribe platform that is capable of running in a distributed mode and handle millions of messages per minute. Here are its key capabilities :</p><ul><li>Publish and subscribe to streams of records, similar to enterprise messaging system.</li><li>Store streams of records in a fault-tolerant durable way for a pre-configured amount of time.</li><li>Process streams of records as they occur or after the fact.</li></ul><p><strong>Genesis of Kafka</strong></p><blockquote>Originally created at LinkedIn in 2010</blockquote><blockquote>LinkedIn’s installation of Kafka processes over 1.4 <em>trillion </em>messages per day</blockquote><blockquote>In use at many organizations (2000+)</blockquote><blockquote>example<em> : LinkedIn, Walmart, Netflix, Goldman Sachs etc</em></blockquote><p><strong>Why Kafka?</strong></p><p>· Publish once and Subscribe for many use-cases .</p><p>· Designed to support batch and real-time stream processing</p><p>· Performs extremely well at very large scale and is Horizontally scalable.</p><p>· Fault Tolerant</p><p>In the example that we discussed above, we could just have the sales data from point of sales streamed to a “sales” topic in Kafka and there could be three independent consumers reading the data separately for accounts, sales and customer insights respectively.</p><p>So you could move from a architecture like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/1*OwcyCZQCQ7B7TV45cjblmQ.png" /></figure><p>to something like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*edlmBwHiS0NUKgBMz8Sdxg.png" /></figure><p><strong>Use-Cases</strong></p><ul><li>Messaging System</li><li>Website Activity Tracking</li><li>Metrics and Log Aggregation</li><li>Stream Processing</li><li>Event Sourcing</li><li>Commit Log</li><li>Decouple system dependencies etc</li></ul><p>Kafka Simplified EcoSystem</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*EAHoZ36xeackeYjgcES2mw.png" /></figure><p>Kafka Extensive EcoSystem</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/1*lz9jbh6q_26yj83Amj1T9w.png" /></figure><p><strong>Topics and Partitions</strong></p><p><strong>Topics</strong></p><p>A <strong>Topic</strong> is a subject/feed name to which messages are stored and published. Messages are byte arrays that can store any object in any format. All Kafka messages are organized into topics. If one wishes to send a message he would send it to a specific topic and if one wishes to read a message he reads it from a specific topic. applications writing data to topics are called Producers and applications reading from topics are called consumers. Here are some key points to remember:</p><ul><li>Topics are identified by a unique “name”</li><li>Stream of records published to a Topic</li><li>Data in topics live in cluster until a preconfigured TTL(Time to Live) is reached.</li><li>Topic can have zero, one ore more subscribers (multi-subscriber)</li><li>You can have multiple topics in a broker</li></ul><p><strong>Partitions</strong></p><p>Kafka topics are divided into a number of partitions, which contains messages in an unchangeable ordered sequence(First in first out). Each message in a partition is assigned and identified by its unique location point called <strong>offset.</strong> A topic can also have multiple partition logs. This allows for multiple consumers to read from a topic in parallel and thus helps with Horizontal scaling. Here are some key points to remember:</p><ul><li>Each partition is <strong>ordered</strong></li><li>Records in the partitions are assigned a sequential id number called the <strong>offset</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*JfeZvnI5fIznjqeXBTesIQ.png" /></figure><ul><li>Messages within a partition are immutable</li><li>Offset are meaningful within a context of a specific partitions</li><li>Order of messages is guaranteed within a partition</li><li>Data is retained for configurable retention period</li><li>Records are distributed in round robin fashion unless a key is specified</li><li>Each partition contains the subset of the Topic’s messages</li><li>More partitions means more parallelism</li><li>Its important to chose the right number of partitions. Here is some guide to do so : <a href="https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/">https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/</a></li></ul><p>Brokers</p><p>A broker is a set of one or more servers running Kafka. Topics reside in brokers. Here are some key points to remember:</p><ul><li>Brokers receive and store messages when they are sent by the Producers</li><li>A Kafka cluster will typically have multiple Brokers</li><li>Each can handle hundreds of thousands, or millions, of messages per second</li><li>Each Broker is identified with its id (<em>broker.id</em>)</li><li>After connecting to one or more broker broker (<em>bootsrap.servers), </em>you will be connected to all the servers in cluster</li><li>A minimum of 3 or 5 brokers in a cluster is recommended for high availability and throughput</li><li>Partitions are distributed across the Kafka cluster</li><li>Each partition is replicated across a configurable number of servers for fault tolerance</li><li>Any given Partition is handled by a single Broker</li><li>Typically, a broker handles multiple partitions</li><li>Example — Two topics with 3 and 2 partitions</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/522/1*hhKTVWkG2SLJJoUFdt8tuQ.png" /></figure><p><strong>Replication Factor</strong></p><p>In order to facilitate resiliency and mitigate any data loss, Kafka like many other distributed technologies maintains copies of data on multiple servers.Replication factor is the number of copies that need to exist for the same data. A replication factor of 3 would mean that there are 3 copies with 2 ISR(In sync replicas). ISR is the number of redundant copies apart the one that exist on the leader.Here are some key points to remember:</p><ul><li>Every topic partition in Kafka is replicated <em>n</em> times, where <em>n</em> is the replication factor of the topic</li><li>RF should be greater than 1 and less or equal to number of brokers</li><li>Out of the <em>n</em> replicas, one replica is designated as the <em>leader</em> while others are <em>followers or ISR (In-Sync replicas)</em></li><li>Only leader handles all read and write requests for the partition</li><li>Example: Topic 1 with <strong>replication-factor 3</strong> and <strong>1 partition</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/522/1*9ux8FgXFCBNn1uP1qM2dZg.png" /></figure><ul><li>Example: Topic 2 with <strong>replication-factor 3</strong> and <strong>2 partitions</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/522/1*4MD3rFpyp-8tVGw1n27bIA.png" /></figure><p><strong>Kafka Producer</strong></p><p>A Kafka Producer is an application that sends messages to a Kafka topic.</p><p>Here are some key Kafka Producer configurations to take note of :</p><p>•<strong>Core Configuration</strong></p><blockquote><strong><em>bootstrap.servers :</em></strong><em> </em>A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping — this list only impacts the initial hosts used to discover the full set of servers. This list should be in the form <em>host1:9092,host2:9092,...</em>. Since these servers are just used for the initial connection to discover the full cluster membership (which may change dynamically), this list need not contain the full set of servers (you may want more than one, though, in case a server is down).</blockquote><blockquote><strong>key.serializer </strong>: Serializer class for key that implements the org.apache.kafka.common.serialization.Serializer interface.</blockquote><blockquote><strong>value.serializer</strong> : Serializer class for value that implements the org.apache.kafka.common.serialization.Serializer interface</blockquote><blockquote><strong><em>client.id : </em></strong>An id string to pass to the server when making requests. The purpose of this is to be able to track the source of requests beyond just ip/port by allowing a logical application name to be included in server-side request logging.</blockquote><p>•<strong>Message Durability</strong> ( <em>Please select ones based on the sensitivity to data loss and over all speed of processing </em>)</p><blockquote><strong><em>acks=0</em></strong> (no acknowledgement, fire and forget, possible data loss)</blockquote><blockquote><strong><em>acks=1</em></strong> (default, wait for leader’s acknowledgement, limited data loss)</blockquote><blockquote><strong><em>acks=2</em></strong> (wait for leader’s and one other replica’s acknowledgement , limited data loss. Behaves like quorum in various other distributed and replicated platforms)</blockquote><blockquote><strong><em>acks=all</em></strong> (wait for leader and all replicas, no data loss)</blockquote><p><strong>Note</strong> : <em>Ack setting is designed for trade-off between performance and the chance of losing data</em>. Its important to note that , the greater the Ack ( acks=all ), the slower the speed of processing as the client would wait on all replicas and the leader to send acknowledgment. Use this only when Loss of any one message is not tolerable and there is no other process to reprocess on failed messages. If set to zero then the producer will not wait for any acknowledgment from the server at all. The record will be immediately added to the socket buffer and considered sent. No guarantee can be made that the server has received the record in this case, and the retriesconfiguration will not take effect (as the client won&#39;t generally know of any failures). The offset given back for each record will always be set to -1.Preferred Ack is 2 for a prod application with a kafka cluster with ISR as 2. Non critical applications and applications where losing one message may trigger immediate resend of data ( chat message failure etc) , one may have acks=0 to gain on speed.</p><p>•<strong>Message Ordering</strong></p><blockquote>Messages are written to the broker in the same order that they are received by producer client</blockquote><blockquote><strong><em>retries</em></strong> &gt; 0 can cause reordering (default is 0)</blockquote><blockquote><strong><em>max.in.flight.requests.per.connection</em></strong><em>=1</em> with retries enabled: The maximum number of unacknowledged requests the client will send on a single connection before blocking. Note that if this setting is set to be greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (i.e., if retries are enabled).</blockquote><p>•<strong>Batching and Compression</strong></p><blockquote><strong><em>batch.size</em> </strong>— (size based batching) : The producer will attempt to batch records together into fewer requests whenever multiple records are being sent to the same partition. This helps performance on both the client and the server. This configuration controls the default batch size in bytes. The default size is 16384</blockquote><blockquote>No attempt will be made to batch records larger than this size.</blockquote><blockquote>Requests sent to brokers will contain multiple batches, one for each partition with data available to be sent.</blockquote><blockquote>A small batch size will make batching less common and may reduce throughput (a batch size of zero will disable batching entirely). A very large batch size may use memory a bit more wastefully as we will always allocate a buffer of the specified batch size in anticipation of additional records.When receiving compressed messages, 0.10.0 brokers avoid recompressing the messages, which in general reduces the latency and improves the throughput. In certain cases, however, this may reduce the batching size on the producer, which could lead to worse throughput. If this happens, users can tune linger.ms and batch.size of the producer for better throughput.</blockquote><blockquote><strong><em>linger.ms</em></strong> — (time based batching) : The producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they can be sent out. However in some circumstances the client may want to reduce the number of requests even under moderate load. This setting accomplishes this by adding a small amount of artificial delay — i.e, rather than immediately sending out a record the producer will wait for up to the given delay to allow other records to be sent so that the sends can be batched together. This setting gives the upper bound on the delay for batching: once we get batch.sizeworth of records for a partition it will be sent immediately regardless of this setting, however if we have fewer than this many bytes accumulated for this partition we will &#39;linger&#39; for the specified time waiting for more records to show up. This setting defaults to 0 (i.e. no delay). Setting linger.ms=5, for example, would have the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load.</blockquote><blockquote><strong><em>compression.type</em></strong> — larger batches mean higher compression. Specify the final compression type for a given topic. This configuration accepts the standard compression codecs (‘gzip’, ‘snappy’, ‘lz4’, ‘zstd’). It additionally accepts ‘uncompressed’ which is equivalent to no compression; and ‘producer’ which means retain the original compression codec set by the producer.</blockquote><p>•<strong>Queuing Limits</strong></p><blockquote><strong><em>buffer.memory</em></strong> — total memory that is available to the Java client for collecting <strong><em>unsent</em></strong> messages</blockquote><blockquote><strong><em>max.block.ms</em></strong> — block additional sends for this duration before raising exception</blockquote><blockquote><strong><em>request.timeout.ms</em></strong> — if message is queued for more than this duration then remove it from queue and raise exception</blockquote><p><strong>Kafka Consumer</strong></p><p>An application that reads data from a kafka Topic is called a kafka consumer. Here are some key points to remember.</p><ul><li>Consumer reads data from the topic</li><li>Consumer reads data in order within each partition</li><li>Consumer reads data in parallel across the partition</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*yMK2SwgIiFk2WvlEe2Ol3w.png" /></figure><p>Here are some key Kafka Consumer configurations to take note of :</p><p>•<strong>Core Configuration</strong></p><blockquote><strong><em>bootstrap.servers : </em></strong>A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping — this list only impacts the initial hosts used to discover the full set of servers. This list should be in the form host1:9092,host2:9092,.... Since these servers are just used for the initial connection to discover the full cluster membership (which may change dynamically), this list need not contain the full set of servers (you may want more than one, though, in case a server is down).</blockquote><blockquote><strong>key.deserializer : </strong>Deserializer class for key that implements the org.apache.kafka.common.serialization.Deserializerinterface. One could create custom deserializer based on the need.</blockquote><blockquote><strong>value.deserializer : </strong>Deserializer class for value that implements the org.apache.kafka.common.serialization.Deserializerinterface. One could create custom deserializer based on the need.</blockquote><blockquote><strong><em>client.id : </em></strong><em>This value is specified by the kafka consumer client and is used to distinguish between different clients. This is different from group.id</em></blockquote><p>•<strong>Group Configuration</strong></p><blockquote><strong><em>group.id : </em></strong>This property defines a unique identity for the set of consumers within the same consumer group.A new consumer with the same group Id would share message with the existing set of consumers.</blockquote><blockquote><strong><em>session.timeout.ms : </em></strong>The timeout used to detect consumer failures when using Kafka’s group management facility. The consumer sends periodic heartbeats to indicate its liveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker will remove this consumer from the group and initiate a rebalance. Note that the value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.msand group.max.session.timeout.ms.</blockquote><blockquote><strong><em>max.poll.records : </em></strong>The maximum number of records returned in a single call to poll().When not specified the default value is 500.</blockquote><blockquote><strong><em>heartbeat.interval.ms : </em></strong>The expected time between heartbeats to the group coordinator when using Kafka’s group management facilities. Heartbeats are used to ensure that the worker’s session stays active and to facilitate rebalancing when new members join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.</blockquote><p>•<strong>Offset Management</strong></p><blockquote><strong><em>enable.auto.commit : </em></strong>This allows for<strong><em> </em></strong>consumer’s offset to be periodically committed in the background if set true.</blockquote><blockquote><strong><em>auto.commit.interval.ms : </em></strong>he frequency in milliseconds that the consumer offsets are auto-committed to Kafka if enable.auto.commit is set to true. default value is 5000</blockquote><blockquote><strong><em>auto.offset.reset : </em></strong><em>This setting helps to identify the offset from where the consumer starts reading the data. Here are the options :</em></blockquote><ul><li>earliest: automatically reset the offset to the earliest offset</li><li>latest: automatically reset the offset to the latest offset</li><li>none: throw exception to the consumer if no previous offset is found for the consumer’s group</li><li>anything else: throw exception to the consumer.</li></ul><p><strong>Consumer group</strong></p><p>A <strong>consumer group</strong> is a set of consumers which cooperate to consume data from some topics</p><ul><li>Each consumer within a group is assigned a set of partitions to read from. For any given partition, only one consumer will read from it.</li><li>If you have more consumers than partitions, some consumers will be idle</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*JFYQCL5V-Mva34mweeY8ng.png" /></figure><ul><li>Consumer uses a group coordination protocol built into Kafka</li><li>One of the brokers is selected as the <strong><em>group coordinator</em></strong></li><li>Coordinator mediates partition assignment</li><li>Each member in the group must send heartbeats to the coordinator in order to remain a member of the group</li><li>Consumer must commit the offsets corresponding to the messages it has read</li><li>In case of consumer shutdown or crash, its partitions will be re-assigned to another member</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/1*1c-5KoTGBPbSQDReu8-KGQ.png" /></figure><p><strong>Kafka Log Store(Segments and Indexes)</strong></p><ul><li>Kafka’s storage unit is a partition</li><li>A partition cannot be split across multiple brokers or even multiple disks</li><li>Topics are split into partitions</li><li>Partitions are split into segments</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*iAMU3xmpiZ0nahBvP7Ovyw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*m5PFYmBIS5PQ-u_DfFJCYQ.png" /></figure><ul><li>Partitions are split into segments</li><li>Segments are named by their base offset</li><li>Only one segment is active</li><li>Segment settings</li></ul><blockquote><strong><em>log.segment.bytes</em></strong> — Maximum size of a single log file</blockquote><blockquote><strong><em>log.roll.hours</em></strong> — Maximum time kafka will wait for rolling a new segment file</blockquote><ul><li>Partition is a directory and each segment is consist of an index file , timeindex file and a log file</li></ul><p>What is a Zookeeper ?</p><ul><li>ZooKeeper is a distributed, open-source coordination service for distributed applications</li><li>Keeps the list of brokers</li><li>Helps in leader election for partitions</li><li>Sends notification to Kafka for changes like</li><li>New Topic</li><li>Broker Failure</li><li>Topic deletion</li><li>Kafka needs Zookeeper to work</li><li>3 to 5 servers works very well (odd quorum)</li><li>Only one server is leader and rest are followers</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*q-nVw6oZ7XK9gnDECtGkuA.png" /></figure><p><strong>Its Time to Make your hands dirty ( Setting up and Running Kafka Locally)</strong></p><p>You would need Zookeeper and Kafka started before we can run the local test.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*8oqlSMn6NB5EQaxHsJ4ABQ.png" /></figure><ul><li>Download the latest version of Confluent Kafka and follow the instructions from the confluent website : <a href="https://docs.confluent.io/current/quickstart/ce-quickstart.html#ce-quickstart">https://docs.confluent.io/current/quickstart/ce-quickstart.html#ce-quickstart</a></li><li>Start Zookeeper :</li></ul><blockquote>bin/zookeeper-server-start etc/kafka/zookeeper.properties</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8KU3oFmEj9jMTjwaLTbPpw.png" /></figure><p>2019–02–16 23:37:10,683] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory)</p><p>[2019–02–16 23:37:10,709] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)</p><ul><li>Start Kafka</li></ul><p>bin/kafka-server-start etc/kafka/server.properties</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x02zf7CbTKH0XztLy2Ji_A.png" /></figure><blockquote>[2019–02–16 23:40:44,452] INFO Cluster ID: cqdE3HyPTQ-O6uGW9hKwVQ (org.apache.kafka.clients.Metadata)</blockquote><blockquote>[2019–02–16 23:40:44,557] INFO [Log partition=__confluent.support.metrics-0, dir=/tmp/kafka-logs] Loading producer state till offset 0 with message format version 2 (kafka.log.Log)</blockquote><blockquote>[2019–02–16 23:40:44,558] WARN [Producer clientId=producer-1] Error while fetching metadata with correlation id 3 : {__confluent.support.metrics=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)</blockquote><blockquote>[2019–02–16 23:40:44,575] INFO [Log partition=__confluent.support.metrics-0, dir=/tmp/kafka-logs] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 141 ms (kafka.log.Log)</blockquote><p>Log message when a Topic is created</p><blockquote>[2019–02–16 23:40:44,580] INFO Created log for partition __confluent.support.metrics-0 in /tmp/kafka-logs with properties {compression.type -&gt; producer, message.format.version -&gt; 2.1-IV2, file.delete.delay.ms -&gt; 60000, max.message.bytes -&gt; 1000012, min.compaction.lag.ms -&gt; 0, message.timestamp.type -&gt; CreateTime, message.downconversion.enable -&gt; true, min.insync.replicas -&gt; 1, segment.jitter.ms -&gt; 0, preallocate -&gt; false, min.cleanable.dirty.ratio -&gt; 0.5, index.interval.bytes -&gt; 4096, unclean.leader.election.enable -&gt; false, retention.bytes -&gt; -1, delete.retention.ms -&gt; 86400000, cleanup.policy -&gt; [delete], flush.ms -&gt; 9223372036854775807, segment.ms -&gt; 604800000, segment.bytes -&gt; 1073741824, retention.ms -&gt; 31536000000, message.timestamp.difference.max.ms -&gt; 9223372036854775807, segment.index.bytes -&gt; 10485760, flush.messages -&gt; 9223372036854775807}. (kafka.log.LogManager)</blockquote><blockquote>[2019–02–16 23:40:44,584] INFO [Partition __confluent.support.metrics-0 broker=0] No checkpointed highwatermark is found for partition __confluent.support.metrics-0 (kafka.cluster.Partition)</blockquote><blockquote>[2019–02–16 23:40:44,597] INFO Replica loaded for partition __confluent.support.metrics-0 with initial high watermark 0 (kafka.cluster.Replica)</blockquote><blockquote>[2019–02–16 23:40:44,608] INFO [Partition __confluent.support.metrics-0 broker=0] __confluent.support.metrics-0 starts at Leader Epoch 0 from offset 0. Previous Leader Epoch was: -1 (kafka.cluster.Partition)</blockquote><blockquote>[2019–02–16 23:40:44,679] WARN [Producer clientId=producer-1] Error while fetching metadata with correlation id 4 : {__confluent.support.metrics=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)</blockquote><blockquote>[2019–02–16 23:40:44,993] INFO [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. (org.apache.kafka.clients.producer.KafkaProducer)</blockquote><blockquote>[2019–02–16 23:40:45,003] INFO Successfully submitted metrics to Kafka topic __confluent.support.metrics (io.confluent.support.metrics.submitters.KafkaSubmitter)</blockquote><blockquote>[2019–02–16 23:40:47,185] INFO Successfully submitted metrics to Confluent via secure endpoint (io.confluent.support.metrics.submitters.ConfluentSubmitter)</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iA1jmejeDRsOxAw3PRQ4Zw.png" /></figure><p>Log folder when no topic is created</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hI-LRbqLlAXaURncSeqChg.png" /></figure><ul><li><strong>Create Topic</strong></li></ul><p>bin/kafka-topics — create — zookeeper localhost:2181 — replication-factor 1 — partitions 1 — topic my-test</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-b-TjrTQsg826mowGD6u7w.png" /></figure><p>Log folder when topic is created</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9QdhWs-bOSNoHExghpUQEg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9QdhWs-bOSNoHExghpUQEg.png" /></figure><p>The necessary log files are created within a folder named after the partition number of the topic</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*ZkMPNiMt-Nm6vdOUjc9i9A.png" /></figure><p>Messages are stored in segment log file</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QCDppGJ7dgDpsWm1F-haJg.png" /></figure><p><strong>Sending a message :</strong></p><p>bin/kafka-console-producer — broker-list localhost:9092 — topic my-test</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GGAalsNyivmBx-2nYF8X7g.png" /></figure><p><strong>Listening and reading message from the topic :</strong></p><p>bin/kafka-console-consumer — bootstrap-server localhost:9092 — topic my-test — from-beginning</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kHe6WFXi-SQufzUfGgyAlg.png" /></figure><p>Alternatively, topic creation and sample message generation, publish and listening can be done through confluent Control center.</p><p><strong>Conclusion</strong></p><p>Kafka is the most definitive and widely used Messaging system leveraged across various industries. This effort of mine is to simplify the process to get started with Kafka. In coming days, effort would be made to publish simplified articles to get started with Kafka Connect and Streams.</p><p><strong>References</strong></p><ul><li><a href="https://docs.confluent.io/current/quickstart/ce-quickstart.html#ce-quickstart">Confluent Platform Quick Start (Local) - Confluent Platform</a></li><li><a href="https://dzone.com/articles/getting-started-with-kafka-tutorial">Getting Started With Kafka - DZone Big Data</a></li><li><a href="https://kafka.apache.org/quickstart">Apache Kafka</a></li><li><a href="https://www.sohamkamani.com/blog/2017/11/22/how-to-install-and-run-kafka/">How to install and run Kafka on your machine 🌪</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=48db3b921cc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/walmartglobaltech/rendezvous-with-kafka-a-simple-guide-to-get-started-48db3b921cc">Rendezvous with Kafka : A simple guide to get started</a> was originally published in <a href="https://medium.com/walmartglobaltech">Walmart Global Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[STORAGES : A Beginners’ guide]]></title>
            <link>https://medium.com/@sumaninvincible/storages-a-beginners-guide-7256bfb163a7?source=rss-e7f58b457ebe------2</link>
            <guid isPermaLink="false">https://medium.com/p/7256bfb163a7</guid>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Suman Pattnaik]]></dc:creator>
            <pubDate>Tue, 27 Nov 2018 12:48:55 GMT</pubDate>
            <atom:updated>2018-11-27T12:48:55.773Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>STORAGES</strong> : A Beginners’ guide</p><p><strong>What is Storage :</strong></p><p>Storage is a device, server or part of the server dedicated to store digital data in persistent manner.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/0*ozT0Q9A5k-v8lSKQ" /></figure><p><strong>Broadly storage is of two types:</strong></p><p><strong>Ephemeral Storage:</strong> It is used by the cluster to store information that does not need to persist. This aids in optimization and helps to reduce the load on the persistent storage.Usually high throughput but life of data is dependent upon the life of the JVM or the server. Data is lost if the server is shut down.</p><p><strong>Non Ephemeral Storage :</strong> Other wise called persistent storage. it is used by the cluster to store information that needs to persist even if a shard or a node fails. For example, if you choose any type of persistence for a database, then the persistence information is stored in this location.</p><p>Ephemeral Storage is what we more than often refer as storage. It is broadly of thress types:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/0*mr6yXR2O4usSaTGK" /></figure><p>Networked storage services are generally classified in the way they are consumed and interfaced with on the client side. The most traditional service type is shared filesystem, or simply “file storage”, which as the name implies offers to multiple clients the ability to access a single shared folder. The two most popular shared filesystem protocols in use today are NFS and SMB/CIFS.</p><p><strong>Block storage</strong> provides network access to the equivalent of raw block devices. A client machine connects to a specific volume on the storage service and formats it as if it were a local block device. Multiple clients do not generally mount the same volume, but they may in master/slave high-availability configurations where the slave needs to be ready to take over the master. Block devices are usually exported over Fibre Channel, iSCSI or AoE (ATA over Ethernet). With <em>block storage</em>, files are split into evenly sized blocks of data, each with its own address but with no additional information (metadata) to provide more context for what that block of data is. You’re likely to encounter block storage in the majority of enterprise workloads; it has a wide variety of uses (as seen by the rise in popularity of SAN arrays).</p><p>Good examples for block storage use cases are structured database storage, random read/write loads, and virtual machine file system (VMFS) volumes. However, since block storage has essentially no additional storage-side metadata that can be associated with a given block other than the address of that block, performance degrades in geographically distributed systems. The further the block storage gets from the application, the more the performance suffers due to latency issues. Transactional data is another good usecase for block storage.</p><p><strong>Object storage</strong> is a relatively new storage type, designed for unstructured data such as media, documents, logs, backups, application binaries and VM images. Conceptually they are like a persistent key/value store; It has three parts : An Id to retrieve the object, a metadata and the the actual data. Each of the objects is immutable.Objects have to be manipulated as a whole unit, requiring the entire object to be accessed, updated, then re-written in their entirety. That can have performance implications.objects are usually submitted via a REST API call, and an identifier returned. Most object stores allow attaching metadata to objects, and aggregating them into containers (or buckets). Both Ceph and Swift offer object store interfaces. The most popular cloud object store is AWS S3, and many object store implementations are compatible with it, including Ceph’s S3 RADOS Gateway service.</p><p>Object storage is ideal for solving the <a href="https://www.druva.com/blog/druva-mirroring-growth-mobile-data-infographic/">increasing problems of data growth</a>. As more and more data is generated, storage systems have to grow at the same pace. What happens when you try to expand a block-based storage system beyond a hundred terabytes or beyond multiple petabytes? You may run into durability issues, hard limitations with the storage infrastructure that you currently have, or your management overhead may go through the roof.Object-based storage architectures can be scaled out and managed simply by adding additional nodes.</p><p>Another advantage to object storage its responsiveness to <a href="https://www.druva.com/blog/cloud-backup-solution-provide-full-data-visibility/">the need for resiliency </a>while mitigating costs. Objects remain protected by storing multiple copies of data over a distributed system; if one or more nodes fail, the data can still be made available, in most cases, without the application or the end user ever being impacted. (Downtime? <em>What</em> downtime?) In most cases, at least three copies of every file are stored.</p><p>Object storage works very well for unstructured data sets where data is generally read but not written-to. Static Web content, data backups and archival images, and multimedia (videos, pictures, or music) files are best stored as objects. Databases in an object storage environment ideally have data sets that are unstructured, where the use cases suggests the data will not require a large number of writes or incremental updates.</p><p>Geographically distributed back-end storage is another great use case for object storage. The object storages applications present as network storage and support extendable metadata for efficient distribution and parallel access to objects. That makes it ideal for moving your back-end storage clusters across multiple data centers.</p><p><strong>Other important Terms :</strong></p><p><strong>Solid State Drive (SSD)</strong></p><p>Solid State Drives look and function similar to traditional mechanical/ magnetic hard drives but the similarities stop there. Internally, they are completely different. They have no moving parts or rotating platers. They rely solely on semiconductors and electronics for data storage making it a more reliable and robust than traditional magnetic. No moving parts also means that they use less power than traditional hard drives and are much faster too.</p><p>With the prices of Solid State Drives coming down and is lower power usage, SSD’s are used extensively on laptops and mobile devices. External SSD’s are also a viable option for data backups.</p><p><strong>Advantages :</strong></p><ul><li>Faster read and write performance</li><li>More robust and reliable than traditional magnetic hard drives</li><li>Highly portable. Can be easily taken offsite</li></ul><p><strong>Disadvantages :</strong></p><ul><li>Still relatively expensive when compared to traditional hard drives</li><li>Storage space is typically less than that of traditional magnetic hard drives.</li></ul><p><strong>Raid(Redundant Array of Independent Disk</strong></p><p>It is a term used to describe several methods for using multiple hard drives in an array to boost performance and provide fault tolerance by ensuring that the failure of one drive does not cause the loss of data. Some RAID configurations allow two or more drives to fail before data is lost. A typical RAID array contains multiple terabytes of storage along with cache memory.</p><p>RAID levels refer to the various configurations of the technology that provide fault tolerance using different methods, including mirroring two drives to each other or striping data across multiple drives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/0*FvaKl1Y4XGP7OJ9s" /></figure><p><strong>Important Architecture references :</strong></p><p><strong>Cloud Storage (Open stack)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/0*Szl061eeXeptfoZt" /></figure><p>Cloud Storage High level Architecture</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/0*LFfA_zbFYTRUUqdr" /></figure><p><strong>References :</strong></p><p><a href="https://www.druva.com/blog/object-storage-versus-block-storage-understanding-technology-differences/">https://www.druva.com/blog/object-storage-versus-block-storage-understanding-technology-differences/</a></p><p><a href="https://callnerds.com/whats-difference-memory-storage/">https://callnerds.com/whats-difference-memory-storage/</a></p><p><a href="https://redislabs.com/redis-enterprise-documentation/cluster-administration/best-practices/persistent-and-ephemeral-storage/">https://redislabs.com/redis-enterprise-documentation/cluster-administration/best-practices/persistent-and-ephemeral-storage/</a></p><p><a href="http://www.crn.com/news?term=raid&amp;x=0&amp;y=0">http://www.crn.com/news?term=raid&amp;x=0&amp;y=0</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7256bfb163a7" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>