<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tharindu Hirantha on Medium]]></title>
        <description><![CDATA[Stories by Tharindu Hirantha on Medium]]></description>
        <link>https://medium.com/@tharindu-hirantha?source=rss-d7b5f3e4c319------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ku_ThpwvCCYScfb1RZPPkA.jpeg</url>
            <title>Stories by Tharindu Hirantha on Medium</title>
            <link>https://medium.com/@tharindu-hirantha?source=rss-d7b5f3e4c319------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 23:20:56 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tharindu-hirantha/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Apache Kafka CLI Commands]]></title>
            <link>https://tharindu-hirantha.medium.com/apache-kafka-cli-commands-744207420d2?source=rss-d7b5f3e4c319------2</link>
            <guid isPermaLink="false">https://medium.com/p/744207420d2</guid>
            <category><![CDATA[cli]]></category>
            <category><![CDATA[basics]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[commands]]></category>
            <category><![CDATA[apache-kafka]]></category>
            <dc:creator><![CDATA[Tharindu Hirantha]]></dc:creator>
            <pubDate>Thu, 12 Aug 2021 13:53:28 GMT</pubDate>
            <atom:updated>2021-08-12T13:53:28.058Z</atom:updated>
            <content:encoded><![CDATA[<p>Basic Commands</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vTs2HMcNb0E49p5L8ST4GQ.png" /></figure><h4>Topic Creation</h4><blockquote>bin/kafka-topics.sh — zookeeper 127.0.0.1:2181 — topic first_topic — create — partitions 3 — replication-factor 3</blockquote><p><em>Arguments</em></p><ul><li>zookeeper: Zookeeper manages all the metadata.</li><li>topic: Topic Name</li><li>partitions: Number of Partitions</li><li>replication-factor: Number of replications</li></ul><h4>List All The Topics</h4><blockquote>bin/kafka-topics.sh — zookeeper 127.0.0.1:2181 — list</blockquote><p><em>Arguments</em></p><ul><li>zookeeper: Zookeeper manages all the metadata.</li></ul><h4>Describe a Topic</h4><blockquote>bin/kafka-topics.sh ––zookeeper 127.0.0.1:2181 ––topic first_topic ––describe</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pqDX3cOPYHhx-Ivgd2vKEw.png" /></figure><p>This command will show the partition count and the Leader broker. And Replica and ISR of each partition.</p><h4>Topic Delete</h4><blockquote>bin/kafka-topics.sh ––zookeeper 127.0.0.1:2181 ––topic second_topic ––delete</blockquote><h4>Produce Messages</h4><blockquote>bin/kafka-console-producer.sh ––broker-list 127.0.0.1:9092 ––topic first_topic</blockquote><h4>Produce Messages with Keys</h4><blockquote>kafka-console-producer — broker-list 127.0.0.1:9092 — topic first_topic — property parse.key=true — property key.separator=,</blockquote><h4>Consume Messages</h4><blockquote>bin/kafka-console-consumer.sh ––bootstrap-server 127.0.0.1:9092 ––topic first_topic</blockquote><h4>Consume Messages From the Beginning</h4><blockquote>bin/kafka-console-consumer.sh ––bootstrap-server 127.0.0.1:9092 ––topic first_topic ––from-beginning</blockquote><h4>Consume Messages as a Group</h4><blockquote>kafka-console-consumer.sh — bootstrap-server 127.0.0.1:9092 — topic first_topic — group app1</blockquote><h4>List all the Consumer Groups</h4><blockquote>kafka-consumer-groups.sh — bootstrap-server 127.0.0.1:9092 — list</blockquote><h4>Reset offset</h4><blockquote>kafka-consumer-groups.sh — bootstrap-server 127.0.0.1:9092 — group app1 — reset-offsets — to-earliest — execute — topic first_topic</blockquote><h4>Shift by</h4><blockquote>kafka-consumer-groups.sh — bootstrap-server 127.0.0.1:9092 — group app1 — reset-offsets — shift-by -2 — execute — topic first_topic</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=744207420d2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Kafka]]></title>
            <link>https://medium.com/geekculture/apache-kafka-a352140977ac?source=rss-d7b5f3e4c319------2</link>
            <guid isPermaLink="false">https://medium.com/p/a352140977ac</guid>
            <category><![CDATA[theory]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[kafka-consumer]]></category>
            <category><![CDATA[kafka-introduction]]></category>
            <category><![CDATA[apache-kafka]]></category>
            <dc:creator><![CDATA[Tharindu Hirantha]]></dc:creator>
            <pubDate>Mon, 28 Jun 2021 09:27:01 GMT</pubDate>
            <atom:updated>2021-06-28T19:33:55.523Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Everything you need to know about Kafka</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*POzg7EiN3VonTnwM6NBsSw.png" /></figure><h4>Real Problem</h4><p>We will talk about the real-world scenarios before we move on to Kafka.</p><p>Let’s assume there is a simple application that consists of one source system and one target system. So there is only one point of integration between them. In addition, the system will be easy to implement and maintain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/0*fmYrTQCPIW7Uhk0U" /></figure><p>Now suppose there are 4 source systems and 4 target systems. In that case, there may be many integrations between these systems. At least 4 and a maximum of 16 integration points. Then things get really complicated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*w82Ry-OoQbvDnP0f" /></figure><p>And below are the difficulties we have to deal with regarding the above architecture.</p><ul><li>Different Data formats — h<em>ow the data is transported</em> (JSON, Plain Text, etc..)</li><li>Different Protocols — how the data is parsed (HTTP, TCP, JDBC, etc..)</li><li>Complexity</li><li>Maintenance</li><li>Data schema — how it is shaped and may change</li><li>Increased load from the connections</li></ul><h4>Why Kafka?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sY822QmGVtUSlGXf" /></figure><p>Kafka is mainly used for de-coupling. Therefore, all the source systems’ data will be stored in Kafka, and target systems will source their data straight from Apache Kafka.</p><p>For example, Kafka will store taxi data, pricing data, financial transactions, user interactions, weather metrics, and many more. Once the data is stored in Kafka, we can send it into any system we want. Like databases, analytics systems, or audits.</p><p>K<strong>afka</strong> is open-source software that provides a framework for storing, reading, and analyzing streaming data. Now, Kafka is much more than a messaging queue. Kafka was originally developed by LinkedIn and is currently maintained by Confluent. It is a powerful tool for working with data streams and can be used in many other things. Kafka is distributed, which means it can be scaled when needed (horizontally scalable). Because of its distributed nature, Kafka is capable of operating very quickly (less than 10MS, near real-time). Large Kafka clusters can be capable of monitoring and handling millions of changes to a dataset every second. Apache Kafka is written in Scala and Java, but it is compatible with many other popular programming languages. Kafka is only used for a transportation mechanism.</p><p>Kafka can be used for the following use cases.</p><ul><li>Message systems (Chats)</li><li>Activity tracking (Taxi company)</li><li>Gather same metrics from different locations (Weather data)</li><li>Application log gathering</li><li>Stream processing</li><li>De-Coupling</li><li>Big Data Technologies (Spark, Flink, Hadoop, etc…)</li></ul><p>Example: LinkedIn uses Kafka to detect spam, and Netflix uses it for real-time movie recommendations. Uber uses Kafka to gather taxi details such as trip details, taxi details, fair details, and passenger details.</p><p>There is a number of concepts we need to learn in Kafka.</p><ul><li><strong>T</strong>opics and <strong>P</strong>artitions</li><li><strong>O</strong>ffsets</li><li><strong>B</strong>roker</li><li><strong>R</strong>eplication</li><li><strong>L</strong>eader</li><li><strong>P</strong>roducers and <strong>C</strong>onsumers</li></ul><h4>Topics</h4><p>The topic is like a database. The topic in Kafka is going to be the basis of everything. All the messages are stored in the Topic. Kafka can have many topics, and it’s identified by the name. Topic stored data is temporary. Default is only for 7 days.</p><h4><strong>P</strong>artitions</h4><p>Each Kafka topic is divided into partitions. There can be multiple partitions for a specific topic. Each partition is going to be ordered. We can specify the number of partitions when we are creating the topic or change the count later. Message order is only guaranteed within the partition. We can not swap or merge messages in a partition.</p><p>Let’s assume 10 messages are being processing on the Kafka topic. But in the background, these messages are stored in partitions. Messages are randomly assigned to the partition, and the partition’s message count is independent. Mean partition 0 can have 6 messages, and the other 2 partitions can have 2 messages each. Kafka is handling these operations. We don’t need to worry about this.</p><p>The below diagram will explain the relationship between a topic and partitions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/915/0*pDhnyi_yifhYdeQt" /></figure><h4><strong>O</strong>ffsets</h4><p>Each message within a partition gets an incremental ID, called ‘Offset.’ It’s like a unique ID in a database table. Offset only have a meaning for a specific partition. Offsets are one of three metrics that, when used together, can locate or identify a message. First, there is the <strong>topic</strong>, then within <strong>topics</strong> are partitions, and then finally, the ordering of the messages in the <strong>topics</strong> is referred to as the <strong>offset</strong>.</p><h4>Kafka Broker</h4><p>The broker is the server, and the Kafka cluster consists of many Kafka brokers on many servers. It’s good to have 3 brokers for a cluster. But 100 brokers are possible. Each Kafka Broker has a unique ID number. Kafka Brokers contains certain topic partitions.</p><p>The following diagram will explain the relationship between Topic, Partitions, and Kafka brokers.</p><p>Consider the following example, Topic A has 3 Partitions, and Topic B has 2 Partitions. And 3 brokers in the Kafka cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/808/0*WoVnM-EsyHsQfBZZ" /><figcaption>Kafka Broker</figcaption></figure><h4><strong>Topic Replication</strong></h4><p>A replication factor is the number of copies of data over multiple brokers. … The replication factor value should be greater than 1 always (between 2 or 3). This helps store a replica of the data in another broker from where the user can access it when there is a Broker failure.</p><p>The following diagram will explain partition replications.</p><p>Ex: 3 Brokers, 1 Topic, 2 Partitions, and 2 Replications.</p><p>Consider the following example, Broker 101 consists of Partition 0, and replication of Partition 0 consists in broker 102. Broker 102 consists of Partition 1, and replication of Partition 1 consists of broker 103.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/0*EX6DLd047At-s1K_" /><figcaption>Partition Replication</figcaption></figure><h4><strong>Leader for a Partition</strong></h4><p>Each partition has one server, which acts as the “leader.” The leader handles all read and write requests for the partition while the followers passively synchronize the data. If the leader fails, one of the followers will automatically become the new leader. When a broker is recovered, it will be the leader again<strong>. </strong>Thus, each partition has one leader and multiple in-sync replicas (ISR).</p><p>The following diagram will give an idea about how leaders and partitions are functioning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/0*Ix_AsBvAckglNQ2J" /></figure><p>Let’s assume broker 102 failed. The leader of partition 1 failed, and automatically, the ISR partition in broker 103 becomes the new leader. Then Kafka will be rebalanced like the following.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/0*zSzUmFCEisF0wEz4" /></figure><h4><strong>Producers</strong></h4><p>A producer is a client process that writes records to a Kafka topic. Producer automatically knows which broker and partition to write to. Because, as explained earlier, only the leader can write or read for the partition. In case of Broker failures, Producers will automatically recover. Kafka appends records from a producer(s) to the end of a topic. The producer will load the balance when the data is written.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0R6iEx1pYE-IMmcM" /></figure><p>There are 3 types of acknowledgments that the producer receives from data writes.</p><ul><li>acks=0 <em>(Producer don’t wait for the acknowledgment — possible data loss)</em></li><li>acks=1 <em>(Producer wait for leader acknowledgment — limited data loss)</em></li><li>acks=all <em>(Producer wait for leader + replicas acknowledgment — no data loss</em></li></ul><p>The following diagram will explain how leaders and replicas sync data between them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hxx7eAJV4391m8Xa" /></figure><h4><strong>Key Based Ordering</strong></h4><p>The partitioners shipped with Kafka guarantee that all messages with the same non-empty key will be sent to the same partition. If the key is provided, the partitioner will hash the key with murmur2 algorithm and divide it by the number of partitions. The result is that the same key is always assigned to the same partition. If a key is not provided, data is assiged based on round robin.</p><p>Key based messagas are helps to ordering for a specific field (object_id). Kafka can not guarantee this key goes to this partition. But all message for that key will always go to the <strong>same partition.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*S6EeLVAGtjkWS5Cv" /></figure><h4><strong>Consumers and Consumer Groups</strong></h4><p>A Consumer is a client process that reads/records from a Kafka topic. Same as producers, consumers automatically know which broker and partition to read from. And also, in case of Broker failures, Consumers will automatically recover. The consumer read in order within each partition.</p><p>Kafka consumers are typically part of a Consumer Group. Therefore, when multiple consumers read from a topic and belong to the same consumer group, each consumer will receive messages from a different subset of the topic’s partitions.</p><p>The following diagrams will explain how consumers work with partitions.</p><p>Scenario 1: Let’s take topic A with four partitions. Suppose we created a new consumer, C1, which is the only consumer in group G1; Consumer C1 will get all messages from all four T1 partitions.</p><p>Scenario 2: If we add another consumer, C2, to group G1. Consumer C1 and C2 will only get messages from two partitions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5uuViAvOZAmZ3oN_" /></figure><p>Scenario 3: If G1 has four consumers, then each will read messages from a single partition.</p><p>Scenario 4: If we add more consumers to a single group with a single topic than the partitions we have, some of the consumers will be idle and will not receive any message. We can apply this for reading heavy use cases where consumers read a lot of messages. Let’s assume C1 got failed; then C5 will start to read from partition 0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bQB39V2Uq3fbEE4i" /></figure><p>Scenario 5: In the previous example, if we add a new consumer group G2 with a single consumer, this consumer will get all the messages in topic T1 independent of what G1 is doing. G2 can have more than a single consumer, in which they will get a subset of partitions, just like we showed for G1, but G2 as a whole will still get all the messages regardless of other consumer groups.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Tr1GahMVDkTbLCf3" /></figure><h4><strong>Consumer Offsets</strong></h4><p>The offset is a simple integer number that Kafka uses to maintain the current position of a consumer. The current offset is a pointer to the last record that Kafka has already sent to a consumer. Each partition maintains the messages it has received in a sequential order where they are identified by an offset, also known as a position. So, the consumer doesn’t get the duplicate record twice because of the current offset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*O5nDQ1lewAcCKLMw" /><figcaption>Consumer Offsets</figcaption></figure><p>There are 3 different consumer delivery semantics.</p><p>1- At most once :</p><p><em>A message should be </em><strong><em>delivered once</em></strong><em>. It’s acceptable to lose a message rather than delivering a message twice in this semantic. Offsets are committed as soon as the message is received, and if the processing goes wrong, the message will be lost.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*em_t1oTbl028iZC5" /><figcaption>At most once</figcaption></figure><p>2- At least once :</p><p><em>It is acceptable to </em><strong><em>deliver a message more than once</em></strong><em>, but no message should be lost. The consumer ensures that all messages are read and processed, and if the processing goes wrong, the message will read again. This may result in processing duplicate messages. Therefore application should be idempotent. This is the most preferred semantics.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MQJWvbR_oZS85-SO" /><figcaption>At least once</figcaption></figure><p>3- Exactly once :</p><p><em>A message must be </em><strong><em>delivered only once, and no message</em></strong><em> should be lost. This can be achieved by Kafka to Kafka workflows using Kafka streaming API.</em></p><p><strong>Kafka Producer/Consumer Discovery</strong></p><p>Any Kafka client needs to connect to one broker, and it will be connected to the entire cluster. <em>(Every Kafka broker is also called a “bootstrap server”)</em>. Each broker knows all the details about the brokers, topics, and partitions (metadata)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/0*Ue7ClchQUR3wm21U" /></figure><h4>ZooKeeper</h4><p>Zookeeper is a centralized open-source server for maintaining and managing configuration information, naming conventions, and synchronization for distributed cluster environment. Zookeeper helps the distributed systems to reduce their management complexity by providing low latency and high availability. Kafka can’t work without Zookeeper.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*F5a--jBZR2txGKuW" /></figure><p>The following are some key points about Zookeeper.</p><ul><li>Manages Brokers (Keeps list of them)</li><li>Helps in performing leader election for partitions</li><li>Sends notifications to Kafka in case of changes (New topic, Broker kill, Delete topics)</li><li>By design, it operates with an odd number of servers (3,5,7)</li><li>Zookeeper has a ‘<strong>leader’</strong> (handles writes), the rest are ‘<strong>followers’</strong> (handle reads)</li><li>Zookeeper does <strong>NOT </strong>store consumer offsets (Kafka &gt; V0.10)</li></ul><h4><strong>Theory Roundup</strong></h4><p>We have covered most of the theory in Kafka. Following are the concepts we covered.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uQwkp3ikZw2uRrlsnPbbiw.jpeg" /></figure><h4>References</h4><blockquote><a href="https://www.udemy.com/share/1013hcBUIcclxXQ3g=/">‘https://www.udemy.com/share/1013hcBUIcclxXQ3g=/</a>’</blockquote><blockquote>‘<a href="https://kafka.apache.org/documentation/">https://kafka.apache.org/documentation/</a>’</blockquote><blockquote>‘<a href="https://www.linkedin.com/pulse/kafka-consumer-delivery-semantics-sylvester-daniel/">https://www.linkedin.com/pulse/kafka-consumer-delivery-semantics-sylvester-daniel/</a>’</blockquote><blockquote>‘<a href="https://docs.cloudera.com/runtime/7.2.9/kafka-overview/topics/kafka-overview-architecture.html">https://docs.cloudera.com/runtime/7.2.9/kafka-overview/topics/kafka-overview-architecture.html</a>’</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a352140977ac" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/apache-kafka-a352140977ac">Apache Kafka</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Consume message from Kafka and Save the read data in Cassandra using Apache Spark]]></title>
            <link>https://tharindu-hirantha.medium.com/consume-message-from-kafka-and-save-the-read-data-in-cassandra-using-apache-spark-b0d5f2d50f7e?source=rss-d7b5f3e4c319------2</link>
            <guid isPermaLink="false">https://medium.com/p/b0d5f2d50f7e</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[cassandra]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[pipeline]]></category>
            <dc:creator><![CDATA[Tharindu Hirantha]]></dc:creator>
            <pubDate>Mon, 21 Jun 2021 15:15:13 GMT</pubDate>
            <atom:updated>2021-06-21T15:15:13.770Z</atom:updated>
            <content:encoded><![CDATA[<h4>Integrate Cassandra and Kafka in Apache Spark</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l1QAB9oN-hkOVYvJbSi8JA.png" /></figure><h4>Prerequisites</h4><p><a href="https://tharindu-hirantha.medium.com/install-scala-kafka-cassandra-and-spark-69ace5a5fb27">Install Scala, Kafka, Cassandra, and Spark</a></p><h4>Implementation</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/691c737f5471748bd056a0032ecf6c7b/href">https://medium.com/media/691c737f5471748bd056a0032ecf6c7b/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e5b85dfc54b343aa11a9c79a70e4a540/href">https://medium.com/media/e5b85dfc54b343aa11a9c79a70e4a540/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8d1fc725a112ce0eb30dd281fc9e21c7/href">https://medium.com/media/8d1fc725a112ce0eb30dd281fc9e21c7/href</a></iframe><h4>Output</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/0*64N_fxP2d6PkX3Z4" /><figcaption>Push user data into kafka</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/0*wKJb-WFG2vLgP3_f" /><figcaption>Spark reads the data from kafka</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*xb3L0HvWe_hiwDwt" /><figcaption>Data saved in Cassandra DB</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b0d5f2d50f7e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Save data in Cassandra using Apache Spark]]></title>
            <link>https://tharindu-hirantha.medium.com/save-data-in-cassandra-using-apache-spark-f70bb35b04e5?source=rss-d7b5f3e4c319------2</link>
            <guid isPermaLink="false">https://medium.com/p/f70bb35b04e5</guid>
            <category><![CDATA[cassandra]]></category>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[save]]></category>
            <category><![CDATA[apache-spark]]></category>
            <dc:creator><![CDATA[Tharindu Hirantha]]></dc:creator>
            <pubDate>Sun, 20 Jun 2021 07:31:57 GMT</pubDate>
            <atom:updated>2021-06-20T07:45:35.866Z</atom:updated>
            <content:encoded><![CDATA[<h4>Integrate Cassandra with Apache Spark</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M94jEibQc0ALRywfFIM1Ng.png" /></figure><h4>Prerequisites</h4><p><a href="https://tharindu-hirantha.medium.com/install-scala-kafka-cassandra-and-spark-69ace5a5fb27">Install Scala, Kafka, Cassandra, and Spark</a></p><h4>Implementation</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/55ceb1a0c9b28549be83e5bd990acebe/href">https://medium.com/media/55ceb1a0c9b28549be83e5bd990acebe/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7ff37f44fb1fa6fe00d70943ca44beaf/href">https://medium.com/media/7ff37f44fb1fa6fe00d70943ca44beaf/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/32932a27796697b227eecf894b235536/href">https://medium.com/media/32932a27796697b227eecf894b235536/href</a></iframe><h4>Output</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/0*5jRwej-TVg8fhGo0" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f70bb35b04e5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Consume messages from the Kafka using Apache Spark]]></title>
            <link>https://tharindu-hirantha.medium.com/consume-messages-from-the-kafka-using-apache-spark-cc8b02a1b0d3?source=rss-d7b5f3e4c319------2</link>
            <guid isPermaLink="false">https://medium.com/p/cc8b02a1b0d3</guid>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[read]]></category>
            <category><![CDATA[kafka-streams]]></category>
            <category><![CDATA[apache-spark]]></category>
            <dc:creator><![CDATA[Tharindu Hirantha]]></dc:creator>
            <pubDate>Sun, 20 Jun 2021 06:33:39 GMT</pubDate>
            <atom:updated>2021-06-20T07:45:06.286Z</atom:updated>
            <content:encoded><![CDATA[<h4>Integrate Kafka with Apache Spark</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oXz2CoxabVs0HLdeogoFhg.png" /></figure><h4>Prerequisites</h4><p><a href="https://tharindu-hirantha.medium.com/install-scala-kafka-cassandra-and-spark-69ace5a5fb27">Install Scala, Kafka, Cassandra, and Spark</a></p><h4>Implementation</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/76ca1e1136be391d69558d464ae9ef3d/href">https://medium.com/media/76ca1e1136be391d69558d464ae9ef3d/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9d9ce904f68ebddf3955bed2fd71a522/href">https://medium.com/media/9d9ce904f68ebddf3955bed2fd71a522/href</a></iframe><h4>Output</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/524/0*tRdZMUuhrhNS2tAi" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/523/0*S5eBc76eFJqvS1_z" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc8b02a1b0d3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Recover or Change Lost Key Pair of AWS EC2 Linux Instance]]></title>
            <link>https://tharindu-hirantha.medium.com/recover-or-change-lost-key-pair-of-aws-ec2-linux-instance-c1c1ef9359a?source=rss-d7b5f3e4c319------2</link>
            <guid isPermaLink="false">https://medium.com/p/c1c1ef9359a</guid>
            <category><![CDATA[login]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[recovery]]></category>
            <category><![CDATA[ec2]]></category>
            <category><![CDATA[keypair]]></category>
            <dc:creator><![CDATA[Tharindu Hirantha]]></dc:creator>
            <pubDate>Sun, 20 Jun 2021 05:26:09 GMT</pubDate>
            <atom:updated>2021-06-20T05:26:09.680Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hX4DPBDZSZTbmnsMgrvCOQ.png" /></figure><p>1 .Launch a Temporary AWS instance. (Launch using a new Key pair or exiting available Key pair)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*35bLGyUYZ8YpjJtZZ29MWw.png" /></figure><p>2. Stop the ‘Live Server’</p><p>3. Navigate to Volumes and Detach the Live server volume from the instance</p><p>4. Attach Live server volume to newly launched ‘Temporary server’</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/1*WY7diAD8zhPLA61FbyHmIg.png" /></figure><p>5. Login to Temporary instance</p><p>6. Command: <em>Attached new volume will be listed</em></p><pre>lsblk</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*I1n0gLqkrBK4gsBL8QWI4A.png" /></figure><p>7. Command : <em>Mount point will be created</em></p><pre>mkdir -p /var/recovery-disk</pre><p>8. Command : <em>Mounting new volume</em></p><pre>mount -o nouuid /dev/xvdf1 /var/recovery-disk</pre><p>9. Command :</p><pre>df -h</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*vYvSEEAHxvsN82bm1QNRrQ.png" /></figure><p>10. Command : <em>Copying Temporary server authorized key to Live server authorized key</em></p><pre>cat /home/ec2-user/.ssh/authorized_keys &gt;&gt; /var/recovery-disk/home/ec2-user/.ssh/authorized_keys</pre><p>11. Command : Unmount the disk</p><pre>umount /var/recovery-disk</pre><p>12. Command :</p><pre>df -h</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*n1of9EPCEf1wIWcvZWF0zQ.png" /></figure><p>13. Stop the Temporary server and detach Live server volume</p><p>14. Reattach Live server volume to Live server. Use device name as ‘/dev/xvda’</p><p>15. Start the Live server and login (Login using a new Key pair or exiting available Key pair)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1c1ef9359a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Run a simple word count example using Apache Spark]]></title>
            <link>https://tharindu-hirantha.medium.com/run-a-simple-word-count-example-using-apache-spark-4d9c68bd3027?source=rss-d7b5f3e4c319------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d9c68bd3027</guid>
            <category><![CDATA[maven]]></category>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[word-count]]></category>
            <dc:creator><![CDATA[Tharindu Hirantha]]></dc:creator>
            <pubDate>Fri, 18 Jun 2021 14:31:24 GMT</pubDate>
            <atom:updated>2021-06-20T05:26:58.804Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7GVPtPuPd5ntV520Yf9KLQ.jpeg" /></figure><p>Prerequisites</p><p><a href="https://tharindu-hirantha.medium.com/install-scala-kafka-cassandra-and-spark-69ace5a5fb27">Install Scala, Kafka, Cassandra, and Spark</a></p><p>Implementation</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/895a487fc6202798049bac13260074c1/href">https://medium.com/media/895a487fc6202798049bac13260074c1/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c43f594c904095cb7b642662f0669ed6/href">https://medium.com/media/c43f594c904095cb7b642662f0669ed6/href</a></iframe><p>Sample input text document</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/367/0*lHzfQ7-dIMh4tToe" /></figure><p>Output</p><p>Console will print 2 outputs. In first output will print the lines that contains the word “spark”. In the second output will print the number of word counts of the lines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/495/0*Gh9wIwCRCR7CWiXA" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d9c68bd3027" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Spark]]></title>
            <link>https://tharindu-hirantha.medium.com/apache-spark-9b7d6331924b?source=rss-d7b5f3e4c319------2</link>
            <guid isPermaLink="false">https://medium.com/p/9b7d6331924b</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[install]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Tharindu Hirantha]]></dc:creator>
            <pubDate>Thu, 17 Jun 2021 14:28:58 GMT</pubDate>
            <atom:updated>2021-06-21T15:15:56.168Z</atom:updated>
            <content:encoded><![CDATA[<h4>Content</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*31XGaWuTRkdAJn5_gbVYuA.jpeg" /></figure><ul><li><a href="https://tharindu-hirantha.medium.com/apache-spark-introduction-b9770a97265c">Apache Spark Introduction</a></li><li><a href="https://tharindu-hirantha.medium.com/install-scala-kafka-cassandra-and-spark-69ace5a5fb27">Install Scala, Kafka, Cassandra, and Spark</a></li><li><a href="https://tharindu-hirantha.medium.com/run-a-simple-word-count-example-using-apache-spark-4d9c68bd3027">Run a simple word count example using Apache Spark</a></li><li><a href="https://tharindu-hirantha.medium.com/consume-messages-from-the-kafka-using-apache-spark-cc8b02a1b0d3">Consume messages from the Kafka using Apache Spark</a></li><li><a href="https://tharindu-hirantha.medium.com/save-data-in-cassandra-using-apache-spark-f70bb35b04e5">Save data in Cassandra using Apache Spark</a></li><li><a href="https://tharindu-hirantha.medium.com/consume-message-from-kafka-and-save-the-read-data-in-cassandra-using-apache-spark-b0d5f2d50f7e">Consume message from Kafka and Save the read data in Cassandra using Apache Spark</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9b7d6331924b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Install Scala, Kafka, Cassandra, and Spark]]></title>
            <link>https://tharindu-hirantha.medium.com/install-scala-kafka-cassandra-and-spark-69ace5a5fb27?source=rss-d7b5f3e4c319------2</link>
            <guid isPermaLink="false">https://medium.com/p/69ace5a5fb27</guid>
            <dc:creator><![CDATA[Tharindu Hirantha]]></dc:creator>
            <pubDate>Thu, 17 Jun 2021 14:21:41 GMT</pubDate>
            <atom:updated>2021-06-17T14:21:41.851Z</atom:updated>
            <content:encoded><![CDATA[<p>For Linux</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*LjRrDTpANwMbi_t1kC7nfQ.jpeg" /></figure><h4>Download</h4><p>Kafka 2.12–1.1.0<br><a href="https://www.apache.org/dyn/closer.cgi?path=/kafka/1.1.0/kafka_2.12-1.1.0.tgz">https://www.apache.org/dyn/closer.cgi?path=/kafka/1.1.0/kafka_2.12–1.1.0.tgz</a></p><p>Spark 2.2.1<br><a href="https://www.apache.org/dyn/closer.lua/spark/spark-2.2.1/spark-2.2.1-bin-hadoop2.6.tgz">https://www.apache.org/dyn/closer.lua/spark/spark-2.2.1/spark-2.2.1-bin-hadoop2.6.tgz</a></p><p>Cassandra 3.5<br><a href="http://cassandra.apache.org/download">http://cassandra.apache.org/download</a></p><p>Scala 2.12.5<br><a href="https://www.scala-lang.org/download/2.12.5.html">https://www.scala-lang.org/download/2.12.5.html</a></p><h4>Setup Environment Variables</h4><p>Unzip all the downloaded packages</p><pre><em>vim ~/.bash_profile<br><br>export SCALA_HOME=/FOLDER_LOCATION/scala-2.12.5<br>export SPARK_HOME=/FOLDER_LOCATION/spark-2.2.1-bin-hadoop2.6 </em><br><em>export CASSANDRA_HOME=/FOLDER_LOCATION/cassandra/apache-cassandra-3.5</em><br><em>export PATH=$PATH:$SPARK_HOME/bin</em><br><em>export PATH=$PATH:$SCALA_HOME/bin</em><br><em>export PATH=$PATH:$CASSANDRA_HOME/bin<br><br>source ~/.bash_profile</em></pre><h4>Start Kafka</h4><p><em>Start zookeeper with default configurations</em></p><pre>bin/zookeeper-server-start.sh config/zookeeper.properties</pre><p><em>Start kafka server with default configurations. With the zookeeper running, open a new terminal window.</em></p><pre>bin/kafka-server-start.sh config/server.properties</pre><p><em>Create new topic in kaffka. Topic name is “test01”</em></p><pre>kafka-topics.bat –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic test01</pre><h3>Start Cassandra</h3><p><em>Start cassandra server</em></p><pre>cassandra -f</pre><p><em>With the cassandra running, open a new terminal window and<br>access the Cassandra Query Language shell.</em></p><pre>cqlsh</pre><p><em>Create new key space</em></p><pre>CREATE KEYSPACE test01 WITH REPLICATION = { ‘class’: ‘SimpleStrategy’, ‘replication_factor’: 1 };</pre><p><em>To view all key spaces use</em></p><pre>DESCRIBE KEYSPACES;</pre><p><em>Switch key space</em></p><pre>USE test;</pre><p><em>Create a table</em></p><pre>CREATE TABLE users ( id UUID PRIMARY KEY, lastname text, firstname text );</pre><h3>Start Spark Shell</h3><pre>spark-shell</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=69ace5a5fb27" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Spark Introduction]]></title>
            <link>https://tharindu-hirantha.medium.com/apache-spark-introduction-b9770a97265c?source=rss-d7b5f3e4c319------2</link>
            <guid isPermaLink="false">https://medium.com/p/b9770a97265c</guid>
            <category><![CDATA[basics]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Tharindu Hirantha]]></dc:creator>
            <pubDate>Tue, 01 May 2018 11:48:31 GMT</pubDate>
            <atom:updated>2021-06-18T14:34:15.316Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*g-2NhqNKFFlXjXfx5sotow.jpeg" /></figure><p><strong>Big data</strong> analytics is the process of identifying hidden patterns and unknown correlation. Big Data has 3 parameters. Unstructured , Various streams and Veracity.</p><p>There are 2 type of big data analytics. <strong>Realtime analytics</strong> and B<strong>atch analytics</strong>.</p><p><strong>Batch analytics</strong> is based on the period of time (historical data) and <strong>Realtime analytics</strong> happens in based on the current data.</p><h4><strong>What is hadoop ?</strong></h4><p>Hadoop is a way to distribute very large files across the machines using simple programming models. Hadoop does not support real time. analaytics.</p><h4><strong>What is mapReduce ?</strong></h4><p>MapReduce is a programming model and an associated implementation for processing and generating big data sets with a parallel, distributed algorithm on a cluster. MapReduce needs to stored data in hdfs. MapReduce writes more data into disk. (IO is high).</p><h3><strong>Spark</strong></h3><p>It was release in 2013 and it was popular because of the easy of use and the speed (100X). Spark can use data stored in variety of formats (cassandra , AWS s3, Hdfs, Kafka). Spark is an extension for the apache hadoop and Spark is Not to replace hadoop. Spark allows for real and batch analysis and it’s faster processing and easy to use. Spark can do proccessing in-memory, while Hadoop MapReduce has to read from and write to a disk.</p><p><strong>Spark Features</strong></p><ul><li><strong>Fast data processing — (</strong>Spark is fast because, IO happens only when saving to memory and keeps data in memory). Faster than hadoop (100X).</li><li><strong>Machine learning</strong>. (Spark has MLlib — a built-in machine learning library, while Hadoop needs a third-party.)</li><li><strong>Joining datasets. </strong>(Due to its speed, Spark can create all combinations faster).</li><li>Spark can spill over to disk if memory is filled).</li><li>Fault tolerance. (<a href="https://data-flair.training/blogs/fault-tolerance-in-apache-spark/">https://data-flair.training/blogs/fault-tolerance-in-apache-spark/</a>).</li><li>Can use more data sources (HDFS, Cassandra, Kafka).</li><li>Can use standalone.</li><li>Parallel operation. (process data in parallel)</li><li>Spark RDD’s are Cacheable.</li><li><strong>Immutable</strong>. (Can’t be modified once created).</li><li><strong>Lazy Evaluated. </strong>(execution will not start until an action is triggered).</li><li><strong>Multiple languages are supported.</strong> (Python, Java, Scala, R)</li></ul><h4><strong>Spark RDD — Resilient Distributed dataset.</strong></h4><p><em>Distributed collection of data</em></p><p>“Resilient Distributed Datasets. Resilient Distributed Datasets (<strong>RDD</strong>) is a fundamental data structure of <strong>Spark</strong>. It is an immutable distributed collection of objects. Each dataset in <strong>RDD</strong> is divided into logical partitions, which may be computed on different nodes of the cluster.”</p><p>There are two types of RDD Operations –</p><p><strong>Transformations : </strong>is a lazy operation on a RDD that returns another RDD.</p><ul><li>filter()</li><li>join()</li></ul><pre>val textFile = sc.textFile(“spark.rtf”);<br>val lineWithSpark = textFile.filter(line =&gt; line.contains(“spark”));</pre><p><strong>Actions : </strong>is an operation that triggers execution of RDD transformations and returns a value.</p><ul><li>collect()</li><li>print()</li></ul><pre>lineWithSpark.collect();<br>lineWithSpark.print();</pre><p><strong>Spark Context :</strong></p><p>“A SparkContext is a client of Spark’s execution environment and it acts as the master of the Spark application. SparkContext sets up internal services and establishes a connection to a Spark execution environment. You can create RDDs, accumulators and broadcast variables, access Spark services and run jobs (until SparkContext stops) after the creation of SparkContext. Only one SparkContext may be active per JVM. You must stop() the active SparkContext before creating a new one.”</p><p>If you are running in shell it will create a <strong>Spark Context </strong>for you.</p><h4><strong>Spark ECO System</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/0*3dv-LakSeTXDWcb1" /></figure><p><strong>Spark SQL :</strong> is a distributed framework for structured data processing. Spark SQL works to access structured and semi-structured information. It also enables powerful, interactive, analytical application across both streaming and historical data</p><p><strong>Spark Streaming :</strong> allows scalable, high-throughput, fault-tolerant stream processing of live data streams. Spark can access data from sources like Kafka, Flume, Kinesis or TCP socket.<strong> </strong>Many applications need the ability to process and analyze not only batch data, but also streams of new data in real-time.</p><p><strong>Machine Learning : </strong>is a scalable machine learning library that delivers both high-quality algorithms (e.g., multiple iterations to increase accuracy) and blazing speed (up to 100x faster than MapReduce). The library is usable in Java, Scala, and Python as part of Spark applications.</p><p><strong>Graph Computation : </strong>enables users to interactively build, transform and reason about graph structured data at scale.</p><p><strong>Spark Core : </strong>general execution engine for the Spark platform that all other functionality is built on top of. It provides in-memory computing capabilities to deliver speed, a generalized execution model to support a wide variety of applications. Supports java, python and Scala languages.</p><p><em>For more : </em><a href="https://data-flair.training/blogs/apache-spark-ecosystem-components/"><em>https://data-flair.training/blogs/apache-spark-ecosystem-components/</em></a><em><br></em><a href="https://www.dezyre.com/article/apache-spark-ecosystem-and-spark-components/219"><em>https://www.dezyre.com/article/apache-spark-ecosystem-and-spark-components/219</em></a></p><h4>Spark Architecture</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/499/0*MoyKl-tj5oK6v68w" /></figure><p>Driver program is where we implement our application and entry point to spark. And then driver will call to worker node will setup executor. Executor will apply the function to data set.</p><p><strong>Driver :</strong> runs the main () function of the application and is the place where the Spark Context is created.</p><p><strong>Executor :</strong> is a distributed agent responsible for the execution of tasks.</p><p><strong>Cluster Manager :</strong> responsible for acquiring resources on the spark cluster and allocating them to a spark job.</p><p><em>For more : </em><a href="https://www.dezyre.com/article/apache-spark-architecture-explained-in-detail/338"><em>https://www.dezyre.com/article/apache-spark-architecture-explained-in-detail/338</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b9770a97265c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>