<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Amarendra Singh on Medium]]></title>
        <description><![CDATA[Stories by Amarendra Singh on Medium]]></description>
        <link>https://medium.com/@singh.amarendra?source=rss-672c997a41cf------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*gnbvTt-AtOa3RHMhRlx98Q.jpeg</url>
            <title>Stories by Amarendra Singh on Medium</title>
            <link>https://medium.com/@singh.amarendra?source=rss-672c997a41cf------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 21:07:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@singh.amarendra/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How Kubernetes aka k8s works? A Refresher...]]></title>
            <link>https://blog.devops.dev/how-kubernetes-aka-k8s-works-a-refresher-c593404b55c2?source=rss-672c997a41cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/c593404b55c2</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[container-orchestration]]></category>
            <category><![CDATA[control-plane]]></category>
            <category><![CDATA[worker-node]]></category>
            <category><![CDATA[architecture]]></category>
            <dc:creator><![CDATA[Amarendra Singh]]></dc:creator>
            <pubDate>Sun, 01 Oct 2023 18:04:43 GMT</pubDate>
            <atom:updated>2023-10-02T17:21:08.522Z</atom:updated>
            <content:encoded><![CDATA[<p>There are so many articles, videos and documents available online to learn about Kubernetes, so what is different in this blog? This blog is just to discuss the very basic question, How Kubernetes works? This is useful for people who are just starting with Kubernetes and have understanding of containers and knows that Kubernetes (sometimes shortened to K8s with the 8 standing for the number of letters between the “K” and the “s”) is an open-source system to deploy, scale, and manage containerized applications anywhere (on-prem or Cloud)</p><p>We’re just starting with the Kubernetes usage for some of our applications and my teammates wanted to understand the various components involved in Kubernetes. With lot of information available based on different viewpoints and scenarios, they were looking something simple to start with.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*J10UJSihQH_bbIHR3sUg9Q.png" /><figcaption>Kubernetes Architecture</figcaption></figure><p>The above diagram captures the Kubernetes components, specifically dividing them on the basis of their existence in Kubernetes Cluster. The components that manage the working of the clusters are deployed on Master node and the actual application code gets deployed on worker node as Containerized apps.</p><p>The Master node components together with kubelet and kube-proxy on worker nodes are called the <strong>Control Plane</strong>, as they are controlling the functioning of Kubernetes cluster.</p><p>Now, let’s try to understand the tasks needed to run any application in distributed system like Kubernetes and how and what Control Plane components are fulfilling those tasks.</p><p>1- How to schedule the application on the worker nodes?</p><p><strong>Scheduler: </strong>Scheduler is responsible of scheduling the Pods (smallest logical unit in Kubernetes) on worker nodes.</p><p>2- How to execute the application?</p><p><strong>Container Runtime: </strong>Container runtime provides a base to run the applications.</p><p>3- How to manage the node failures, onboarding new nodes, application failures, application replicas update etc.?</p><p><strong>Controller: </strong>Controller is responsible to manage the nodes using Nodes controller, manage the application instances using Replication controller etc.</p><p>4- How and where the data related with the cluster state maintained?</p><p><strong>etcd</strong>: As the applications are distributed on various nodes, we need some centralized mechanism to manage the state of the cluster, desired as well as current state. etcd is a distributed key-value store which stores all the cluster data.</p><p>5- Who manages the communication among various components?</p><p><strong>API Server:</strong> There are many components available in the Control Plane, which needs to be communicating with each other and API server is there as the medium through which the components interact with each other and also with the worker nodes.</p><p>6- Who handles the instructions on Worker node coming from Master node?</p><p><strong>kubelet</strong>: kubelet available on worker nodes handles the communication coming from API server</p><p>7- How the applications communicate with each other?</p><p><strong>kube-proxy:</strong> Applications/Pods need to communicate with each other as well. This is possible using kube-proxy component of Control Plane which is available on the worker nodes.</p><p>Now as we understood the functioning of each component in Control Plane, let us now see how they are involved when we try to apply the POD manifest file to the cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*pvvGLYqDo_H3HpDp16FsXg.png" /><figcaption>Pod Creation workflow</figcaption></figure><p><strong>kubectl :</strong> The user interacts with kubectl to apply a YAML manifest for the pod.</p><p><strong>API Server:</strong> kubectl communicates with the API server, which is the entry point to the control plane.</p><p><strong>Admission Controller: </strong>Admission controllers validate and mutate the incoming requests. This can include validting policies like ResourceQuotas, Pod Security policies etc.</p><p><strong>etcd:</strong> The API server stores the desired state of the pod in etcd.</p><p><strong>Controller Manager:</strong> The controller manager watches the API server for chages in the desired state, in this case, a new pod. It’s responsible for controllers like Replication, which may create pods to match the desired state.</p><p><strong>Scheduler:</strong> If the pod doesn’t have a Node specified in the configuration using node affinity etc., the Scheduler is responsible for assigning the pod to appropriate Node based on resource requirements, constraints and other policies.</p><p><strong>kubelet:</strong> Once the Scheduler assigns a node to the POD, the kubelet running on that node communicates with the API server to fetch the Pod’s manifest. It ensures that Pod runs on that Node as specified.</p><p><strong>Container runtime:</strong> The Container Runtime (Docker, containerd etc.) is responsible for pulling the container image and running the container.</p><p><strong>kubelet (Again):</strong> After the Pod is scheduled and started running, the kubelet will update the status of the pod in the API server, indicating its current state.</p><p><strong>Controller Manager (Again):</strong> If the pod is managed by a higher-level controller (e.g., Deployment). the Controller Manager ensures that the desired number of replicas are maintained.</p><p><strong>Event Log:</strong> Throughout this process, events and logs are generated and can be viewed using kubectl or other monitoring tolls.</p><p>This is a simplified representation of Pod creation workflow just to understand the Kubernetes components. There are many additional components and interactions involved in the Kubernetes control plane. The flow may vary based on specific configurations and resource types.</p><p>This is a blog explaining about core components of Kubernetes architecture. There are many other components in Kubernetes like Deployment, ReplicaSet, StatefulSet, Network Policies, Services, CRDs etc. which makes it a robust, scalable and extensible platform.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c593404b55c2" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/how-kubernetes-aka-k8s-works-a-refresher-c593404b55c2">How Kubernetes aka k8s works? A Refresher...</a> was originally published in <a href="https://blog.devops.dev/">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[KRaft Quorum: Kafka without Zookeeper]]></title>
            <link>https://medium.com/@singh.amarendra/kraft-quorum-kafka-without-zookeeper-28eb05017d10?source=rss-672c997a41cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/28eb05017d10</guid>
            <category><![CDATA[quorum]]></category>
            <category><![CDATA[broker]]></category>
            <category><![CDATA[kraft]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[zookeeper]]></category>
            <dc:creator><![CDATA[Amarendra Singh]]></dc:creator>
            <pubDate>Fri, 13 Jan 2023 18:57:40 GMT</pubDate>
            <atom:updated>2023-01-13T18:57:40.300Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*FnMRCIM8memCPhSfsey29A.png" /></figure><p>From the release of Apache Kafka 2.8.0, users have had early access to <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum">KIP-500</a>, which removes the Zookeeper dependency from Kafka. The release of Apache Kafka 3.3.0 saw KIP-500 become production-ready, meaning Kafka now relies on an internal Raft quorum that uses a Kafka Topic to store metadata, and some Kafka servers act as Controllers. The new feature simplifies cluster administration and infrastructure management and makes the Kafka cluster more scalable.</p><p>Please find my blog on this topic on <a href="https://www.conduktor.io/">Conduktor</a> —</p><p><a href="https://www.conduktor.io/blog/kraft-quorum-run-kafka-without-zookeeper">KRaft Quorum: How to run Kafka without Zookeper (conduktor.io)</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=28eb05017d10" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kafka Partition Assignment Strategy]]></title>
            <link>https://medium.com/@singh.amarendra/kafka-partition-assignment-strategy-eab799650835?source=rss-672c997a41cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/eab799650835</guid>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[assignment-strategy]]></category>
            <category><![CDATA[consumer]]></category>
            <category><![CDATA[partition]]></category>
            <category><![CDATA[consumer-group]]></category>
            <dc:creator><![CDATA[Amarendra Singh]]></dc:creator>
            <pubDate>Fri, 13 Jan 2023 18:52:42 GMT</pubDate>
            <atom:updated>2023-01-13T18:52:42.027Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/525/1*DykVjZxSnN3EaZCALY68pA.png" /><figcaption>RangeAssignor</figcaption></figure><p>Kafka consumers pull data from topic partitions, but what happens if a consumer dies? How will the partitions be reassigned, and what is the best method of reassignment?</p><p>Please find my blog on this topic on <a href="https://www.conduktor.io/">Conduktor</a> —</p><p><a href="https://www.conduktor.io/blog/kafka-partition-assignment-strategy">Apache Kafka Partition Assignment Strategy (conduktor.io)</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eab799650835" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Kafka’s Internal Storage and Log Retention]]></title>
            <link>https://medium.com/@singh.amarendra/understanding-kafkas-internal-storage-and-log-retention-5e7d71e88efe?source=rss-672c997a41cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e7d71e88efe</guid>
            <category><![CDATA[indexing]]></category>
            <category><![CDATA[storage]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[retention]]></category>
            <category><![CDATA[partition]]></category>
            <dc:creator><![CDATA[Amarendra Singh]]></dc:creator>
            <pubDate>Fri, 11 Nov 2022 23:37:37 GMT</pubDate>
            <atom:updated>2022-11-11T23:37:37.585Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/985/1*6GHeGstdTAK-vlsGmURT9A.png" /></figure><p>Apache Kafka is a commit-log system. The records are appended at the end of each Partition, and each Partition is also split into segments. Segments help delete older records through Compaction, improve performance, and much more.</p><p>Please find my blog on this topic on <a href="https://www.conduktor.io/">Conduktor</a> —</p><p><a href="https://www.conduktor.io/blog/understanding-kafkas-internal-storage-and-log-retention">Understanding Kafka’s Internal Storage and Log Retention (conduktor.io)</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e7d71e88efe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My experience in getting the Confluent Certified Developer for Apache Kafka(CCDAK) certification]]></title>
            <link>https://blog.devgenius.io/my-experience-in-getting-the-ccdak-confluent-certified-developer-for-apache-kafka-certification-2bd6089c599c?source=rss-672c997a41cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/2bd6089c599c</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[certification]]></category>
            <category><![CDATA[ccdak]]></category>
            <category><![CDATA[confluent]]></category>
            <category><![CDATA[experience]]></category>
            <dc:creator><![CDATA[Amarendra Singh]]></dc:creator>
            <pubDate>Sun, 12 Jun 2022 18:08:38 GMT</pubDate>
            <atom:updated>2022-06-14T16:31:56.666Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/805/1*whjaAk2_G29LBDuOzxE2LQ.png" /></figure><p>It started in the end of April-2022 when I first thought of giving the CCDAK certification as it was the only available Kafka certification apart from CCAAK (Confluent Certified Administrator for Apache Kafka), conducted by <a href="https://www.confluent.io/">Confluent</a>. I was having good enough experience working with Kafka products be it Kafka Core, Kafka Connect, basic Kafka Streams and was successfully created, deployed and managed couple of Kafka based projects in Production environment. When I started, I didn’t find any content which provides all the details related with certification content, preparations, important topics, exam details etc. In this blog, I’ll try to capture all these details at one place to guide you to get this certification in your first attempt.</p><p>I am going to structure this blog in following step by step process to get certified with CCDAK. This is applicable to all whether you’re just starting with Kaka, have some experience or have exposure to many of its products and just want to validate your knowledge with this certification-</p><ol><li>Check certification content</li><li>Verify what you already know and where you stand</li><li>Plan your preparations</li><li>Schedule your exam</li><li>During the exam</li><li>After the exam</li></ol><p><strong>Check certification content: <br></strong>The first and the foremost thing is to check the exam content. You can always check this on<a href="https://www.confluent.io/certification/#prep-for-certification"> Confluent site</a>, while it mentions how to prepare, what are the requirements for CDAK but doesn’t provide exact details on what topics to cover (because you need to know the all :) ). Here is a list of Kafka products you should be aware-</p><ul><li>Core Kafka (Producer, Consumer, Consumer group, Brokers, Zookeeper, Topic, Partition, Compaction, Deliver Guarantees, Configurations etc.)</li><li>Kafka Connect</li><li>Kafka Streams</li><li>KSQL and KTable</li><li>Schema Registry and Rest proxy</li></ul><p><strong>Verify what you already know and where you stand: <br></strong>You can verify this by taking the <a href="https://training.confluent.io/learningpath/confluent-fundamentals-accreditation">Confluent’s fundamental accreditation</a> exam. This exam, once passed, provides a <a href="https://www.credential.net/c6881b48-cffd-428a-9990-0d11a8a5543b#gs.yb79cx"><strong>digital badge</strong></a> that can be used to reflect that you’ve basic understanding of Kafka.</p><p><strong>Plan your preparations: <br></strong>This is obviously the most important part of the process. You would have some basic knowledge of Kafka and would be using that in your projects but to pass the CCDAK certification, you should prepare all the topics as we don’t know the passing percentage (you only get either the pass status or fail). For preparations, I would highly recommend the <a href="https://medium.com/@stephane.maarek">Stephane Maarek’s</a> Apache Kafka series on Udemy.</p><ul><li><a href="https://www.udemy.com/course/apache-kafka/">Apache Kafka Series — Learn Apache Kafka for Beginners v3</a></li><li><a href="https://www.udemy.com/course/kafka-connect/">Apache Kafka Series — Kafka Connect Hands-on Learning</a></li><li><a href="https://www.udemy.com/course/kafka-streams/">Apache Kafka Series — Kafka Streams for Data Processing</a></li><li><a href="https://www.udemy.com/course/kafka-ksql/">Apache Kafka Series — KSQL on ksqlDB for Stream Processing !</a></li><li><a href="https://www.udemy.com/course/confluent-schema-registry/">Apache Kafka Series — Confluent Schema Registry &amp; REST Proxy</a></li></ul><p>Some of the reasons for recommending these courses are -</p><ul><li>You get the whole exam content at one place and by the same instructor, so you can easily accommodate with his teaching skills, pace etc.</li><li>The courses are very well articulated and provides good in-depth information on each of these Kafka products</li></ul><p>If you follow these courses and also do the practical exercises, you should be ready to take on the exam BUT I would suggests to read the concepts in detail in the <a href="https://www.confluent.io/resources/kafka-the-definitive-guide-v2/">Kafka Definitive Guide</a>, which has been written by the creators of Kafka.</p><p>If you’re starting with Kafka, consider spending good enough time ( at least 2–3 months) completing these courses with practical exercises and understanding the concepts. And if you’re already using Kafka, writing Producers, Consumers, Kafka Connect etc. you can skip some of these courses (you can still do it to brush-up on your skills).</p><p>After completing your training materials, you would need some good tests to verify your learning. Unfortunately not much of the practice tests are available but again Stephane comes for the rescue here :) with his 3 set of 50 unique questions in each set. This gives a lot of confidence to you about your preparations and also provides topic-wise analysis of your responses. You can then focus on those specific areas and retry the quiz again.</p><ul><li><a href="https://www.udemy.com/course/confluent-certified-developer-for-apache-kafka/">CCDAK Confluent Certified Developer for Apache Kafka TESTS</a></li></ul><p><strong>Schedule your exam: <br></strong>You can schedule your CCDAK exam on <a href="https://prod.examity.com/Confluent/">Examity</a>. I had scheduled my exam 1 day after I was done with my preparations but you can set a timeline for your preparations and can schedule it even before starting your prep.</p><p>You can reschedule your exam 48 hours before the exam time. To schedule your exam within the next 24 hours, you need to pay some extra bucks.</p><p>Cost of CCDAK proctored exam — $150 + applicable local taxes</p><p>Cost of CCDAK in India — I scheduled my exam on 3rd June 2022 and had to pay around INR 14,440 (including taxes)</p><p>Check FAQs at- <a href="https://www.confluent.io/certification/">https://www.confluent.io/certification/</a></p><p><strong>During the exam: <br></strong>On your exam day, try to be calm and relax. If you have prepared well, you are going to do well and get your certification easily. <br>You need to answer 60 multiple-choice questions in 90 minutes from your laptop (with webcam) under the supervision of online proctor. There is no negative scoring so try to answer all the questions.<br>Confluent doesn’t specify the correct number of questions or percentage of marks in order to pass the exam. Result will be shown immediately (PASS or FAIL) at the end of exam. No scoring or percentage is provided.</p><p><strong>After the exam: <br></strong>Once you’re done with your exam, you will get the result immediately on the screen, you will also get the result on your email.<br>If you clear the exam, you will get the<a href="https://www.credential.net/7f1e8041-a332-4002-bde2-3e22095f23a3"> certificate and badge</a> in an email from Confluent. The email also lists the following things to do do join the exclusive peer group-</p><ol><li><strong>Join</strong> Confluent’s exclusive <a href="https://www.linkedin.com/groups/8967626/">LinkedIn Confluent Certified Group</a></li><li><strong>Subscribe</strong> to the #certification channel on the <a href="https://join.slack.com/t/confluentcommunity/shared_invite/zt-rsph55gh-ayipVttndzQtWnWPGAepgQ">Community Slack</a></li><li><strong>Bookmark </strong>the Certification and Training page on the <a href="https://forum.confluent.io/c/certification-and-training/30">Community Forum</a></li><li><strong>Visit</strong> <a href="https://www.confluent.io/certification/">confluent.io/certification</a></li></ol><p><strong>Some Tips:</strong></p><ol><li>Prepare well for the exam, if you complete the exam contents, you shall easily clear the certification.</li><li>There is no retake available with the exam price. You can only re-take the exam after 14 days and will have to pay the whole amount again. So schedule it only when you have enough confidence.</li><li>There is no negative scoring so try to answer all the questions.</li><li>Timing is not an issue with this exam, so try to answer all the questions in one go and revisit the whole lot again for verification.</li><li>Don’t waste time searching for CCDAK Certification Dumps, there isn’t any available. If you follow the courses described here and understand the concepts, you’ll do well in the exam.</li><li>In my case, most of the questions were from Kafka Streams and KSQL so make sure you’ve good command in those areas.</li><li>Some very important topics in each product-</li></ol><blockquote><strong>Core Kafka</strong>- Partition, Segment, Log retention and cleanup policy, Replication, Producer acknowledgment, min.insync.replica, serialization/de-serialization, Producer configuration- linger.ms, batch.size, max.in.flight.per.connection, Message key, Consumer group and Consumers, Consumer offset, Delivery semantics, Commit strategy, Delivery guarantee, Some Kafka CLI commands, consumer rebalances and static group membership, Compaction, Partition Assignor etc.</blockquote><blockquote><strong>Zookeeper</strong>- Zookeeper ensemble, zookeeper cluster to withstand loss of N server, require 2N+1 servers, Zookeeper tickTime, initLimit and syncLimit</blockquote><blockquote><strong>Kafka Connect</strong>- Source Connect default configs, Sink connect default configs, Debezium allows max tasks equal to the number of table associated</blockquote><blockquote><strong>Kafka Streams and KSQL</strong>- Topology, Stateful Operators, Stateless operators, Window, KStream, KTable, GlobalKTable, Joins, co-partitioning concept, write to topic and get new stream, KStream to KTable and vice-versa, Exactly once, use of persistent volume state store, KSQL CLI commands, push query, Merging streams etc.</blockquote><blockquote><strong>Confluent Schema registry</strong>- Avro format, Schema compatibility , Schema evolution, where does schema gets stored, schema caching etc.</blockquote><blockquote><strong>Rest Proxy</strong>- encoding and decoding of messages, Supported types,</blockquote><p><strong>Conclusion: </strong>Every exam requires a good preparation and practice. Preparations make sure you understand the concepts well and able to apply them in various designs etc. You don’t just want to clear the exam for the sake and not applying your knowledge to the critical design scenarios, making better products and giving back to the society.</p><p><strong>All the very best for your exam, I know you’re going to get that.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2bd6089c599c" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/my-experience-in-getting-the-ccdak-confluent-certified-developer-for-apache-kafka-certification-2bd6089c599c">My experience in getting the Confluent Certified Developer for Apache Kafka(CCDAK) certification</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kafka on Kubernetes: Using Strimzi — Part 6 (Monitoring)]]></title>
            <link>https://blog.devgenius.io/kafka-on-kubernetes-using-strimzi-part-6-monitoring-709a43198bf5?source=rss-672c997a41cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/709a43198bf5</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[kafkaonkubernetes]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[strimzi]]></category>
            <dc:creator><![CDATA[Amarendra Singh]]></dc:creator>
            <pubDate>Sun, 29 May 2022 13:16:12 GMT</pubDate>
            <atom:updated>2022-05-31T07:19:56.337Z</atom:updated>
            <content:encoded><![CDATA[<h3>Kafka on Kubernetes: Using Strimzi — Part 6(Monitoring)</h3><p>This is the final part of the blog series , <strong>Kafka on Kubernetes: Using </strong><a href="https://strimzi.io/"><strong>Strimzi</strong></a><strong>.</strong> In the <a href="https://medium.com/@singh.amarendra/kafka-on-kubernetes-using-strimzi-part-5-security-fc878178cd04">previous part</a> we have discussed about the various security aspects in Strimzi to secure the Kafka cluster on Kubernetes. In this part we’re going to discuss the very important part of the Cluster reliability i.e. Monitoring. Many times people just ignore setting up the Monitoring for their Kafka cluster and it becomes nightmare in case of any issue in the cluster.</p><p>Well, I guess nobody wants to be in the situation where your Kafka cluster is not working properly in Production environment without you knowing about it. But the pros is that you don’t need to wake up in the night :P</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*Nm5Ri9AkyEn9VXGTclFvJw.png" /></figure><p>Kafka is a Java application and it exposes its metrics using <a href="https://www.oracle.com/technical-resources/articles/javase/jmx.html">JMX</a> (Java Management Extension), And hence almost all the Kafka monitoring tools integrates with JMX metrics and get all the Kafka related metrics. Here are some of the Kafka monitoring tools on Kubernetes-</p><ol><li>Prometheus + Grafana</li><li><a href="https://docs.newrelic.com/docs/infrastructure/host-integrations/host-integrations-list/kafka/kafka-integration/">New Relic</a></li><li><a href="https://www.datadoghq.com/blog/monitor-kafka-with-datadog/">Datadog</a> etc.</li></ol><p>There are three main parts to Monitor your Cluster-</p><ul><li>Scraping and storing the Metrics</li><li>Querying and showing them on a meaningful Dashboard</li><li>Alerting in case of any condition violation</li></ul><p>As Prometheus has become the standard for Monitoring the Kubernetes native applications, Strimzi supports it out of the box and provides many Grafana Dashboards, easy configuration to setup Prometheus+Grafan+Alert Manager for your Kafka cluster.</p><p><strong>For Scraping and storing</strong> — <a href="https://prometheus.io/">Prometheus</a> is an open source monitoring solution which has become the de-facto standard for metrics and alerting in the cloud native world. The best thing about Prometheus is that it scrapes the metrics from your Kafka Cluster and store them in its time-series database, unlike other monitoring tools where your application needs to push these metrics.</p><p><strong>For Querying and Dashboarding</strong> — Once the metrics is available in Prometheus DB, you can query it using the PromQL. Many tools support PromQL like Grafana, New Relic etc.</p><p><strong>For Alerting</strong> — Prometheus will evaluate the rules against the metrics which it is scraping and when any of the rules is matched, it will send it to <a href="https://prometheus.io/docs/alerting/alertmanager/">Alertmanager</a>. AlertManager will be then responsible for managing these alerts. You can setup the notification channel in case of any alert like email, slack, PagerDuty etc.</p><p>Strimzi uses two of the open source projects to get all the metrics out of the Kafka cluster and send them to Prometheus.</p><ul><li><a href="https://github.com/prometheus/jmx_exporter">Prometheus JMX Exporter</a>- Apache Kafka doesn’t support Prometheus metrics natively by default. As discussed above, it exposes JMX metrics. Prometheus JMX exporter project takes the JMX metrics and exposes them as Prometheus endpoint.</li><li><a href="https://github.com/danielqsj/kafka_exporter">Kafka Exporter</a>- Kafka brokers provide a lot of useful metrics related to the broker state, usage, and performance. But some important metrics are missing. For example it doesn’t provide any metrics about the consumer lag or information about topics. Kafka Exporter connects to Kafka as a client and collects different information about topics, partitions and consumer groups. It then exposes this information as a Prometheus metric endpoint.</li></ul><p>So with Prometheus we’re going to monitor following things-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TpugnaqxW-n_SI1YDAs-rw.png" /></figure><p>We can see that apart from monitoring the Kafka metrics, Strimzi specific components, we have Strimzi Canary as well.</p><p><strong>Strimzi Canary</strong>- Strimzi team has created a project <a href="https://github.com/strimzi/strimzi-canary">Strimzi-Canary</a> to identify whether the Kafka Cluster is working properly or not. It does that by creating a Canary topic with partitions equal to the number of brokers in the cluster, and creates a Producer and Consumer to produce and consume the data from the canary topic.</p><p>We can enable JMX Prometheus Exporter easily by adding following block in our Kafka resource and adding the rules in <a href="https://github.com/AmarendraSingh88/kafka-on-kubernetes/blob/main/kafka-demo/demo3-monitoring/kafka-metrics-config.yaml">kafka-metrics-config.yaml</a>-</p><pre>spec:<br>  kafka:<br>    metricsConfig:<br>      type: jmxPrometheusExporter<br>      valueFrom:<br>        configMapKeyRef:<br>          name: kafka-metrics<br>          key: kafka-metrics-config.yaml</pre><p>To enable Kafka Exporter we just need to add below lines of code in our Kafka definition-</p><pre>spec:<br>  kafkaExporter:<br>    topicRegex: &quot;.*&quot;<br>    groupRegex: &quot;.*&quot;</pre><p>Now let’s discuss how to configure Prometheus easily in our setup. The resources used in these steps can be found <a href="https://github.com/AmarendraSingh88/kafka-on-kubernetes/tree/main/kafka-demo/demo3-monitoring">here</a>.</p><p>1- Create monitoring namespace:</p><pre>kubectl create namespace monitoring</pre><p>2- Deploy Prometheus operator:</p><pre>kubectl apply -f prometheus-operator-deployment.yaml -n monitoring --force-conflicts=true --server-side</pre><p>3- Create configmap for jmx metrics:</p><pre>kubectl apply -f kafka-metrics-config.yaml -n monitoring<br>kubectl apply -f zookeeper-metrics.yaml -n monitoring</pre><p>4- Update the Kafka resource with jmxPrometheusExporter to scrape the jmx metrics and kafkaExporter for exporting the topic and consumer lag metrics</p><pre>kubectl apply -f kafka.yaml -n kafka</pre><p>5- Deploy Prometheus -</p><pre>kubectl apply -f prometheus.yaml -n monitoring</pre><p>6- Deploy pod monitor-</p><pre>kubectl apply -f strimzi-pod-monitor.yaml -n monitoring</pre><p>7- Deploy Grafana -</p><pre>kubectl apply -f grafana.yaml -n monitoring</pre><p>8- Port forward for Prometheus and Grafana-</p><pre>kubectl port-forward svc/grafana 3000:3000 -n monitoring<br>kubectl port-forward svc/prometheus-operated 9090:9090 -n monitoring</pre><p>9- Add Prometheus datasource in Grafana and upload Grafana dashboards from<a href="https://github.com/strimzi/strimzi-kafka-operator/tree/0.28.0/examples/metrics/grafana-dashboards"> Strimzi provided dashboards</a> for Kafka, Zookeeper, Kafka Connect, MirrorMaker etc.</p><p>Here is a sample Grafana dashboard for Kafka overview-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ubM2Fi-vlQf3rr-w5SJJJA.png" /><figcaption><a href="https://grafana.com/blog/2021/07/26/get-comprehensive-monitoring-for-your-apache-kafka-ecosystem-instances-quickly-with-grafana-cloud/">source</a></figcaption></figure><ul><li><a href="https://prometheus.io/docs/alerting/alertmanager/">Prometheus Alertmanager</a> is a plugin for handling alerts and routing them to a notification service. Alertmanager supports an essential aspect of monitoring, which is to be notified of conditions that indicate potential issues based on alerting rules.</li></ul><p>Strimzi provides <a href="https://strimzi.io/docs/operators/0.21.1/full/deploying.html#ref-metrics-config-files-str">example configuration files for Prometheus Alertmanager</a>.</p><p>See <a href="https://strimzi.io/docs/operators/0.21.1/full/deploying.html#assembly-metrics-prometheus-alertmanager-str">Strimzi documentation</a> for setting up the Alert Manager.</p><p><strong>Conclusion</strong>- This is the part-6 and the final blog of the series <strong>Kafka on Kubernetes: Using Strimzi</strong> where we discussed on how to easily monitor the Kafka cluster. In this series, we’ve discussed</p><ol><li><a href="https://medium.com/@singh.amarendra/kafka-on-kubernetes-using-strimzi-part-1-83d74564135e">basics of Strimzi to setup and maintain Kafka Cluster on Kubernetes</a></li><li><a href="https://medium.com/@singh.amarendra/kafka-on-kubernetes-using-strimzi-part-2-71a8ba8e9605">how to setup Kafka cluster using Strimzi</a></li><li><a href="https://medium.com/@singh.amarendra/kafka-on-kubernetes-using-strimzi-part-3-configuration-options-f8aa027e9ba0">discussed about production ready configurations</a></li><li><a href="https://medium.com/@singh.amarendra/kafka-on-kubernetes-using-strimzi-part-4-scalability-59da50575fec">have seen how to easily scale the Kafka clients using Keda</a></li><li><a href="https://medium.com/@singh.amarendra/kafka-on-kubernetes-using-strimzi-part-5-security-fc878178cd04">How to secure the Kafka cluster</a></li><li>discussed about monitoring the Kafka cluster easily with Prometheus+Grafana+Alert Manager (this part)</li></ol><p>So, we’ve setup the <strong>production ready Kafka cluster on Kubernetes which is scalable, secure and has monitoring and alerting setup on it.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=709a43198bf5" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/kafka-on-kubernetes-using-strimzi-part-6-monitoring-709a43198bf5">Kafka on Kubernetes: Using Strimzi — Part 6 (Monitoring)</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kafka on Kubernetes: Using Strimzi — Part 5(Security)]]></title>
            <link>https://blog.devgenius.io/kafka-on-kubernetes-using-strimzi-part-5-security-fc878178cd04?source=rss-672c997a41cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/fc878178cd04</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[kafkaonkubernetes]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[strimzi]]></category>
            <dc:creator><![CDATA[Amarendra Singh]]></dc:creator>
            <pubDate>Mon, 16 May 2022 17:57:58 GMT</pubDate>
            <atom:updated>2022-05-31T07:19:54.694Z</atom:updated>
            <content:encoded><![CDATA[<h3>Kafka on Kubernetes: Using Strimzi — Part 5 (Security)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/465/1*lVgF8q-hoYg993YLLOotCg.png" /><figcaption>source — <a href="http://joyreactor.com/post/4708653">http://joyreactor.com/post/4708653</a></figcaption></figure><p>Welcome to the Part-5 of the series, <strong>Kafka on Kubernetes: Using </strong><a href="https://strimzi.io/"><strong>Strimzi</strong></a><strong>. </strong>So far we have seen the Strimzi basics in <a href="https://medium.com/@singh.amarendra/kafka-on-kubernetes-using-strimzi-part-1-83d74564135e">Part-1</a>, created a Kafka cluster setup on Kubernetes using Strimzi in <a href="https://medium.com/@singh.amarendra/kafka-on-kubernetes-using-strimzi-part-2-71a8ba8e9605">Part-2</a>, discussed about important configurations and suggestions while making the cluster production-ready in <a href="https://medium.com/@singh.amarendra/kafka-on-kubernetes-using-strimzi-part-3-configuration-options-f8aa027e9ba0">Part-3</a>. In the <a href="https://medium.com/@singh.amarendra/kafka-on-kubernetes-using-strimzi-part-4-scalability-59da50575fec">previous section</a> we have seen how can we easily scale the Consumers using KEDA.</p><p>In this section we’ll discuss about a very important aspect- <a href="https://en.wikipedia.org/wiki/Security">SECURITY</a>. As Kafka is a streaming platform and we deal with lot of data, which could be sensitive in nature as well, this becomes utmost important.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aDX_nbe9OMzqYOdJRuPFKw.png" /><figcaption>source — <a href="https://fastnforward.blog/introduction-to-kafka-security/">fastnforward.blog</a></figcaption></figure><p>Broadly, security for any application/platform/system can be validated by following three things-</p><ul><li><a href="https://www.investopedia.com/terms/e/encryption.asp#:~:text=Encryption%20is%20a%20means%20of,makes%20the%20original%20information%20unreadable."><strong>Encryption</strong></a> — Whether the data is encrypted during exchange or in rest</li><li><a href="https://en.wikipedia.org/wiki/Authentication"><strong>Authentication</strong></a> — Whether only the authenticated users are allowed into the system</li><li><a href="https://en.wikipedia.org/wiki/Authorization"><strong>Authorization</strong></a> — Only the authorized users should have access to different part of the application/system</li></ul><p>If all the above aspects are inline with the security standards, we can say that our platform is secure. But the things doesn’t end here, security is a critical requirement for any system and we should re-visit the security standards/policies periodically.</p><p>Now let’s see what all options Strimzi provides in order to secure the Kafka cluster on Kubernetes-</p><p><strong>Encryption: </strong>Strimzi supports <a href="https://en.wikipedia.org/wiki/Transport_Layer_Security">Transport Layer Security (TLS)</a>, a protocol for encrypted communication.</p><p>Communication is always encrypted for communication between:</p><ul><li>Kafka brokers</li><li>ZooKeeper nodes</li><li>Operators and Kafka brokers</li><li>Operators and ZooKeeper nodes</li><li>Kafka Exporter</li></ul><p>For configuring secure communication between client and brokers, we can setup <a href="https://en.wikipedia.org/wiki/Transport_Layer_Security">CA (certificate authority)</a>. A TLS Certificate Authority (CA) issues certificates to authenticate the identity of a component. Strimzi verifies the certificates for the components against the CA certificate.</p><ul><li>Strimzi components are verified against the <em>cluster CA</em> Certificate Authority (CA)</li><li>Kafka clients are verified against the <em>clients CA</em> Certificate Authority (CA)</li></ul><p><strong>Authentication: </strong>Kafka listeners use authentication to ensure a secure client connection to the Kafka cluster.</p><p>Supported authentication mechanisms:</p><ul><li><a href="https://en.wikipedia.org/wiki/Mutual_authentication">Mutual TLS</a> client authentication (on listeners with TLS enabled encryption)</li><li><a href="https://strimzi.io/docs/0.12.1/#con-scram-sha-authentication-using-uo">SASL SCRAM-SHA-512</a></li><li><a href="https://strimzi.io/blog/2019/10/25/kafka-authentication-using-oauth-2.0/">OAuth 2.0 token based authentication</a></li><li>Custom authentication</li></ul><p>The User Operator manages user credentials for TLS and SCRAM authentication, but not OAuth 2.0. For example, through the User Operator you can create a user representing a client that requires access to the Kafka cluster, and specify TLS as the authentication type.</p><p>Using OAuth 2.0 token-based authentication, application clients can access Kafka brokers without exposing account credentials. An authorization server handles the granting of access and inquiries about access.</p><p>Custom authentication allows for any type of kafka-supported authentication. It can provide more flexibility, but also adds complexity.</p><p>For most of the cases, mutual TLS is sufficient. In mutual TLS the authenticity of Client and Server gets verified. Here’s how we can easily configure in Kafka resource for mutual TLS -</p><pre>...<br>spec:<br>  kafka:<br>    version: 2.8.0<br>    replicas: 3<br>    listeners:<br>      ...<br>     <strong> tls:<br>        authentication:<br>          type: tls<br>...</strong></pre><p>When we deploy the Strimzi Kafka cluster, with authentication enabled, it creates a<em> kafka-cluster-ca-certificate</em> as Kubernetes secret. This is the cluster CA certificate which can be used to verify the server.</p><p><strong>Authorization</strong>: Kafka clusters use authorization to control the operations that are permitted on Kafka brokers by specific clients or users.</p><p>Supported authorization mechanisms:</p><ul><li>Simple authorization</li><li>OAuth 2.0 authorization (if you are using OAuth 2.0 token-based authentication)</li><li><a href="https://strimzi.io/blog/2020/08/05/using-open-policy-agent-with-strimzi-and-apache-kafka/">Open Policy Agent (OPA) authorization</a></li><li>Custom authorization</li></ul><p>Simple authorization uses <strong>AclAuthorizer</strong>, the default Kafka authorization plugin. AclAuthorizer uses Access Control Lists (ACLs) to define which users have access to which resources.</p><p>Here is how we can create a simple Kafka user and add required permission to it-</p><pre>apiVersion: kafka.strimzi.io/v1beta2<br>kind: KafkaUser<br>metadata:<br>  name: my-user<br>  labels:<br>    strimzi.io/cluster: my-cluster<br>spec:<br>  authentication:<br>    type: tls<br>  <strong>authorization:<br>    type: simple (1)</strong><br>    acls:<br>      - resource:<br>          type: topic <strong>(2)</strong><br>          name: my-topic<br>          patternType: literal<br>        operation: Read<br>      - resource:<br>          type: topic<br>          name: my-topic <strong>(3)</strong><br>          patternType: literal<br>        operation: Describe <strong>(4)</strong><br>      - resource:<br>          type: group<br>          name: my-group<br>          patternType: prefix <strong>(5)</strong><br>        operation: Read</pre><ol><li>To enable the authorization for Kafka cluster</li><li>Adding ACL for <strong>type = topic</strong>, other types are <strong>group</strong>, <strong>cluster</strong> etc.</li><li>Name of the resource i.e. topic name, group name etc.</li><li>Allowed operation on the resource such as — Read, Write, Describe, Create, Alter etc.</li><li>Supported pattern type on the resource name. It could be one of <strong>literal</strong> or <strong>prefix</strong></li></ol><p>OAuth 2.0 and OPA provide policy-based control from an authorization server. Security policies and permissions used to grant access to resources on Kafka brokers are defined in the authorization server.</p><p>URLs are used to connect to the authorization server and verify that an operation requested by a client or user is allowed or denied. Users and clients are matched against the policies created in the authorization server that permit access to perform specific actions on Kafka brokers.</p><h3><strong>Accessing Kafka:</strong></h3><p>Now let us understand how can we restrict Kafka access only from the Kubernetes cluster and how what options are available if we want to access it from outside the cluster, but before that let’s understand the Kafka discovery protocol-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oJorvPNE0sEFDGyN9nzzuw.png" /><figcaption><a href="https://strimzi.io/blog/2019/04/17/accessing-kafka-part-1/">Source</a></figcaption></figure><p>Here is how the discovery protocol works-</p><p>1- An authenticated Kafka client connects to any of the broker during the first connection<br>2- The broker returns the metadata of all the topics/partitions etc.<br>3- After getting the details of desired partition, client opens up a new connection to that specific broker. Even if the client needs to connect to the first broker, it terminates the existing connection (#1) and starts a new one with that broker.</p><p>A Kubernetes service can be used for the initial connection only — it will take the client to one of the brokers within the cluster where it can get the metadata. But the subsequent connections cannot be done through that service because it would route the connection randomly to one of the brokers in the cluster instead of leading it to one particular broker. how does Strimzi deal with this?</p><blockquote>use the <strong>advertised.listeners</strong> option in the broker configuration in a way which allows the clients to connect directly to the broker</blockquote><h4><strong>Connecting from inside the same Kubernetes cluster:</strong></h4><p>Strimzi uses <strong>Statefulset </strong>to run the Kafka brokers. We can use the Kubernetes headless service to give each of the pods a stable DNS name. Strimzi is using these DNS names as the advertised addresses for the Kafka brokers. So with Strimzi:</p><ul><li>The initial connection is done using a regular Kubernetes service to get the metadata.</li><li>The subsequent connections are opened using the DNS names given to the pods by another headless Kubernetes service. The diagram below shows how does it look with an example Kafka cluster named my-cluster.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/929/1*7RHemsH11a-XsSSyG2KjAw.png" /><figcaption><a href="https://strimzi.io/blog/2019/04/17/accessing-kafka-part-1/">Source</a></figcaption></figure><p>Listeners configure how clients connect to a Kafka cluster. Multiple listeners can be configured by specifying unique name and port for each listener. Internal listeners (for accessing the cluster from inside of the Kubernetes) are specified using an internal type.</p><pre># ...<br>listeners:<br># ...<br>  - name: tls<br>    port: 9093<br>   <strong> type: internal</strong><br>    tls: true<br># ...</pre><h4>Connecting from outside the Kubernetes cluster:</h4><p>External Listener (for accessing the cluster from outside of the Kubernetes). TLS encryption can also be enabled for listeners.</p><p>For external types, the values for type can be one of the below:</p><p>• <strong>route -</strong> to use OpenShift routes and the default HAProxy router<br>• <strong>loadbalancer -</strong> to use loadbalancer services<br>• <strong>nodeport -</strong> to use ports on Kubernetes nodes (external access)<br>• <strong>ingress -</strong> to use Kubernetes Ingress and the NGINX Ingress Controller for Kubernetes.</p><p>Here is how we can configure it in Kafka-</p><pre># ...<br>listeners:<br># ...<br>  - name: external<br>    port: 9094<br>    type: <strong>route/nodeport/loadbalancer/ingress (any one of them)</strong><br>    tls: true<br># ...</pre><p>Following diagram depicts the services required, and communication flow for <strong>type = loadbalancer</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/873/1*105m3n-LCSuorETSG9Gh0Q.png" /><figcaption><a href="https://strimzi.io/blog/2019/05/13/accessing-kafka-part-4/">Source</a></figcaption></figure><p><strong>Conclusion:</strong> In this section, we have explored various options Strimzi provides to secure the Kafka cluster and the communication. We have seen how to restrict access only within the cluster (a typical use case would be, if only the internal services are accessing Kafka) or how can we extend it to external access securely. Strimzi provides some of the standard way of securing the things and also provides ways to add customize security settings as well.</p><p>In the next and last section of this series, we will discuss an important topic — Monitoring.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fc878178cd04" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/kafka-on-kubernetes-using-strimzi-part-5-security-fc878178cd04">Kafka on Kubernetes: Using Strimzi — Part 5(Security)</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kafka on Kubernetes: Using Strimzi — Part 4 (Scalability)]]></title>
            <link>https://blog.devgenius.io/kafka-on-kubernetes-using-strimzi-part-4-scalability-59da50575fec?source=rss-672c997a41cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/59da50575fec</guid>
            <category><![CDATA[kafkaonkubernetes]]></category>
            <category><![CDATA[strimzi]]></category>
            <category><![CDATA[keda]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Amarendra Singh]]></dc:creator>
            <pubDate>Mon, 09 May 2022 17:12:26 GMT</pubDate>
            <atom:updated>2022-05-31T07:19:52.591Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong><em>Kafka on Kubernetes: Using Strimzi — Part 4 (Scalability)</em></strong></h3><p>This is the Part-4 of the blog series, <strong>Kafka on Kubernetes: Using </strong><a href="https://strimzi.io/"><strong>Strimzi</strong></a><strong>. </strong>In <a href="https://medium.com/@singh.amarendra/kafka-on-kubernetes-using-strimzi-part-2-71a8ba8e9605">Part-2</a> we have discussed how to setup Kafka using Strimzi and in <a href="https://medium.com/@singh.amarendra/kafka-on-kubernetes-using-strimzi-part-3-configuration-options-f8aa027e9ba0">Part-3</a>, we’ve seen production ready Kafka configurations and their impact on the cluster performance. In this part we shall see how to auto-scale consumers for better throughput and resource usability.</p><p>First let’s try to understand what scalability is- <br><strong>Scalability</strong> — Let’s say our application has a hard limit to serve 1000 requests/sec, but on a busy day the number of requests/sec grew more than 1000. If our application still manages to serve those requests with the same efficiency, we can say our application is scalable.<br>Kafka provides a highly scalable system with Horizontal scaling, with an advantage of scaling infinitely.</p><p><a href="https://developer.confluent.io/learn-kafka/apache-kafka/partitions/">Partitions</a> are one way by which Kafka provides redundancy and scalability. Each partition can be hosted on a different server, which means that a single topic can be scaled horizontally across multiple servers to provide performance far beyond the ability of a single server.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3AMLMlKIPintn2n8foBOVQ.png" /><figcaption>Topic scalability using partitions</figcaption></figure><p>Okay so we have created topics with partitions equal to 100, now we are relaxed and we can get 100 times better throughput than with 1 partition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/353/1*LvBI8yYr_UrKSL98UypH4w.png" /><figcaption><a href="https://www.google.com/imgres?imgurl=https%3A%2F%2Fi.chzbgr.com%2Ffull%2F8489388288%2Fh9A761FAD%2Fahhhhh-what-a-relief&amp;imgrefurl=https%3A%2F%2Fcheezburger.com%2F8489388288%2Fahhhhh-what-a-relief&amp;tbnid=_G9c0Nrtzee2tM&amp;vet=12ahUKEwjZ2pLR3NL3AhW7gGMGHVhmCK8QMygAegUIARDAAQ..i&amp;docid=ZUlh3K6hdGjfXM&amp;w=500&amp;h=374&amp;q=what%20a%20relief%20funny%20image&amp;safe=active&amp;ved=2ahUKEwjZ2pLR3NL3AhW7gGMGHVhmCK8QMygAegUIARDAAQ">source</a></figcaption></figure><p>Well that’s not the case, even if we have multiple partitions available which can get consumed parallelly, we still need one thing and that is <a href="https://docs.confluent.io/platform/current/clients/consumer.html">the Consumers</a>. Before going ahead and discuss how to scale consumers, let’s first understand about Shared message queue and <a href="https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern">pub-sub systems</a>.</p><p><strong>Shared Message queue </strong>-A shared message queue system allows for a stream of messages from a producer to reach a single consumer. Once one consumer pulls a message, it is erased from the queue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*ZutErNyjpfdAmbjyCYlptQ.png" /></figure><p>Consider in above scenario, if message m1 gets consumed by consumer C1, then it will not be available for other consumer.</p><p><strong>Pub-Sub system</strong>-The publish-subscribe model allows for multiple publishers to publish messages to topics hosted by brokers which can be subscribed to by multiple subscribers. A message is thus broadcast to all the subscribers of a topic. Here the messages are available to be consumed by all the consumers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*j00Fh7OZYatW1SmLhcEYSQ.png" /></figure><p><strong>Kafka builds on the publish-subscribe model with the advantages of a message queuing system</strong>. It achieves this with-</p><ul><li>the use of consumer groups</li><li><a href="https://docs.confluent.io/platform/current/installation/configuration/topic-configs.html#topicconfigs_retention.ms">message retention</a> by brokers</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*dYFuRIIrvZElbQGpLQUjtw.png" /></figure><p>Consumer groups give Kafka the flexibility to have the advantages of both message queuing and publish-subscribe models.</p><ul><li>The consumers in a group divides the topic partitions as fairly amongst themselves as possible by establishing that each partition is only consumed by a single consumer from the group. If all consumers are from the same group, the Kafka model functions as a traditional message queue would.</li><li>When multiple consumer groups exist, the flow of the data consumption model aligns with the traditional publish-subscribe model. The messages are broadcast to all consumer groups.</li></ul><p>Now as we understood about consumers and consumer groups, let’s see how we can easily scale consumers based on the load using <a href="https://keda.sh/">KEDA</a> (Kubernetes event driven auto-scaling). KEDA provides various scaler which can easily scale-up to provide better throughput or scale-down to effectively use the available resources. It provides <a href="https://keda.sh/docs/1.4/scalers/apache-kafka/">Kafka scaler</a> to auto scale Kafka consumers.</p><p>Consider the scenario when there’s no auto scaling available, all’s good when the producer is sending fewer messages. But if it starts sending more messages the consumer will be overwhelmed and it’ll have lot of lag to cover. We can see in the below image that when producer started sending 100 messages/sec as compared to 10 messages/sec, consumer is not able to process the messages fast enough and it is lagging behind the producer service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/861/1*T2zGZ5Vgflh0J51_rXbTAg.png" /></figure><p>We can easily handle it using KEDA’s Kafka auto-scaler. We need to adjust the number of consumer instances to the traffic rate generated by the producer service. The value of offset lag can’t exceed the desired threshold. If we increase the traffic rate on the producer side KEDA should scale up the number of consumer instances. Consequently, if we decrease the producer traffic rate it should scale down the number of consumer instances</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*pDUpXH4hagRQWebX0LvpzQ.png" /></figure><p><strong>Steps to setup and configure Kafka auto-scaler in KEDA:</strong></p><p>We had already setup the Kafka cluster in <a href="https://medium.com/@singh.amarendra/kafka-on-kubernetes-using-strimzi-part-2-71a8ba8e9605">Part-2</a>, now let’s configure the KEDA operator and a simple consumer for verifying the auto-scaling.</p><p>1- Create keda namespace:</p><p>kubectl create namespace keda</p><p>2- Deploy keda operator:</p><pre>a. helm repo add kedacore https://kedacore.github.io/charts<br>b. helm repo update<br>c. helm install keda kedacore/keda --namespace keda</pre><p>3- Deploy Consumer application:</p><pre>a. kubectl apply -f consumer-service.yml -n keda<br>b. kubectl apply -f consumer-deployment.yml -n keda</pre><p>4- Deploy Keda scaled object:<br>kubectl apply -f keda.yaml -n keda</p><p>Here is the sample keda.yaml with Kafka scaler. It has lag threshold of 10, so Keda will spawn a new consumer when the message lag reaches to 10.</p><pre>apiVersion: keda.sh/v1alpha1                 <br>kind: ScaledObject<br>metadata:                                   <br>  name: consumer-scaler<br>  namespace: keda<br>spec:<br>  scaleTargetRef:<br>    name: consumer                  <br>  pollingInterval: 1            <br>  cooldownPeriod:  10         <br>  minReplicaCount: 0             <br>  maxReplicaCount: 10<br>  triggers:<br>    - type: kafka<br>      metadata:<br>        topic: my-topic<br>        bootstrapServers: kafka-kafka-bootstrap.kafka:9092<br>        consumerGroup: keda-consumer<br>        <strong>lagThreshold: &#39;10&#39;</strong><br>        offsetResetPolicy: earliest</pre><p>5- Run Kafka Producer:<br>kubectl -n kafka run kafka-producer -ti --image=quay.io/strimzi/kafka:0.28.0-kafka-3.1.0 --rm=true --restart=Never -- bin/kafka-console-producer.sh --bootstrap-server kafka-kafka-bootstrap:9092 --topic my-topic</p><p>6- Produce some messages and see if the consumer pods are getting scaled</p><p>You can find the required resources for these steps <a href="https://github.com/AmarendraSingh88/kafka-on-kubernetes">here</a> on GitHub. You can use this repository for setting up and verifying the concepts of this <strong>KafkaOnKubernetes</strong> blog series.</p><p><strong>Conclusion</strong> — This part covers how can we easily scale the consumer groups to get the desired throughput. Obviously we need to have the same or more number of partitions for better throughput, as, we can only attach one consumer (in the same consumer group)to one partition. So if we have more consumers than the number of partitions, then those consumers will sit idle.</p><p>In the next section, we shall see how to secure the Kafka cluster using Strimzi.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=59da50575fec" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/kafka-on-kubernetes-using-strimzi-part-4-scalability-59da50575fec">Kafka on Kubernetes: Using Strimzi — Part 4 (Scalability)</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kafka on Kubernetes: Using Strimzi — Part 3(Configuration Options)]]></title>
            <link>https://blog.devgenius.io/kafka-on-kubernetes-using-strimzi-part-3-configuration-options-f8aa027e9ba0?source=rss-672c997a41cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/f8aa027e9ba0</guid>
            <category><![CDATA[strimzi]]></category>
            <category><![CDATA[kafkaonkubernetes]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[configuration]]></category>
            <dc:creator><![CDATA[Amarendra Singh]]></dc:creator>
            <pubDate>Sat, 30 Apr 2022 14:04:23 GMT</pubDate>
            <atom:updated>2022-05-31T07:19:51.285Z</atom:updated>
            <content:encoded><![CDATA[<h3>Kafka on Kubernetes: Using Strimzi — Part 3(Production Ready Configurations)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RbwGmA5Djpfn2STgzP7RCQ.png" /></figure><p>This is the Part-3 of the series <strong>Kafka on Kubernetes: Using </strong><a href="https://strimzi.io/"><strong>Strimzi</strong></a><strong>. </strong>In the <a href="https://medium.com/@singh.amarendra/kafka-on-kubernetes-using-strimzi-part-2-71a8ba8e9605">previous part</a>, we’ve setup and deployed the Strimzi operator and Kafka cluster on Kubernetes. In this article, we shall discuss about tuning some of the configurations, using the correct instance type for the brokers, proper distribution of resources on worker nodes, scalability using optimum partition count, availability using replication, security configs etc.</p><p>Let’s discuss the following important things when running the Kafka cluster in production environment-</p><ol><li><strong>Number of Brokers and Zookeepers:<br>Zookeeper Servers</strong> — One of the main purposes of Zookeeper in distributed systems is to do leader election in case of any broker failure and It needs to have a strict majority when the vote happens.<br>Therefore Zookeeper quorum can have 1,3,5,7.. (2N+1) servers which allows 0, 1, 2, 3.. N servers to go down without making the whole cluster unusable.<br>Let’s see the Pros and Cons of using 1 and 3 Zookeeper servers in a quorum using the below diagram-</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vPe7d-P0chGp6EM8ek5ijQ.png" /></figure><blockquote>So, it is advisable to use at-least 3 Zookeeper servers in a Quorum for small Kafka clusters in production environments and go with 5 or 7 Zookeeper servers for medium to very large Kafka clusters.</blockquote><p><strong>Kafka Brokers </strong>— There are many things to consider while deciding on the number of brokers-</p><ul><li><em>replication-factor</em>: For making the system highly available, it is recommended that we have at-least 3 replicas of each partition so that even in case of a broker failure, we still can get the data from other two replicas.</li><li><em>min.insysnc.replica</em>: If our Kafka cluster has 3 brokers and and all replicas are in sync, the min-ISR equals to 2. With this configuration, we accept the risk of losing 1 replica (either leader or follower) while working. For instance, if we lose the leader, there has at least one follower that in-sync for switching. If we lose one of the followers, we still have a remaining follower to keep the min-ISR to 2.</li><li>Also in the newer version of Kafka (2.8.0 onwards), the <a href="https://www.confluent.io/blog/kafka-without-zookeeper-a-sneak-peek/">dependency on Zookeeper has been removed</a> and Kafka brokers itself will participate in the leader election and hence we should have the odd number of brokers to elect the new leader.</li></ul><blockquote>You can setup a Kafka HA Cluster with only 2 brokers, but the recommended <strong>replication-factor</strong> for production is <strong>3</strong>, so you <strong>need 3 brokers</strong> in order to achieve this. If you have higher number of Topics/Partitions available in the cluster, you can opt for more number of brokers.</blockquote><p><strong>2. Instance Types (in AWS):</strong><br>The choice of <a href="https://aws.amazon.com/ec2/instance-types/">instance types</a> is generally driven by the type of storage required for your streaming applications on a Kafka cluster. If your application requires ephemeral storage, h1, i3, and d2 instances are your best option.<br>If you need EBS storage, then AWS has a newer-generation r4 instance.</p><p><em>Note</em>: It’s always best practice to check for <a href="https://aws.amazon.com/ec2/instance-types/">the latest changes in instance types</a>.</p><p>We should create a separate<a href="https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html"> worker node group</a> for Kafka resources. This way we can define the optimized Instance type and use <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/">node affinity</a> to deploy the Kafka resources on these nodes.<br>We can also define <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/">anti-affinity</a> to make sure Kafka brokers and Zookeeper nodes run on separate nodes.</p><p><strong>3. Zookeeper Performance:</strong><br>Latency is the key for Zookeeper performance and any of the below variables can affect it-</p><ul><li>Fast disk (Preferable SSD)</li><li>Separate disk for snapshots and logs</li><li>High performance network (low latency)</li><li>Reasonable number of Zookeeper servers</li><li>Isolation of Zookeeper instances from other processes</li></ul><p><strong>4. Setting Requests resources along with JVM memory:<br></strong>We should add<a href="https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits"> request resources</a> like CPU and Memory to Kafka resources like Broker, Zookeeper, KafkaConnect etc. Requests for reservation of <a href="https://strimzi.io/docs/operators/latest/configuring.html#con-common-configuration-resources-reference">supported resources</a>, currently cpu and memory, and limits to specify the maximum resources that can be consumed.</p><p>We can set it like this-</p><blockquote>resources:<br>requests:<br> memory: 32Gi<br> cpu: “6”<br>limits:<br> memory: 32Gi<br> cpu: “8”</blockquote><p>My take would be to set only the requests parameters for Kafka broker and Zookeeper and not limits, this will let them use the available resources on the machine. Check a better explanation of not using the CPU limits <a href="https://home.robusta.dev/blog/stop-using-cpu-limits/">here</a>.</p><p>As Kafka is a Java application, we should set JVM options for maximum and minimum memory allocation. We can set it in the Kafka resource-</p><blockquote>jvmOptions: <br> -Xms: 8192m <br> -Xmx: 8192m</blockquote><p><strong>5. Volume sizing and retention period:<br></strong>We can define the storage options for Kafka and choose any of the below types for Kafka storage-</p><p><strong><em>a. JBOD(just a bunch of devices)</em></strong>- JBOD storage allows you to use multiple disks in each Kafka broker for storing commit logs. We can add it like this in the Kafka resource-</p><blockquote>storage: <br> type: jbod <br> volumes: <br> - id: 0 <br> type: persistent-claim <br> size: 1000Gi <br> deleteClaim: false <br> - id: 1 <br> type: persistent-claim <br> size: 1000Gi <br> deleteClaim: false</blockquote><p>There are some things you should keep in mind and take care with:</p><ul><li>The id numbers have to be unique.</li><li>The id numbers do not have to be in sequence.</li><li>Changing the id number is the same as removing the volume with the original id, and adding a new volume with the new id. Strimzi will stop using the old PVC with the old volume and create a new PVC for the new volume.</li><li>By default the PVCs are not deleted. So if you reuse an id which you already used in the past you should check first whether the old PVC still exists or not. If it exists, it will be reused instead of creating a new PVC with a new volume.</li></ul><p><strong><em>b. persistent-claim- </em></strong>We can also use the storage type as persistent-claim, in that case will have to add the total size in one Volume itself and we won’t be able to get the benefit of using multiple persistent-claim type as in JBOD.</p><blockquote>storage: <br> type: persistent-claim <br> size: 10000Gi</blockquote><p>The<a href="https://www.baeldung.com/kafka-message-retention#:~:text=Basics&amp;text=We%20can%20notice%20here%20that%20the%20default%20retention%20time%20is%20seven%20days."> retention-period</a> also impacts deriving the storage size as, in case of high retention time, we’ll need more storage and in case of lower retention, we can live with smaller storage size as well.</p><p><strong>6. High availability:<br></strong>Kafka provides the following different ways to achieve high durability and availability of data-</p><ul><li><em>Using Replication</em> — With replication, we can ensure that even if a broker goes down, we can ensure that all partitions of a topic remain available without sacrificing consistency.</li></ul><blockquote>config:<br> default.replication.factor: 3</blockquote><ul><li>Using <a href="https://banzaicloud.com/blog/kafka-rack-awareness">Rack Awareness</a> — With replication, we can ensure that broker level failures do not impact the data availability. We still need to be concerned about the possibility of an entire data center/availability zone outage. Kafka makes it easy to handle data center outages using a feature called Rack Awareness. A rack can represent an availability zone, data center, or an actual rack in the data center.</li></ul><blockquote>broker.rack=rack-id</blockquote><ul><li>Disaster recovery — Disaster recovery plans are built carefully to minimize the business impact of natural and man-made disasters such as power outages, catastrophic software failures, and network outages. We can use <a href="https://docs.confluent.io/4.0.0/multi-dc/mirrormaker.html">Kafka MirrorMaker</a> to mirror/copy the whole cluster to a geographically distributed data centers.</li></ul><p>I have also written a blog to discuss the <a href="https://medium.com/@singh.amarendra/understanding-message-durability-in-kafka-8f6e7006aea8">high availability and durability in Kafka</a>. You can check it to get more details.</p><p><strong>7. Scalability using optimum partition count:<br></strong>As we know, Topics are divided into partitions. We can have the level of parallelism and write data in parallel, that will automatically increase the throughput. However, if the system configuration is not capable to handle it then increasing the number of the partition can slow down our performance and throughput. Basically, if a system does not have sufficient threads or just have single disk then it does not make sense in creating lots of partition for better throughput. So, we can say, the creation of more partition for a topic is directly dependent on available threads and disk.<br> It is recommended that starting with one partition per physical storage disk and one consumer per partition.</p><p>We will discuss the scalability in more detail in the next section.</p><p><strong>8. Security configs:</strong><br>Strimzi provides lot of options to configure security in the cluster. With Strimzi, Communication is always encrypted between:</p><ul><li>Kafka brokers</li><li>Zookeeper nodes</li><li>Operators and Kafka brokers</li><li>Operators and Zookeeper nodes</li><li>Kafka Exporter</li></ul><p>Supported authentication mechanisms:</p><ul><li>Mutual TLS client authentication (on listeners with TLS enabled encryption)</li><li>SASL SCRAM-SHA-512</li><li>OAuth 2.0 token based authentication</li><li>Custom authentication</li></ul><p>Kafka clusters use authorization to control the operations that are permitted on Kafka brokers by specific clients or users. Supported authorization mechanisms:</p><ul><li>Simple authorization</li><li>OAuth 2.0 authorization (if you are using OAuth 2.0 tokenbased</li><li>authentication)</li><li>Open Policy Agent (OPA) authorization</li><li>Custom authorization</li></ul><p>We shall discuss these options in detail in the Part-5 of this series.</p><p><strong>9. Other Configurations:</strong></p><p><strong>a. open file handlers:</strong><br> It’s a scenario that occurs too often: brokers go down from what appears to be too much load, but in reality is a benign (though nonetheless stressful) “too many open files” error. By editing /etc/sysctl.conf and configuring Ulimit to allow 128,000 or more open files, you can avoid this error from happening.</p><p>An example to increase the ulimit on CentOS:</p><ol><li>Create a new file /etc/security/limits.d/nofile.conf</li><li>Enter contents:</li></ol><p>* soft nofile 128000</p><p>* hard nofile 128000</p><ol><li>Restart the system or relogin.</li><li>Verify by issuing below command.</li></ol><p>ulimit -a</p><p>*Note that there are various methods to increase ulimit. You can follow any suitable method for your own Linux distribution.</p><p><strong>b. Maximum message size:<br> </strong>Kafka configuration limits the size of messages that it’s allowed to send. By default, this <strong>limit is 1MB</strong>. However, if there’s a requirement to send large messages, we need to tweak these configurations as per our requirements.</p><p>Data flows from Producer to Topic in Broker to Consumers, so will have to add the configuration at these places to send large messages.</p><ul><li>Producer property — <strong><em>max.request.size</em></strong></li><li>Topic Property — <strong><em>max.message.bytes.</em></strong><em> We can also set </em><strong><em>message.max.bytes</em></strong><em> property at broker level to </em>allow all topics on a <em>Broker</em> to accept messages of greater than 1MB in size</li><li>Consumer Property — Let’s look into the configuration settings available for a Kafka consumer. Although these changes aren’t mandatory for consuming large messages, avoiding them can have a performance impact on the consumer application.</li></ul><p><em>max.partition.fetch.bytes</em>: This property limits the number of bytes a consumer can fetch from a Topic’s partition.</p><p><em>fetch.max.bytes</em>: This property limits the number of bytes a consumer can fetch from the Kafka server itself. A Kafka consumer can listen on multiple partitions as well.</p><p><strong>c. compression.type:</strong><br> Kafka lets you compress your messages as they travel over the wire. This compression process lets you achieve two things:</p><ul><li>Reducing network bandwidth usage.</li><li>Saving disk space on Kafka brokers.</li></ul><p>The only tradeoff with these two benefits of compression is slightly higher CPU utilization.</p><p>Kafka supports four primary types of compression:</p><ul><li>Gzip</li><li>Snappy</li><li>Lz4</li><li>Zstd</li></ul><p>Let’s take a look into characteristics of these compression types:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*0tK9AxW7HXGMJigO8sfZOw.png" /><figcaption><a href="https://developer.ibm.com/articles/benefits-compression-kafka-messaging/">source</a></figcaption></figure><p>Kafka supports compression via property compression.type. The default value is none, which means messages are sent un-compressed. Otherwise, you specify the supported types: gzip, snappy, lz4, or zstd.</p><p><strong>Conclusion:</strong> Kafka has a lot of configuration options and not utilizing them effectively can hamper the efficiency and performance of your Kafka cluster. In this article we have seen the various options to setup correctly in Production Kafka clusters. Find the list of <a href="https://kafka.apache.org/documentation/#configuration">Kafka configurations </a>and tune them based on your requirement.<br>In the next section, we will discuss the how Kafka achieves the scalability and will configure consumer auto scalability using <a href="https://keda.sh/">KEDA</a>(Kubernetes event driven auto scaler).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f8aa027e9ba0" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/kafka-on-kubernetes-using-strimzi-part-3-configuration-options-f8aa027e9ba0">Kafka on Kubernetes: Using Strimzi — Part 3(Configuration Options)</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kafka on Kubernetes: Using Strimzi — Part 2]]></title>
            <link>https://blog.devgenius.io/kafka-on-kubernetes-using-strimzi-part-2-71a8ba8e9605?source=rss-672c997a41cf------2</link>
            <guid isPermaLink="false">https://medium.com/p/71a8ba8e9605</guid>
            <category><![CDATA[strimzi]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[setup]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[helm]]></category>
            <dc:creator><![CDATA[Amarendra Singh]]></dc:creator>
            <pubDate>Sun, 24 Apr 2022 12:39:32 GMT</pubDate>
            <atom:updated>2022-05-31T07:19:46.641Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong><em>Kafka on Kubernetes: Using Strimzi — Part 2 (Setup and Deployment)</em></strong></h3><p>This is the 2nd part of the series <strong>Kafka on Kubernetes: Using </strong><a href="https://strimzi.io/"><strong>Strimzi</strong></a><strong>. </strong>The <a href="https://medium.com/@singh.amarendra/kafka-on-kubernetes-using-strimzi-part-1-83d74564135e">Part-1</a> of the series discusses what Strimzi offers to deploy and maintain Kafka on Kubernetes. In this part we’ll see how to deploy and setup Kafka on Kubernetes using Strimzi.</p><p><strong>Kafka Deployment using Strimzi:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*Jzt3RGzVhCVSP8hkNbXP3Q.png" /><figcaption>Kafka quick start on Kubernetes using Strimzi</figcaption></figure><p>Here are the steps to deploy the Strimzi operator and Kafka Cluster on Kubernetes-</p><ol><li><strong>Deploy Strimzi Operator- </strong>The Strimzi helm chart contains all the necessary <a href="https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/">CRDs</a> (as shown in the above image, like Kafka, KafkaConnect etc.) which shall use to easily deploy and maintain the Kafka cluster.</li></ol><ul><li>First add the Strimzi Chart Repository</li></ul><pre>helm repo <strong>add</strong> strimzi https://strimzi.io/charts/</pre><ul><li>Then you can install a version of the Strimzi Cluster Operator</li></ul><pre>helm <strong>install</strong> strimzi/strimzi-kafka-<strong>operator</strong> \ --name my-strimzi-release \ --namespace kafka \ --version 0.28.0</pre><p>We can also use terraform’s <a href="https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release">helm_release</a> to easily deploy the Strimzi Operator.</p><pre>resource &quot;helm_release&quot; &quot;strimzi&quot; {<br>  chart = &quot;strimzi-kafka-operator&quot;<br>  name = &quot;strimzi&quot;<br>  namespace = &quot;kafka&quot;<br>  repository = &quot;<a href="https://strimzi.io/charts/">https://strimzi.io/charts/</a>&quot;<br>  version = &quot;0.28.0&quot;<br>  values = [yamlencode({<br> resources = {<br>   requests = {<br>     memory = &quot;512Mi&quot;<br>     cpu = &quot;250m&quot;<br>   }<br>   limits = {<br>     memory = &quot;1536Mi&quot;<br>     cpu = &quot;1000m&quot;<br>   }<br>  }<br> })]<br>}</pre><p>The Strimzi cluster operator reconciles the various custom resources like Kafka, KafkaConnect etc. in specific or all namespaces. If any custom resource gets added/updated/deleted in the namespace, Strimzi Cluster Operator will perform the same action on it in the cluster.</p><p><strong>2. Deploy Kafka Custom resource</strong>- As we discussed in the <a href="https://medium.com/@singh.amarendra/kafka-on-kubernetes-using-strimzi-part-1-83d74564135e">part-1</a>, Strimzi Cluster operator can manage multiple Kafka Clusters. Here as part of the 2nd step, we will add a Kafka <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/">Custom Resource(CR)</a>. The Kafka CR can deploy following components-</p><ul><li>Kafka</li><li>Zookeeper</li><li>Topic Operator</li><li>User Operator</li><li>Kafka Exporter</li><li>Cruise Control</li></ul><p>For this part, we’ll just deploy the Kafka, Zookeeper and entity operator(Topic and User operators).</p><pre>apiVersion: kafka.strimzi.io/v1beta2<br>kind: Kafka<br>metadata:<br>  name: my-cluster<br>spec:<br>  kafka:<br>    replicas: 3<br>    listeners:<br>      - name: plain<br>        port: 9092<br>        type: internal<br>        tls: false<br>    storage:<br>      type: jbod<br>      volumes:<br>      - id: 0<br>        type: persistent-claim<br>        size: 100Gi<br>        deleteClaim: false<br>    config:<br>      offsets.topic.replication.factor: 1<br>      transaction.state.log.replication.factor: 1<br>      transaction.state.log.min.isr: 1<br>      default.replication.factor: 3<br>      min.insync.replicas: 2<br>  zookeeper:<br>    replicas: 3<br>    storage:<br>      type: persistent-claim<br>      size: 100Gi<br>      deleteClaim: false<br>  entityOperator:<br>    topicOperator: {}<br>    userOperator: {}</pre><p>Let’s save this as<em> kafka-resource.yaml</em>. As we can see in above configuration, we can specify the following things-</p><p>For Kafka:</p><ul><li>The number of brokers</li><li>Listeners (internal or external, more on this in Part-5)</li><li>Storage (<a href="https://strimzi.io/blog/2019/03/07/strimzi-0.11.0-release/">JBOD</a> (only for Kafka) or persistent-claim)</li><li>Override the default Kafka configurations like replication factor etc.</li><li>More configurations in the following parts of the series</li></ul><p>For Zookeeper:</p><ul><li>Number of Zookeeper server</li><li>Storage</li><li>More configurations in the following parts of the series</li></ul><p>For EntityOperator:</p><ul><li>topicOperator- To deploy the topic operator</li><li>userOperator- To deploy the user operator</li></ul><p>Now just deploy this file-</p><pre>&gt; kubectl apply -f kafka-resource.yaml -n kafka</pre><p>We shall see more configurations in these sections and other sections like KafkaExporter, CruiseControl etc. in following parts. We can also deploy other custom resources like KafkaConnect, KafkaConnector, MirrorMaker etc., if required.</p><p><strong>3. Create a Topic- </strong>Now the cluster(my-cluster) is ready and we can use it to flow the data from Producer to Consumer. Let’s first create a simple topic with 3 partitions and 1 replica -</p><pre>cat &lt;&lt; EOF | kubectl create -n my-kafka-project -f -<br>apiVersion: kafka.strimzi.io/v1beta2<br>kind: KafkaTopic<br>metadata:<br>  name: my-topic<br>  labels:<br>    strimzi.io/cluster: &quot;my-cluster&quot;<br>spec:<br>  partitions: 3<br>  replicas: 1<br>EOF</pre><p><strong>4. Start Console Producer- </strong>Now we can start the <a href="http://cloudurable.com/blog/kafka-tutorial-kafka-from-command-line/index.html#:~:text=Run%20Kafka%20Producer%20Console&amp;text=Kafka%20provides%20the%20utility%20kafka,console.sh%20and%20run%20it.">console producer</a> from inside the Kafka broker pod(as console-producer comes with Kafka binary) and send data to the topic created in previous step. We can also install the Kafka on our local machine and connect to this cluster (but we’ll have to add the external listener like loadBalancer or nodePort, we’ll cover this in Part-5 while discussing the Kafka security).</p><pre>&gt;kubectl exec -i -t my-cluster-kafka-0 -- /bin/bash</pre><p>Now let’s start the console producer-</p><pre>&gt; bin/kafka-console-producer.sh --bootstrap-server <em>localhost:9092</em> --topic my-topic<br>Hello<br>My name is Amarendra</pre><p><strong>5. Start the consumer- </strong>We can now start the consumer and assign it to listen to the topic created above. We should be getting the produced messages-</p><pre>&gt;bin/kafka-console-consumer.sh --bootstrap-server --group consumer-group-a <em>localhost:9092</em> --topic my-topic --from-beginning<br>Hello<br>My name is Amarendra</pre><p>We’ve got the produced messages in the consumer group. Any other consumer in the same consumer group won’t be able to get these messages again (It’s like shared message queue behavior of Kafka). Now if some other consumer also wants these messages, it still can get that by using different group (It’s like the pub-sub mechanism). Consumer groups give Kafka the flexibility to have the advantages of both message queuing and publish-subscribe models.</p><pre>&gt;bin/kafka-console-consumer.sh --bootstrap-server --group consumer-group-b <em>localhost:9092</em> --topic my-topic --from-beginning<br>Hello<br>My name is Amarendra</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/1*m_URulS_7Zli2dRmVognuw.png" /></figure><p><strong>Conclusion-</strong> We have seen how to deploy the Strimzi Cluster operator and how it can easily deploy and manage a Kafka cluster. We’ve also created a topic and flown data to it using Kafka console producer and consume it using Kafka console consumer.</p><p>In the next part we shall see the other important configurations to deploy the highly available Kafka cluster on Kubernetes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=71a8ba8e9605" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/kafka-on-kubernetes-using-strimzi-part-2-71a8ba8e9605">Kafka on Kubernetes: Using Strimzi — Part 2</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>