<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Varun Pal on Medium]]></title>
        <description><![CDATA[Stories by Varun Pal on Medium]]></description>
        <link>https://medium.com/@varunkumarpal?source=rss-3a721f06cfe7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*9htQl7D2CFOXmKYI.jpg</url>
            <title>Stories by Varun Pal on Medium</title>
            <link>https://medium.com/@varunkumarpal?source=rss-3a721f06cfe7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 02:07:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@varunkumarpal/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Kafka: Monitoring your MirrorMaker with Prometheus]]></title>
            <link>https://varunkumarpal.medium.com/kafka-monitoring-your-mirrormaker-with-prometheus-fe10d6db4d2d?source=rss-3a721f06cfe7------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe10d6db4d2d</guid>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[mirrormaker]]></category>
            <dc:creator><![CDATA[Varun Pal]]></dc:creator>
            <pubDate>Thu, 17 Dec 2020 02:21:03 GMT</pubDate>
            <atom:updated>2020-12-17T02:21:03.671Z</atom:updated>
            <content:encoded><![CDATA[<p>With the launch of MirrorMaker 2 (MM2), kafka can be used at a global scale while increasing the resiliency by allowing data to be duplicated across clusters globally.</p><p>While MM2 is majorly being used for cloud migrations, where organizations are moving from on-prem installations to the cloud, it is exceedingly growing where the data is produced and consumed in geographically distributed locations. Replication of data between clusters ensures minimal latency and optimal throughput.</p><p>Prometheus is a popular free, open-source metric instrumentation, collection and monitoring tool and can be setup easily on a client machine where MM2 is also installed.</p><p>Following steps illustrate how Prometheus can be used to monitor MM2:</p><ol><li>Download the prometheus jar file</li></ol><pre>cd /home/ec2-user<br>wget <a href="https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.13.0/jmx_prometheus_javaagent-0.13.0.jar">https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.13.0/jmx_prometheus_javaagent-0.13.0.jar</a></pre><p>2. Create a kafka-connect.yml with the following contents</p><pre>lowercaseOutputName: true<br>rules:<br>  #kafka.connect:type=app-info,client-id=&quot;{clientid}&quot;<br>  #kafka.consumer:type=app-info,client-id=&quot;{clientid}&quot;<br>  #kafka.producer:type=app-info,client-id=&quot;{clientid}&quot;<br>  - pattern: &#39;kafka.(.+)&lt;type=app-info, client-id=(.+)&gt;&lt;&gt;start-time-ms&#39;<br>    name: kafka_$1_start_time_seconds<br>    labels:<br>      clientId: &quot;$2&quot;<br>    help: &quot;Kafka $1 JMX metric start time seconds&quot;<br>    type: GAUGE<br>    valueFactor: 0.001<br>  - pattern: &#39;kafka.(.+)&lt;type=app-info, client-id=(.+)&gt;&lt;&gt;(commit-id|version): (.+)&#39;<br>    name: kafka_$1_$3_info<br>    value: 1<br>    labels:<br>      clientId: &quot;$2&quot;<br>      $3: &quot;$4&quot;<br>    help: &quot;Kafka $1 JMX metric info version and commit-id&quot;<br>    type: GAUGE</pre><pre>#kafka.producer:type=producer-topic-metrics,client-id=&quot;{clientid}&quot;,topic=&quot;{topic}&quot;&quot;, partition=&quot;{partition}&quot;<br>  #kafka.consumer:type=consumer-fetch-manager-metrics,client-id=&quot;{clientid}&quot;,topic=&quot;{topic}&quot;&quot;, partition=&quot;{partition}&quot;<br>  - pattern: kafka.(.+)&lt;type=(.+)-metrics, client-id=(.+), topic=(.+), partition=(.+)&gt;&lt;&gt;(.+-total|compression-rate|.+-avg|.+-replica|.+-lag|.+-lead)<br>    name: kafka_$2_$6<br>    labels:<br>      clientId: &quot;$3&quot;<br>      topic: &quot;$4&quot;<br>      partition: &quot;$5&quot;<br>    help: &quot;Kafka $1 JMX metric type $2&quot;<br>    type: GAUGE</pre><pre>#kafka.producer:type=producer-topic-metrics,client-id=&quot;{clientid}&quot;,topic=&quot;{topic}&quot;<br>  #kafka.consumer:type=consumer-fetch-manager-metrics,client-id=&quot;{clientid}&quot;,topic=&quot;{topic}&quot;&quot;, partition=&quot;{partition}&quot;<br>  - pattern: kafka.(.+)&lt;type=(.+)-metrics, client-id=(.+), topic=(.+)&gt;&lt;&gt;(.+-total|compression-rate|.+-avg)<br>    name: kafka_$2_$5<br>    labels:<br>      clientId: &quot;$3&quot;<br>      topic: &quot;$4&quot;<br>    help: &quot;Kafka $1 JMX metric type $2&quot;<br>    type: GAUGE</pre><pre>#kafka.connect:type=connect-node-metrics,client-id=&quot;{clientid}&quot;,node-id=&quot;{nodeid}&quot;<br>  #kafka.consumer:type=consumer-node-metrics,client-id=consumer-1,node-id=&quot;{nodeid}&quot;<br>  - pattern: kafka.(.+)&lt;type=(.+)-metrics, client-id=(.+), node-id=(.+)&gt;&lt;&gt;(.+-total|.+-avg)<br>    name: kafka_$2_$5<br>    labels:<br>      clientId: &quot;$3&quot;<br>      nodeId: &quot;$4&quot;<br>    help: &quot;Kafka $1 JMX metric type $2&quot;<br>    type: UNTYPED</pre><pre>#kafka.connect:type=kafka-metrics-count,client-id=&quot;{clientid}&quot;<br>  #kafka.consumer:type=consumer-fetch-manager-metrics,client-id=&quot;{clientid}&quot;<br>  #kafka.consumer:type=consumer-coordinator-metrics,client-id=&quot;{clientid}&quot;<br>  #kafka.consumer:type=consumer-metrics,client-id=&quot;{clientid}&quot;<br>  - pattern: kafka.(.+)&lt;type=(.+)-metrics, client-id=(.*)&gt;&lt;&gt;(.+-total|.+-avg|.+-bytes|.+-count|.+-ratio|.+-age|.+-flight|.+-threads|.+-connectors|.+-tasks|.+-ago)<br>    name: kafka_$2_$4<br>    labels:<br>      clientId: &quot;$3&quot;<br>    help: &quot;Kafka $1 JMX metric type $2&quot;<br>    type: GAUGE</pre><pre>#kafka.connect:type=connector-task-metrics,connector=&quot;{connector}&quot;,task=&quot;{task}&lt;&gt; status&quot;<br>  - pattern: &#39;kafka.connect&lt;type=connector-task-metrics, connector=(.+), task=(.+)&gt;&lt;&gt;status: ([a-z-]+)&#39;<br>    name: kafka_connect_connector_status<br>    value: 1<br>    labels:<br>      connector: &quot;$1&quot;<br>      task: &quot;$2&quot;<br>      status: &quot;$3&quot;<br>    help: &quot;Kafka Connect JMX Connector status&quot;<br>    type: GAUGE</pre><pre>#kafka.connect:type=task-error-metrics,connector=&quot;{connector}&quot;,task=&quot;{task}&quot;<br>  #kafka.connect:type=source-task-metrics,connector=&quot;{connector}&quot;,task=&quot;{task}&quot;<br>  #kafka.connect:type=sink-task-metrics,connector=&quot;{connector}&quot;,task=&quot;{task}&quot;<br>  #kafka.connect:type=connector-task-metrics,connector=&quot;{connector}&quot;,task=&quot;{task}&quot;<br>  - pattern: kafka.connect&lt;type=(.+)-metrics, connector=(.+), task=(.+)&gt;&lt;&gt;(.+-total|.+-count|.+-ms|.+-ratio|.+-avg|.+-failures|.+-requests|.+-timestamp|.+-logged|.+-errors|.+-retries|.+-skipped)<br>    name: kafka_connect_$1_$4<br>    labels:<br>      connector: &quot;$2&quot;<br>      task: &quot;$3&quot;<br>    help: &quot;Kafka Connect JMX metric type $1&quot;<br>    type: GAUGE</pre><pre>#kafka.connect:type=connector-metrics,connector=&quot;{connector}&quot;<br>  #kafka.connect:type=connect-worker-metrics,connector=&quot;{connector}&quot;<br>  - pattern: kafka.connect&lt;type=connect-worker-metrics, connector=(.+)&gt;&lt;&gt;([a-z-]+)<br>    name: kafka_connect_worker_$2<br>    labels:<br>      connector: &quot;$1&quot;<br>    help: &quot;Kafka Connect JMX metric $1&quot;<br>    type: GAUGE</pre><pre>#kafka.connect:type=connect-worker-metrics<br>  - pattern: kafka.connect&lt;type=connect-worker-metrics&gt;&lt;&gt;([a-z-]+)<br>    name: kafka_connect_worker_$1<br>    help: &quot;Kafka Connect JMX metric worker&quot;<br>    type: GAUGE</pre><pre>#kafka.connect:type=connect-worker-rebalance-metrics<br>  - pattern: kafka.connect&lt;type=connect-worker-rebalance-metrics&gt;&lt;&gt;([a-z-]+)<br>    name: kafka_connect_worker_rebalance_$1<br>    help: &quot;Kafka Connect JMX metric rebalance information&quot;<br>    type: GAUGE</pre><pre>#kafka.connect.mirror:type=MirrorSourceConnector<br>  - pattern: kafka.connect.mirror&lt;type=MirrorSourceConnector, target=(.+), topic=(.+), partition=([0-9]+)&gt;&lt;&gt;([a-z-]+)<br>    name: kafka_connect_mirror_source_connector_$4<br>    help: Kafka Connect MM2 Source Connector Information<br>    labels:<br>      destination: &quot;$1&quot;<br>      topic: &quot;$2&quot;<br>      partition: &quot;$3&quot;<br>    type: GAUGE</pre><pre>#kafka.connect.mirror:type=MirrorCheckpointConnector<br>  - pattern: kafka.connect.mirror&lt;type=MirrorCheckpointConnector, source=(.+), target=(.+)&gt;&lt;&gt;([a-z-]+)<br>    name: kafka_connect_mirror_checkpoint_connector_$3<br>    help: Kafka Connect MM2 Checkpoint Connector Information<br>    labels:<br>      source: &quot;$1&quot;<br>      target: &quot;$2&quot;<br>    type: GAUGE</pre><p>3. Configure the environment variable KAFKA_OPTS to the above kafka_connect.yml and the jar file</p><pre>export KAFKA_OPTS=-javaagent:/root/prometheus/jmx_prometheus_javaagent-0.13.0.jar=3600:/root/prometheus/kafka-connect.yml</pre><p>4. Start MM2 in the same terminal</p><pre>$KAFKA/connect-mirror-maker.sh $CONFIG/connect-mirror-maker.properties</pre><p>where $KAFKA points to /root/kafka/bin and $CONFIG points to /root/kafka/config locations</p><p>5. You can now verify the latency via</p><pre>curl localhost:3600 | grep -i kafka_connect_mirror_source_connector_replication_latency_ms_max</pre><p>When MM2 is in sync the output will be as follows:</p><pre>% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current<br> Dload  Upload   Total   Spent    Left  Speed<br> 100  230k  100  230k    0     0   197k      0  0:00:01  0:00:01 --:--:--  197k</pre><pre>#HELP kafka_connect_mirror_source_connector_replication_latency_ms_max Kafka Connect MM2 Source Connector Information</pre><pre>#TYPE kafka_connect_mirror_source_connector_replication_latency_ms_max gauge</pre><pre>kafka_connect_mirror_source_connector_replication_latency_ms_max{destination=&quot;destination-name&quot;,partition=&quot;7&quot;,topic=&quot;test-topic&quot;,} NaN</pre><p>There will be a row for every partition of the topic for which MM2 is running and when it ends with <em>NaN</em> , it means that the partition is in sync.</p><blockquote>Note: if you don’t start the MM2 in the same session where the environment variable KAFKA_OPTS is set then you will encounter an error with the below curl command. The error being</blockquote><blockquote>curl: (7) Failed to connect to localhost port 3600: Connection refused</blockquote><p>6. Other metrics that can be viewed are as follows:</p><pre>curl localhost:3600 | grep -i &lt;topic-name&gt;</pre><p>The output will have different metric points for the &lt;topic-name&gt; that is being syncd by MM2. The ones that need attention are the following:</p><blockquote>kafka_connect_mirror_source_connector_record_count will tell us the record that is being migrated/ syncd by MM2</blockquote><pre>kafka_connect_mirror_source_connector_record_count{destination=&quot;destination-name&quot;,partition=&quot;7&quot;,topic=&quot;test-topic&quot;,} 32.0</pre><blockquote>kafka_connect_mirror_source_connector_replication_latency_ms_avg shows the average latency which in the following case is 7 ms</blockquote><pre>kafka_connect_mirror_source_connector_replication_latency_ms_max{destination=&quot;destination-name&quot;,partition=&quot;7&quot;,topic=&quot;test-topic&quot;,} 7.0</pre><p>MM2 is in sync when both the above record ends with NaN.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe10d6db4d2d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kafka: Steps to increase Replication Factor for a topic]]></title>
            <link>https://varunkumarpal.medium.com/kafka-steps-to-increase-replication-factor-for-a-topic-af0fd2340ec5?source=rss-3a721f06cfe7------2</link>
            <guid isPermaLink="false">https://medium.com/p/af0fd2340ec5</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[replication-factor]]></category>
            <dc:creator><![CDATA[Varun Pal]]></dc:creator>
            <pubDate>Sun, 15 Nov 2020 23:41:31 GMT</pubDate>
            <atom:updated>2020-11-15T23:41:31.037Z</atom:updated>
            <content:encoded><![CDATA[<p>Below are the steps to increase the Replication Factor for a topic</p><p>To list all the topics in your cluster</p><blockquote>$KAFKA/kafka-topics.sh — zookeeper $ZOOKEEPER — list</blockquote><p>To view all the default and custom configurations for a topic</p><blockquote>$KAFKA/kafka-topics.sh — zookeeper $ZOOKEEPER — topic &lt;topic_name&gt; — describe</blockquote><p>The first line of the output describes the configs. For example:</p><blockquote>Topic: &lt;topic_name&gt; PartitionCount: 8 ReplicationFactor: 2 Configs:</blockquote><p>The above output tells us that this topic has a Partition Count as 8 with a Replication Factor of 2.</p><p>Say we would like to increase the Replication Factor to 3 for this topic. In which case, we need to construct the following increase-replication-factor.json file that elaborates the increase in Replication Factor from 2 to 3.</p><blockquote>cat increase-replication-factor.json<br>{“version”:1,<br> “partitions”:[<br> {“topic”:”&lt;topic_name&gt;”,”partition”:0,”replicas”:[1,2,3]},<br> {“topic”:”&lt;topic_name&gt;”,”partition”:1,”replicas”:[1,2,3]},<br> {“topic”:”&lt;topic_name&gt;”,”partition”:2,”replicas”:[1,2,3]},<br> {“topic”:”&lt;topic_name&gt;”,”partition”:3,”replicas”:[1,2,3]},<br> {“topic”:”&lt;topic_name&gt;”,”partition”:4,”replicas”:[1,2,3]},<br> {“topic”:”&lt;topic_name&gt;”,”partition”:5,”replicas”:[1,2,3]},<br> {“topic”:”&lt;topic_name&gt;”,”partition”:6,”replicas”:[1,2,3]},<br> {“topic”:”&lt;topic_name&gt;”,”partition”:7,”replicas”:[1,2,3]}<br>]}</blockquote><p>To execute the replication execute the following command:</p><blockquote>$KAFKA/kafka-reassign-partitions.sh — zookeeper $ZOOKEEPER — reassignment-json-file $CONFIG/increase-replication-factor.json — execute</blockquote><p>To view the progress of the process the following command can be used</p><blockquote>$KAFKA/kafka-reassign-partitions.sh — list — bootstrap-server $BROKER — command-config $CONFIG/client.properties</blockquote><p>You can verify the progress of the process via the following command</p><blockquote>$KAFKA/kafka-reassign-partitions.sh — zookeeper $ZOOKEEPER — reassignment-json-file $CONFIG/increase-replication-factor.json — verify</blockquote><p>Once the replication factor is increased the above command will display the following output</p><blockquote>Status of partition reassignment:<br>Reassignment of partition &lt;topic_name&gt;-0 is complete.<br>Reassignment of partition &lt;topic_name&gt;-1 is complete.<br>Reassignment of partition &lt;topic_name&gt;-2 is complete.<br>Reassignment of partition &lt;topic_name&gt;-3 is complete.<br>Reassignment of partition &lt;topic_name&gt;-4 is complete.<br>Reassignment of partition &lt;topic_name&gt;-5 is complete.<br>Reassignment of partition &lt;topic_name&gt;-6 is complete.<br>Reassignment of partition &lt;topic_name&gt;-7 is complete.</blockquote><p>The following environment variables have been considered</p><p>$KAFKA — location of the bin directory /root/kafka/bin</p><p>$CONFIG — location of config directory /root/kafka/config</p><p>$ZOOKEEPER — zookeeper string in the localhost:port format</p><p>$BROKER — broker string in the localhost:port format</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af0fd2340ec5" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>