<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Leegin Bernads T S on Medium]]></title>
        <description><![CDATA[Stories by Leegin Bernads T S on Medium]]></description>
        <link>https://medium.com/@leegin-bernads?source=rss-8111d0f31642------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*K3lCsRXhGlHISDoK1gU5cA.jpeg</url>
            <title>Stories by Leegin Bernads T S on Medium</title>
            <link>https://medium.com/@leegin-bernads?source=rss-8111d0f31642------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 12:13:37 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@leegin-bernads/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Migration from Thanos to Grafana Mimir]]></title>
            <link>https://tech.groww.in/migration-from-thanos-to-grafana-mimir-cab21eed7311?source=rss-8111d0f31642------2</link>
            <guid isPermaLink="false">https://medium.com/p/cab21eed7311</guid>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[metrics]]></category>
            <category><![CDATA[grafana-mimir]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[infra]]></category>
            <dc:creator><![CDATA[Leegin Bernads T S]]></dc:creator>
            <pubDate>Tue, 01 Nov 2022 08:56:50 GMT</pubDate>
            <atom:updated>2025-04-21T06:49:45.299Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r9_FuzodWpiciEViyiC9Fw.png" /></figure><h3><strong>Problem statement</strong></h3><p>Grafana Mimir stores series in TSDB blocks uploaded in an object storage bucket like GCS, S3, etc which is very similar to the blocks in Thanos. When we decided to move away from Thanos, we had an important question before us which was “<strong><em>How are we going to migrate the historic blocks from Thanos to Mimir?</em></strong>” because we needed 6 months of the data to be compliant. Grafana Mimir supports uploading historical TSDB blocks, notably from Prometheus and Thanos. The Grafana Labs team also has a migration <a href="https://grafana.com/docs/mimir/latest/migration-guide/migrating-from-thanos-or-prometheus/">document</a> on the process of migrating a deployment of Thanos or Prometheus to Grafana Mimir. The steps are straightforward when it comes to migrating the blocks directly from Prometheus to Mimir but when it comes to Thanos, we have some specific considerations that have to be taken into account</p><ul><li>Blocks that were originally created by Thanos will not include their external labels in the results when queried by Grafana Mimir. So the relabelling of the blocks has to be done if we need these external labels.</li><li>If you have Prometheus running in HA mode, only upload Thanos blocks from a single Prometheus replica from each HA pair for better query performance.</li><li>To ensure the correctness of the query results only upload original (raw) Thanos blocks into Mimir’s storage.</li></ul><p>Mimir team is working on some tool that would have the features to do all the above considerations but the tool is not yet ready. Until then, all these modifications have to be done by us if we want to move the historical blocks to Mimir and have it working as we expected. There is no guide on the internet to support this use case, so we have come up with detailed documentation on how this can be achieved. A big shoutout to <strong><em>Taylor Mutch, Senior Software Engineer, at Unity Technologies</em></strong>, and <strong><em>Peter</em></strong> <strong><em>Stibrany, Engineer at Grafana Labs</em> </strong>for helping us with the process.</p><blockquote>While we will be working on the historical data, we need to enable remote write from the prometheus cluster to Mimir so that we don’t lose the latest data.</blockquote><h3>Solution</h3><h4>Migration Plan</h4><p>The entire migration plan to move from Thanos to Grafana Mimir is as follows.</p><ul><li><strong>Enable remote write from all the Prometheus instances to a tenant “<em>latest</em>” in Mimir. (For the latest blocks)</strong></li><li><strong>Copy historical blocks from Thanos bucket to Mimir intermediate bucket.</strong></li><li><strong>Remove downsampled, duplicated blocks and add external labels to the historical blocks in the intermediate bucket.</strong></li><li><strong>Copy these blocks from the Mimir intermediate bucket to the “<em>latest</em>” tenant in the final Mimir bucket.</strong></li><li><strong>Confirm if the compactor is picking up the blocks uploaded.</strong></li></ul><p>We have not covered the enabling of remote-write in this blog as it is already available on the internet.</p><p>Thanos provides the thanos tools bucket rewrite tool for manipulating blocks in the bucket. We will leverage this CLI tool from Thanos to embed external labels into blocks. First of all, you need to have this tool installed on your local machine or any VM. You can refer to the <a href="https://gist.github.com/leegin/0ee3af1c64e4e6590b628cead789d0af">link</a> for the procedure to install the Thanos tool.</p><h4><strong>Step1: Copy the blocks from Thanos&#39;s bucket to an Intermediate bucket</strong></h4><p>We are using GCP as our cloud provider so from here on when we mention bucket or object storage anywhere it refers to the <strong>GCS bucket. </strong>You have to create an intermediate bucket in GCP(or your cloud provider) where you do the below steps and then copy the processed blocks to the final Mimir bucket. In our case, we copied the blocks from our Thanos bucket to the intermediate Mimir bucket using the gcloud command line utility.</p><pre>gsutil -m cp -r &quot;gs://{Thanos bucket}/*&quot; &quot;gs://{mimir-intermediate-bucket}/&quot;</pre><p>we had around 17 TB of data and it took approximately 1 hour &amp; 30 minutes to copy all the blocks. So it is recommended to run the above command in a screen(command).</p><p>Once the copy process is done, inspect the blocks in the bucket to ensure it’s valid from Thanos perspective.</p><pre>thanos tools bucket inspect \<br>    --objstore.config-file bucket.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nSKfv93shU69DcgLG7HYpA.png" /></figure><p>In the bucket.yaml, you should enter the name of the bucket(to which you copied the blocks), and a service account key to access the bucket. In case you are copying the blocks from the Thanos bucket to a separate folder in the intermediate Mimir bucket, you need to mention the prefix as well. If your folder name is “<strong><em>prod</em></strong>”, you need to provide the prefix “<strong><em>prod</em></strong> “, only then the Thanos tool will be able to detect the blocks and verify if they are valid.</p><h4><strong>Step2: Remove the downsampled blocks</strong></h4><p>Mimir doesn’t understand the downsampled blocks(ie. blocks with non-zero Resolution field in meta.json file) in Thanos. Therefore we need to remove the 5m and 1hr downsampled blocks from this bucket.</p><p>For this, we have to mark the downsampled blocks for deletion using the below command.</p><pre>thanos tools bucket retention \<br>    --objstore.config-file bucket.yaml \<br>    --retention.resolution-1h=1s \<br>    --retention.resolution-5m=1s \<br>    --retention.resolution-raw=0d</pre><p>Once you execute the above command the downsampled blocks will be marked for deletion which you can confirm from the blocks themselves. You’ll find an additional file named “<strong><em>deletion-mark.json</em></strong>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VQMYcH7CZajrpGUxfhPn9g.png" /></figure><p>Next, let&#39;s clean up the blocks marked for deletion.</p><pre>thanos tools bucket cleanup \<br>    --objstore.config-file bucket.yaml \<br>    --delete-delay=0</pre><h4>Step3: Remove the duplicated blocks</h4><p>For best query performance, only upload Thanos blocks from a single Prometheus replica from each HA pair. If you upload blocks from both replicas, the query results returned by Mimir will include samples from both replicas. Therefore if you have deployed two replicas(say <strong>kube-prometheus-0 &amp; kube-prometheus-1</strong>) of Prometheus instances for HA, then only upload the blocks from one of the replicas and drop the other. In our case, we chose to keep the blocks from “<strong><em>kube</em></strong>-<strong><em>prometheus-0</em></strong>”.</p><pre>thanos tools bucket inspect \<br>    --objstore.config-file bucket.yaml \<br>    --output=tsv \<br>    | grep prometheus_replica=kube-prometheus-1 \<br>    | awk &#39;{print $1}&#39; \<br>    | xargs -n 1 -P 1 \<br>    thanos tools bucket mark \<br>    --marker=&quot;deletion-mark.json&quot; \<br>    --objstore.config-file bucket.yaml \<br>    --details=&quot;Removed as duplicate&quot; \<br>    --id</pre><p>Once you execute the above command the blocks from will “<strong>kube-prometheus-1</strong>” replica will be marked for deletion. We can again clean up the blocks marked for deletion.</p><pre>thanos tools bucket cleanup \<br>    --objstore.config-file bucket.yaml \<br>    --delete-delay=0</pre><blockquote><strong>Note</strong>: If the <em>--deduplication.replica-label=&quot;prometheus_replica&quot;</em> is set, then the above step to remove duplicate blocks need to be performed only on blocks which were uploaded recently by the sidecar and not yet worked on by the compactor. Once the blocks are taken by compactor, the replica label will be removed after deduplication.</blockquote><p><strong>Pro tip:</strong></p><p><strong><em>If you want to visualize what exactly is happening in the blocks with respect to the source of blocks, labels, compaction levels, etc, you can use the below command to get the output as CSV and import it to a google sheet.</em></strong></p><pre>thanos tools bucket inspect \<br>    --objstore.config-file bucket.yaml \<br>    --output=csv &gt; thanos-blocks.csv </pre><h4>Step4: Relabel the blocks with external labels</h4><p>Finally, let us relabel the blocks with the labels that are required. In our case “<strong><em>cluster</em></strong>” was considered to be the important label. Follow the below to rewrite the data so that each series includes the <strong>cluster </strong>label.</p><p>Create a rewrite config like the following:</p><pre># relabel-config.yaml<br>- action: replace<br>  target_label: &quot;cluster&quot;<br>  replacement: &quot;prod-cluster&quot;</pre><p>Perform the rewrite <strong><em>dry run</em></strong> to confirm all work well.</p><pre>thanos tools bucket inspect \<br>    --objstore.config-file bucket.yaml \<br>    --output=tsv \<br>    | grep &lt;PROMETHEUS-CLUSTER&gt; \<br>    | awk &#39;{print $1}&#39; \<br>    | xargs -n 1 -P 1 \<br>    thanos tools bucket rewrite  \<br>    --objstore.config-file bucket.yaml \<br>    --rewrite.to-relabel-config-file relabel-config.yaml \<br>    --dry-run \<br>    --id</pre><p>Once you confirm that the rewrite is working as expected, you can apply the changes with the “<strong>no-dry-run</strong>” flag. If this flag is not mentioned and you just remove the <strong>dry-run </strong>flag<strong>, </strong>the changes won’t be applied.</p><pre>thanos tools bucket inspect \<br>    --objstore.config-file bucket.yaml \<br>    --output=tsv \<br>    | grep &lt;PROMETHEUS-CLUSTER&gt; \<br>    | awk &#39;{print $1}&#39; \<br>    | xargs -n 1 -P 1 \<br>    thanos tools bucket rewrite  \<br>    --objstore.config-file bucket.yaml \<br>    --rewrite.to-relabel-config-file relabel-config.yaml \<br>    --no-dry-run \<br>    --delete-blocks \<br>    --id</pre><p>This command will create a new block out of the original block by rewriting the labels which you can see from the output of the command.</p><pre>level=info ts=2022-10-10T13:03:32.032820262Z caller=factory.go:50 msg=&quot;loading bucket configuration&quot;<br>level=info ts=2022-10-10T13:03:32.516953867Z caller=tools_bucket.go:1160 msg=&quot;downloading block&quot; source=01GEGWPME2187SVFH63G8DH7KH<br>level=info ts=2022-10-10T13:03:35.825009556Z caller=tools_bucket.go:1197 msg=&quot;changelog will be available&quot; file=/tmp/thanos-rewrite/01GF0ZWPWGEPHG5NV79NH9KMPV/change.log<br>level=info ts=2022-10-10T13:03:35.836953593Z caller=tools_bucket.go:1212 msg=&quot;starting rewrite for block&quot; source=01GEGWPME2187SVFH63G8DH7KH new=01GF0ZWPWGEPHG5NV79NH9KMPV toDelete= toRelabel=&quot;- action: replace\n  target_label: \&quot;cluster\&quot;\n  replacement: \&quot;prod-cluster\&quot;\n&quot;<br>level=info ts=2022-10-10T13:04:47.57624244Z caller=compactor.go:42 msg=&quot;processed 10.00% of 701243 series&quot;<br>level=info ts=2022-10-10T13:04:53.4046885Z caller=compactor.go:42 msg=&quot;processed 20.00% of 701243 series&quot;<br>level=info ts=2022-10-10T13:04:59.649337602Z caller=compactor.go:42 msg=&quot;processed 30.00% of 701243 series&quot;<br>level=info ts=2022-10-10T13:05:02.986219042Z caller=compactor.go:42 msg=&quot;processed 40.00% of 701243 series&quot;<br>level=info ts=2022-10-10T13:05:05.990498497Z caller=compactor.go:42 msg=&quot;processed 50.00% of 701243 series&quot;<br>level=info ts=2022-10-10T13:05:09.349918024Z caller=compactor.go:42 msg=&quot;processed 60.00% of 701243 series&quot;<br>level=info ts=2022-10-10T13:05:12.040895624Z caller=compactor.go:42 msg=&quot;processed 70.00% of 701243 series&quot;<br>level=info ts=2022-10-10T13:05:15.253899238Z caller=compactor.go:42 msg=&quot;processed 80.00% of 701243 series&quot;<br>level=info ts=2022-10-10T13:05:18.471471014Z caller=compactor.go:42 msg=&quot;processed 90.00% of 701243 series&quot;<br>level=info ts=2022-10-10T13:05:21.536267363Z caller=compactor.go:42 msg=&quot;processed 100.00% of 701243 series&quot;<br>level=info ts=2022-10-10T13:05:21.536466158Z caller=tools_bucket.go:1222 msg=&quot;wrote new block after modifications; flushing&quot; source=01GEGWPME2187SVFH63G8DH7KH new=01GF0ZWPWGEPHG5NV79NH9KMPV<br>level=info ts=2022-10-10T13:05:28.675240198Z caller=tools_bucket.go:1231 msg=&quot;uploading new block&quot; source=01GEGWPME2187SVFH63G8DH7KH new=01GF0ZWPWGEPHG5NV79NH9KMPV<br>level=info ts=2022-10-10T13:05:38.922348564Z caller=tools_bucket.go:1241 msg=uploaded source=01GEGWPME2187SVFH63G8DH7KH new=01GF0ZWPWGEPHG5NV79NH9KMPV<br>level=info ts=2022-10-10T13:05:38.979696873Z caller=block.go:203 msg=&quot;block has been marked for deletion&quot; block=01GEGWPME2187SVFH63G8DH7KH<br>level=info ts=2022-10-10T13:05:38.979832767Z caller=tools_bucket.go:1249 msg=&quot;rewrite done&quot; IDs=01GEGWPME2187SVFH63G8DH7KH<br>level=info ts=2022-10-10T13:05:38.980197796Z caller=main.go:161 msg=exiting</pre><p>The “<strong>delete-blocks</strong>” flag will mark the original block for deletion which can be cleaned up later with the command which we used in the previous steps.</p><blockquote><strong>Note</strong>: If you have multiple clusters and if you want to run this relabelling each of them separately, you can get the blocks ids of the blocks that are responsible for the cluster and write a for loop to iterate over them.</blockquote><pre>#Get the block ids of the cluster and save it to a file.<br>$ thanos tools bucket inspect \<br>     --objstore.config-file bucket.yaml \<br>     --output=tsv \<br>     | grep &lt;PROMETHEUS-CLUSTER&gt; \<br>     | awk &#39;{print $1}&#39; \<br>     | xargs -n 1 -P 1 &gt; &lt;PROMETHEUS-CLUSTER&gt;-blocks.txt</pre><pre>#Iterate through all the block ids in the file using for loop and rewrite them.</pre><pre>$ for i in $(cat &lt;PROMETHEUS-CLUSTER&gt;-blocks.txt)<br>  do<br>      thanos tools bucket rewrite \<br>      --objstore.config-file bucket.yaml \<br>      --rewrite.to-relabel-config-file relabel-config.yaml \<br>      --no-dry-run \<br>      --delete-blocks \<br>      --id $i<br>  done</pre><p>The relabelling process takes a lot of time(it took over 10 days for us) as the block has to be downloaded and relabelled. As the number of blocks increases, the time to convert would also take a lot of time. You can run them in parallel in a screen command to get this done soon.</p><h4>Step5: Remove external labels from meta.json</h4><p>Mimir compactor will not be able to compact the blocks having external labels with Mimir’s own blocks that don’t have any such labels in their meta.json. Therefore we need to remove these external labels before copying them to the Mimir bucket.</p><pre># Create a folder named &quot;orig&quot;.<br>mkdir -p orig</pre><pre># Copy the meta.json file from all the blocks to orig folder in your local machine excluding the index and chunk folders.</pre><pre>gsutil -m rsync -r -x &#39;.*(index|chunks|debug).*&#39; &quot;gs://{Intermediate Mimir bucket}/&quot; orig/</pre><p>We used the below script to remove the labels from the meta.json. We had external labels like env, cluster, and Prometheus. You can replace them with your external labels.</p><pre>#!/bin/bash<br># Description: Update Thanos external labels</pre><pre>mkdir -p output<br>find orig -name &#39;meta.json&#39; |<br>  while read orig; do<br>  block=&quot;$(echo &quot;${orig}&quot; | cut -f2 -d&#39;/&#39;)&quot;<br>  tmpfile=$(mktemp)<br>  jq &#39;del(.thanos.labels)&#39; &quot;${orig}&quot; &gt; &quot;${tmpfile}&quot;<br>  if ! diff -u &lt;(jq -S . &quot;${orig}&quot;) &lt;(jq -S . &quot;${tmpfile}&quot;) &gt; /dev/null; then<br>     echo &quot;Updating ${block}&quot;<br>     mkdir -p &quot;output/${block}&quot;<br>     cp &quot;${tmpfile}&quot; &quot;output/${block}/meta.json&quot;<br>  else<br>     echo &quot;No diff on ${block}&quot;<br>  fi<br>  rm &quot;${tmpfile}&quot;<br>done</pre><p>The script will create a folder named “<strong><em>output</em></strong>” and update the meta.json file of all the blocks and store the result here. Next, we need to sync the meta.json in the output folder and the blocks folder in the GCS bucket.</p><pre>for ID in `ls -al ~/output/| awk &#39;{print $9}&#39;`<br>do<br>    gsutil -m rsync -r -x &#39;.*(index|chunks|debug).*&#39; ~/output/$ID gs://{mimir-intermediate-bucket}/$ID<br>done</pre><h4><strong>Step6: Copy the blocks from the intermediate bucket to the Mimir bucket</strong></h4><p>Once the relabelling and removal of the external labels are done, we are ready to copy the blocks from the intermediate bucket to the final Mimir bucket where the latest blocks from the clusters are written.</p><pre>gsutil -m cp -r gs://{mimir-intermediate-bucket}/ gs://{mimir-gcs-bucket}/{tenant-name&gt;/</pre><p>Again it is recommended to run this command on the screen as well to prevent any interruptions as this may take time depending on the amount of data to be copied.</p><h4>Conclusion</h4><p>You will not be able to immediately query the historical blocks as soon as they are copied as the bucket index to include all available blocks in the index have to be updated by the compactor and it would again take time(depending on the number of blocks to be updated and the index updation normally runs every 15 minutes). Once this is done, other components like querier and store-gateway will be able to work with these historical blocks and they will be available for querying through Grafana.</p><p>You can check the store gateway HTTP endpoint at <strong>http://&lt;storegateway-endpoint&gt;/store-gateway/tenant/&lt;tenant-name&gt;/blocks </strong>and you should be able to see the uploaded blocks there.</p><p>You can refer to the official documentation of Grafana Mimir for your queries related to capacity planning for your scale or if you want to understand how compaction in Mimir is different from Thanos.</p><p>If you’ve enjoyed this story, please click the 👏 button and share it, so that others can find it as well! Also, feel free to leave a comment below.</p><p><a href="http://tech.groww.in/">Groww Engineering</a> publishes technical anecdotes, the latest technologies, and better ways to tackle common programming problems. You can <a href="https://medium.com/groww-engineering">subscribe here</a> to get the latest updates.</p><p>We are hiring. View job openings <a href="https://groww.in/careers">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cab21eed7311" width="1" height="1" alt=""><hr><p><a href="https://tech.groww.in/migration-from-thanos-to-grafana-mimir-cab21eed7311">Migration from Thanos to Grafana Mimir</a> was originally published in <a href="https://tech.groww.in/">The Groww Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Render Build Parameters dynamically in Jenkins Pipeline]]></title>
            <link>https://tech.groww.in/render-build-parameters-dynamically-in-jenkins-pipeline-a367a2d0a84c?source=rss-8111d0f31642------2</link>
            <guid isPermaLink="false">https://medium.com/p/a367a2d0a84c</guid>
            <category><![CDATA[dynamic-parameters]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[jenkins-pipeline]]></category>
            <category><![CDATA[jenkins]]></category>
            <dc:creator><![CDATA[Leegin Bernads T S]]></dc:creator>
            <pubDate>Fri, 19 Aug 2022 05:49:05 GMT</pubDate>
            <atom:updated>2025-04-21T06:53:25.435Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*AgQvUXrosf1tKEGN.png" /></figure><h4>Problem statement:</h4><p>During our initial days in observability, we had individual pipelines for configuring and installing different components (Prometheus, Thanos, Loki, etc) of observability. As the number of clusters increased, we started having separate pipelines for each cluster. As a result, we had so many pipelines and soon we realized this approach is not scalable. We decided to clean up these multiple pipelines and have a single pipeline for installing say Prometheus in all our clusters.</p><p>The initial solution we implemented was using the Jenkins plugin “<strong>Choice Parameter</strong>” which passes some parameters which we declare and use as Groovy variables in our Jenkins jobs. For example, we had pipelines to deploy the Prometheus operator, additional scrape config of Prometheus, etc where we need to pass multiple parameters like the cluster, project, release name, values file, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9c-NVEZCZBdW4nCaZekVYA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VNMNMksV-iw-nmW4dU8w7w.png" /></figure><p>This solved our issue partially as we were able to deploy with a single pipeline. But as the clusters increased, there was confusion in selecting the right combination of options for a cluster for many of our teammates. However, not everyone in the team is aware of all the parameters at the beginning and we had a requirement to render the parameters dynamically based on the value selected in any other parameter. We cannot achieve the use-case with the native Jenkins parameters available.</p><p><strong>Solution:</strong></p><p>After some research, we came across this Jenkins plugin called the “<strong>Active Choices</strong>” plugin.</p><h4>What is the “<strong>Active Choices</strong>” plugin and what does it do?</h4><blockquote>The Active Choices plugin is used in parametrized freestyle Jenkins jobs to create scripted, dynamic, and interactive job parameters. Active Choices parameters can be dynamically updated and can be rendered as combo-boxes, check-boxes, radio buttons, or rich HTML UI widgets.</blockquote><p>Active Choices plugin provides 3 types of parameters.</p><ol><li><strong>Active Choices Parameter</strong></li><li><strong>Active Choices Reactive Parameter</strong></li><li><strong>Active Choices Reactive Reference Parameter</strong></li></ol><p>For our use case, we chose the <strong>Active Choices Parameter </strong>and the <strong>Active Choices Reactive Parameter.</strong></p><h4>Active Choices Parameter:</h4><ul><li>An Active Choices parameter dynamically generates a list of value options for a build parameter using a Groovy script or a script from the Scriptler catalog.</li></ul><h4>Active Choices Reactive Parameter:</h4><ul><li>An Active Choices Reactive Parameter can generate the same set of value options as the Active choice parameter; in addition to that, it can be dynamically updated with the values based on the selection in some other build parameter that was referred to in the Active choices reactive parameter.</li><li>Active Choices Reactive parameters can be rendered as standard selection lists, checkboxes, and radio buttons.</li></ul><h4>Prerequisites</h4><ul><li>This plugin requires Jenkins version &gt;= 2.204.1.</li><li>The Active Choices plugin is installed on Jenkins.</li></ul><h4>Implementation</h4><p>We will see how we can configure the Active Choices parameter and Active Choices Reactive parameter.</p><ul><li>On the configure job page select the “<strong>This project is parameterized</strong>” checkbox in the general tab.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VFwWatA58gOQU5sr.gif" /></figure><ul><li>Now click on the <strong>Add Parameters</strong> dropdown and select the <strong>Active Choices Parameter</strong> from the list.</li><li>In the <strong>Name</strong> field add “<strong>cluster</strong>” as the name for the parameters and select the Groovy script in the Script radio buttons. Add a suitable description for your parameter.</li><li>Next is the <strong>Choice Type</strong> dropdown which has 4 choices available.</li></ul><blockquote><strong>Single Select</strong> : It will render a dropdown list from which a user can select only one value. Value will be accessible in the pipeline code via a variable name.</blockquote><blockquote><strong>Multi Select</strong> : It will render a dropdown list from which a user can select multiple values. If the user has selected multiple values then in the pipeline code value will be passed as comma separated string, we can get the value via parameter name as we would have got it for any other parameter.</blockquote><blockquote><strong>Radio Buttons</strong> : It will render a radio button group for the list we have returned from the Groovy script. This will be a single selection parameter only.</blockquote><blockquote><strong>Checkboxes</strong> : It will render a checkbox group for the list we have returned from the Groovy script. If we select the multiple checkboxes then the values will be comma separated.</blockquote><ul><li>Now we will add the values to this parameter in the Groovy script box.</li></ul><pre>return [&quot;cluster-001&quot;, &quot;cluster-002&quot;, &quot;cluster-003&quot;]</pre><ul><li>This will render a dropdown list with these values. A fallback script is used when there is some exception in the script execution. Save the job and open the Build with parameters page. This is how it will be visible on the Build With Parameters page.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AdSQseQ4FuqYT49Q.gif" /></figure><p>Now we will add an <strong>Active Choices Reactive Parameter</strong> to select the value of the GCP <strong>project</strong> in which the cluster is deployed based on the selection in the <strong>cluster</strong> parameter.</p><ul><li>Click on the configure and scroll down to the <strong>cluster</strong> parameter we have added.</li><li>Click again on the A<strong>dd parameter</strong> dropdown and select the <strong>Active Choices Reactive Parameter</strong>.</li><li>In the <strong>Name</strong> field add “<strong>project</strong>” as the name for the parameters and select the Groovy script in the Script radio buttons.</li><li>Fill in the following code in the Groovy script box.</li></ul><pre>if (cluster == &#39;cluster-001&#39;){<br> return[&#39;project-01&#39;]<br>} else if (cluster == &#39;cluster-002&#39;){<br> return[&#39;project-02&#39;]<br>} else if (cluster == &#39;cluster-003&#39;){<br> return[&#39;project-03&#39;]<br>}</pre><ul><li>In <strong>Referenced parameters </strong>field add “<strong>cluster</strong>” since we want to parameters to be selected based on the value of the <strong>cluster</strong> parameter. Now, hit the save button and click on the Build with parameters.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7kWH6rzb5PkN7s_c.gif" /></figure><p>It will look something like this on the Build with parameters page. Based on the selection in the <strong>cluster</strong> dropdown list the options will be updated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KuZaZ0HIv8foG4zt.gif" /></figure><p>We had more parameters which have to be updated based on the selection in the <strong>cluster</strong> dropdown. Therefore we repeated the process with “<strong>Active Choices Reactive Parameter</strong>” and referenced to parameter “<strong>cluster</strong>”.</p><h4>Conclusion</h4><p>The Active Choices plugin helped us to solve this use-case by allowing us to render the parameter values and even the parameters dynamically by means of Groovy. This plugin is very simple to use as it needs just a basic level of understanding of Groovy and HTML. Hope the article was informative and you enjoyed reading it.</p><p>If you’ve enjoyed this story, please click the 👏 button and share it, so that others can find it as well! Also, feel free to leave a comment below.</p><p><a href="http://tech.groww.in/">Groww Engineering</a> publishes technical anecdotes, the latest technologies, and better ways to tackle common programming problems. You can <a href="https://medium.com/groww-engineering">subscribe here</a> to get the latest updates.</p><p>We are hiring. View job openings <a href="https://groww.in/careers">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a367a2d0a84c" width="1" height="1" alt=""><hr><p><a href="https://tech.groww.in/render-build-parameters-dynamically-in-jenkins-pipeline-a367a2d0a84c">Render Build Parameters dynamically in Jenkins Pipeline</a> was originally published in <a href="https://tech.groww.in/">The Groww Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Prevent metrics explosion in Prometheus(JMX Exporter)]]></title>
            <link>https://tech.groww.in/prevent-metrics-explosion-in-prometheus-jmx-exporter-2c7e6c6da8fb?source=rss-8111d0f31642------2</link>
            <guid isPermaLink="false">https://medium.com/p/2c7e6c6da8fb</guid>
            <category><![CDATA[metrics]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[infra]]></category>
            <category><![CDATA[observability]]></category>
            <dc:creator><![CDATA[Leegin Bernads T S]]></dc:creator>
            <pubDate>Fri, 19 Aug 2022 05:47:07 GMT</pubDate>
            <atom:updated>2025-04-21T06:53:57.140Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O7s_yre6BkCQNZ77okIcVg.png" /></figure><p>In this blog, let&#39;s look into the solution to prevent metrics explosion over time in Prometheus while using a JMX exporter.</p><h4>Problem statement:</h4><p>We are running JMX exporter in our Kafka clusters to fetch Kafka metrics and scrape them using Prometheus for our monitoring. We observed that the JMX exporter is exposing so many metrics that are not being used by us in our monitoring system. This metrics explosion affects the stability, reliability, and performance of our Prometheus clusters. We have also seen instances where the number of metrics in the brokers is so high that they take too long to expose the metrics at the endpoint than the configured scrape timeout which results in a “<strong>Context Deadline Exceeded</strong>” error.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZEAJphhCuIjZWL6kQBvZFg.png" /></figure><p>We are observing this issue for quite some time now and this is causing false alerts as well. To solve this metrics explosion we need to drop metrics that are not useful to us. This can be done either at the source(JMX exporter) or from Prometheus while scraping. The disadvantage of doing this at the Prometheus end is that the brokers would still take the same amount of time to expose the metrics. So the ideal option is to drop them from the source itself.</p><p>We were running the exporter without any whitelist or blacklist of the object names and therefore they were first fetching all of the MBean objects available within Kafka, which was collecting far more data than we were actually using.</p><h4>Solution:</h4><ol><li><strong>Figure out unused metrics</strong></li></ol><p>The first step here is to figure out the metrics that are producing a lot of time series and then confirm if they are being used by us. I figured out the following 8 metrics which have too many time series and are unused in our monitoring.</p><pre>kafka_server_brokertopicmetrics_totalfetchrequests_total<br>kafka_cluster_partition_replicascount<br>kafka_cluster_partition_laststableoffsetlag<br>kafka_log_numlogsegments<br>kafka_cluster_partition_underminisr<br>kafka_cluster_partition_atminisr<br>kafka_server_fetcherlagmetrics_consumerlag<br>kafka_log_logstartoffset</pre><p><strong>2. Blacklisting unused metrics</strong></p><p>We can achieve this in two ways. Either by whitelisting only the metrics that we want or by blacklisting the metrics that need to be dropped. Since there are so many metrics in Kafka which are useful to us and as we already know the unused metrics, the better approach here was the blacklisting of metrics.</p><pre>blacklistObjectNames: []</pre><p>In the exporter config which is under <strong>/opt/jmxexporter/config.yml, </strong>we need to add the above line and this blacklist is a list.</p><pre>blacklistObjectNames: [&quot;kafka.log:type=Log,name=NumLogSegments,*&quot;,&quot;kafka.log:type=Log,name=LogStartOffset,*&quot;,&quot;kafka.server:type=BrokerTopicMetrics,name=TotalFetchRequestsPerSec,*&quot;,&quot;kafka.cluster:type=Partition,name=ReplicasCount,*&quot;,&quot;kafka.cluster:type=Partition,name=LastStableOffsetLag,*&quot;,&quot;kafka.cluster:type=Partition,name=UnderMinIsr,*&quot;,&quot;kafka.cluster:type=Partition,name=AtMinIsr,*&quot;,&quot;kafka.server:type=FetcherLagMetrics,name=ConsumerLag,*&quot;]</pre><h4>3. Format to blacklist a metric</h4><p>If you want to blacklist the metrics “<strong>TotalTimeMs</strong>”, the format to be added is as follows.</p><pre>&quot;kafka.network:type=RequestMetrics, name=TotalTimeMs,*&quot;</pre><p>You can get this from the metrics endpoint itself. Here just above any metric as a description, you would find a few lines and this format can be found there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NTpkFb1viZjzLknT1wT1FQ.png" /></figure><p>Instead of <strong>&lt;type</strong>, we would be adding “<strong>:type</strong>” in the config so “<strong>kafka.network&lt;type=RequestMetrics, name=TotalTimeMs,*</strong>” would turn to “<strong>kafka.network:type=RequestMetrics, name=TotalTimeMs,*</strong>” when it is added to the config.</p><p><strong>4. Restart the exporter</strong></p><p>Once the blacklist is added to the config, restart the JMX exporter service in the broker.</p><pre><strong>$ </strong>service jmxexporter restart</pre><pre><strong>$ </strong>service jmxexporter status</pre><p>After the metrics are blacklisted come back to the metrics endpoint of the broker and check for the metrics again and it should be gone.</p><p>By performing the prevention of metrics explosion in JMX exporter we have reduced a good number of time series per scrape just from Kafka brokers and there would be no false alerts because of the scrape timeout getting exceeded as well.</p><h4>Reference:</h4><p><a href="http://www.whiteboardcoder.com/2017/04/prometheus-and-jmx.html">Prometheus and JMX</a></p><p>If you’ve enjoyed this story, please click the 👏 button and share it, so that others can find it as well! Also, feel free to leave a comment below.</p><p><a href="http://tech.groww.in/">Groww Engineering</a> publishes technical anecdotes, the latest technologies, and better ways to tackle common programming problems. You can <a href="https://medium.com/groww-engineering">subscribe here</a> to get the latest updates.</p><p>We are hiring. View job openings <a href="https://groww.in/careers">here</a>.</p><p>Groww Engineering publishes technical anecdotes, the latest technologies, and better ways to tackle common programming problems. You can subscribe here to get the latest updates.</p><p>We are hiring. View job openings here.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2c7e6c6da8fb" width="1" height="1" alt=""><hr><p><a href="https://tech.groww.in/prevent-metrics-explosion-in-prometheus-jmx-exporter-2c7e6c6da8fb">Prevent metrics explosion in Prometheus(JMX Exporter)</a> was originally published in <a href="https://tech.groww.in/">The Groww Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Custom alert priorities in Opsgenie don’t work with grouped alerts]]></title>
            <link>https://tech.groww.in/custom-alert-priorities-in-opsgenie-dont-work-with-grouped-alerts-53b8f7e7365a?source=rss-8111d0f31642------2</link>
            <guid isPermaLink="false">https://medium.com/p/53b8f7e7365a</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[infra]]></category>
            <category><![CDATA[opsgenie]]></category>
            <category><![CDATA[alerting]]></category>
            <category><![CDATA[prometheus]]></category>
            <dc:creator><![CDATA[Leegin Bernads T S]]></dc:creator>
            <pubDate>Mon, 18 Apr 2022 05:27:29 GMT</pubDate>
            <atom:updated>2025-04-21T07:06:02.152Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nj7wytLc2Zhhn7oKIzZlLA.png" /></figure><h3>Introduction</h3><p>The Priority field of an integration defines the level of importance of an alert that is created by that integration. An alert may have one of the five different types of priority levels: <strong>P1</strong>, <strong>P2</strong>, <strong>P3</strong>, <strong>P4,</strong> or <strong>P5</strong> (importance level decreases from P1 to P5).</p><p>You can select any of the <strong>P1</strong>, <strong>P2</strong>, <strong>P3</strong>, <strong>P4,</strong> or <strong>P5</strong> manually or you can either select <em>{{priority}}</em> dynamic field or the <em>Custom Value to Opsgenie Priority</em> to define how you want to parse the priority from the payload.</p><p>If you select manual values, the alert created from that integration will always have that priority.</p><p>If the {{priority}} dynamic field is selected, then it may have two different outcomes. If the integration sends a priority value and Opsgenie parses and maps it to its known priority levels, then the priority is set according to the priority value from the payload that the tool sends. If the tool does not carry priority information, the alert will have the default priority value, P3. From the alert manager, we can map severity to priority by adding the below config to the Opsgenie integration block of the alert manager configuration file.</p><pre>priority: &#39;{{ range .Alerts }}{{ if eq .Labels.severity &quot;critical&quot;}}P1{{else if eq .Labels.severity &quot;warning&quot;}}P2{{else if eq .Labels.severity &quot;info&quot;}}P3{{else}}P4{{end}}{{end}}&#39;</pre><p>The above condition where an unknown value has a default priority of P3 applies only when you have a single alert.</p><h3>Problem statement</h3><h4>What happens with grouped alerts</h4><p>In Groww, we have grouped our alerts based on certain criteria&#39;s like alertname, severity, stack, etc. We applied the above priority mapping config in our alert manager and then we observed a strange behavior where we get alerts when there is only a single alert and when multiple alerts are grouped, we could see the alert is fired to the alert manager but they don’t reach Opsgenie and we observed the below error in the alert manager logs.</p><pre>level=error ts=2022-04-01T01:09:23.350Z caller=dispatch.go:354 component=dispatcher msg=&quot;Notify for alerts failed&quot; num_alerts=3 err=&quot;opsgenie/opsgenie[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 422: {\&quot;message\&quot;:\&quot;Request body is not processable. Please check the errors.\&quot;,\&quot;errors\&quot;:{\&quot;priority\&quot;:\&quot;should be one of [ P1, P2, P3, P4, P5 ]\&quot;},\&quot;took\&quot;:0.001,\&quot;requestId\&quot;:\&quot;a6390ce3-f99c-4149-820d-1886421ef123\&quot;}&quot;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*pp1tWP4B5TetoFINvXx4ew.jpeg" /></figure><p>After in depth investigation we found that when we group our alerts based on certain conditions, the “<strong>range</strong>” here will iterate over all the alerts, so if we have more than one alert this will generate a priority like “P1P2P2” or something similar, which Opsgenie will reject as it doesn’t get the expected priority.</p><h3>Solution</h3><p>We need to stop the alert priorities from getting concatenated. The solution is to group the alerts by some label-like <em>severity </em>and ignore taking the <em>range</em>.</p><pre>priority: &#39;{{ if eq .GroupLabels.severity &quot;critical&quot;}}P1{{else if eq .GroupLabels.severity &quot;warning&quot;}}P2{{else if eq .GroupLabels.severity &quot;info&quot;}}P3{{else}}P4{{end}}&#39;</pre><p>After adding the above configuration, the alerts were assigned the correct priorities and grouped alerts were accepted by Opsgenie.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*o_dICt3HM16eHf3nai9Eyg.jpeg" /></figure><p>If you’ve enjoyed this story, please click the 👏 button and share it, so that others can find it as well! Also, feel free to leave a comment below.</p><p><a href="http://tech.groww.in/">Groww Engineering</a> publishes technical anecdotes, the latest technologies, and better ways to tackle common programming problems. You can <a href="https://medium.com/groww-engineering">subscribe here</a> to get the latest updates.</p><p>We are hiring. View job openings <a href="https://groww.in/careers">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=53b8f7e7365a" width="1" height="1" alt=""><hr><p><a href="https://tech.groww.in/custom-alert-priorities-in-opsgenie-dont-work-with-grouped-alerts-53b8f7e7365a">Custom alert priorities in Opsgenie don’t work with grouped alerts</a> was originally published in <a href="https://tech.groww.in/">The Groww Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Event-driven Autoscaling in Kubernetes using KEDA]]></title>
            <link>https://tech.groww.in/event-driven-autoscaling-in-kubernetes-using-keda-c6aa27a10a80?source=rss-8111d0f31642------2</link>
            <guid isPermaLink="false">https://medium.com/p/c6aa27a10a80</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[keda]]></category>
            <category><![CDATA[autoscaling]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[infra]]></category>
            <dc:creator><![CDATA[Leegin Bernads T S]]></dc:creator>
            <pubDate>Sun, 06 Jun 2021 20:04:42 GMT</pubDate>
            <atom:updated>2025-04-21T07:33:18.182Z</atom:updated>
            <content:encoded><![CDATA[<p>Kubernetes out of the box supports Autoscaling through the Horizontal Pod Autoscaler (HPA) which automatically scales the number of pods in a deployment, replica set, or stateful set based on CPU and memory utilization. But at times those alone are not enough, especially if you follow the event-driven or serverless architecture and developing cloud-native applications. One of the most common use cases in such systems would be to scale workloads based on the number of messages in a queue/topic rather than CPU or memory. Ultimately we would want to scale out the consumers of the topic/queue when the message count is increasing and scale in again once the count is reducing. Even though there is no built-in support for this in Kubernetes, we can achieve this through an open-source project named “<strong>KEDA</strong>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NToHDgykx8c4CncjwFcFEw.png" /></figure><p><a href="https://keda.sh/"><em>KEDA</em></a><em> stands for Kubernetes-based Event-Driven Autoscaler. </em><strong><em>KEDA</em></strong><em> is a single-purpose and lightweight component that can be added to any Kubernetes cluster. KEDA works alongside standard Kubernetes components like the </em><a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/"><em>Horizontal Pod Autoscaler</em></a><em> and can extend functionality without overwriting or duplication. With KEDA you can explicitly map the apps you want to use event-driven scale, with other apps continuing to function. This makes KEDA a flexible and safe option to run alongside any number of any other Kubernetes applications or frameworks.</em></p><h3>Architecture</h3><p>The diagram below shows how KEDA works in conjunction with the Kubernetes Horizontal Pod Autoscaler, external event sources, and Kubernetes <a href="https://etcd.io/">etcd</a> datastore:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aKKlPS2J2wE_HnJF8DvUDg.png" /></figure><p>You can read up on how KEDA works and more in-depth details about KEDA from their <a href="https://keda.sh/docs/1.4/concepts/">official documentation</a>. Here I will be discussing the journey of how Groww as an organization has leveraged KEDA for scaling applications and our learnings using KEDA.</p><blockquote>The limit of HPA embedded in Kubernetes is that it cannot handle scale down to zero.</blockquote><p>We were running on the latest version of Kubernetes 1.19.9 and I deployed the latest version of KEDA v2.2 in this cluster. You can refer to the <a href="https://keda.sh/docs/2.3/deploy/">link</a> for the instructions for deploying KEDA in your Kubernetes cluster.</p><p>After the deployment is successful you should see the KEDA operator and KEDA metrics API server pods running.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bORMRb_-L0MurGD8k6PkOg.png" /></figure><p>We used <a href="https://kafka.apache.org/">Kafka</a> as our message broker and scale the pods based on lag in a Kafka topic. Although we are using Kafka as our message broker here, KEDA has built-in support for many other scalers(scalers represent event sources that KEDA can scale based on). You can check out more in-built scalers of KEDA <a href="https://keda.sh/docs/2.2/scalers/">here</a>.</p><p>Once we have deployed KEDA in our clusters, it will create four <strong>CRD</strong>(Custom Resource Definition) namely,</p><ol><li>ScaledObject</li><li>ScaledJob</li><li>TriggerAuthentications</li><li>ClusterTriggerAuthentications</li></ol><p>You can read more about these CRDs from their <a href="https://keda.sh/docs/2.2/concepts/">official documentation</a>.</p><p>The flow of information would be as follows for KEDA with Kafka scaler.</p><ul><li>When no messages are pending to be processed, KEDA can scale the deployment to zero or one as per the minReplica set.</li><li>When a message arrives, KEDA detects this event and activates the deployment.</li><li>When the deployment starts running, one of the containers connects to Kafka and starts pulling messages.</li><li>As more messages arrive on the Kafka Topic, KEDA can feed this data to the HPA to drive scale-out.</li><li>Each replica of the deployment is actively processing messages. Very likely, each replica is processing a batch of messages in a distributed manner.</li></ul><p>Let&#39;s bring in KEDA to the party and show the power of Autoscaling and see how KEDA scales out deployments in real-world Kubernetes clusters.</p><h3>Prerequisites</h3><ul><li>Kubernetes cluster and <a href="https://kubernetes.io/docs/tasks/tools/install-kubectl/">kubectl</a> setup.</li><li>Apache Kafka <a href="https://kafka.apache.org/quickstart">setup</a>.</li><li>A Kafka topic</li><li>Kafka producer and a Kafka consumer application</li></ul><p>If you don’t have a producer or the consumer application ready, you can clone the demo application from this git <a href="https://github.com/leegin/KEDA-kafka">repo</a>. You can find all the instructions to build the images in the repo itself.</p><p>Now we have the application images ready so let’s deploy these to the cluster. Before that make sure that you have updated the Kafka topic and the Kafka broker details in the deployment file for the respective applications. First, we will deploy the Kafka producer.</p><pre>kubectl apply -f kafka-producer/deployment.yaml</pre><p>Since we created a service of typeNodePort you can use the format of SERVER_IP:32000 to access the service. You can get the IP address of the server by executing the below command.</p><pre>kubectl get pods -n keda -o wide</pre><blockquote>I have deployed all these objects in a name space “keda”. In production cluster you can deploy them to any namespace.</blockquote><p>We are now all set to publish some messages to the queue now, we can use the rest endpoint of the producer to publish messages. To put some order messages to the topic all you have to do is to do a GET (or just paste the URL in the browser)</p><pre><a href="http://192.168.64.3:32000/orders/2">http://10.68.64.3:32000/orders/</a>2 #number of orders</pre><pre>#output<br>[ {<br>  &quot;id&quot; : &quot;0&quot;,<br>  &quot;status&quot; : &quot;NEW&quot;,<br>  &quot;price&quot; : 650.69841600315,<br>  &quot;quantity&quot; : 19,<br>  &quot;createdTime&quot; : &quot;2021-06-06T06:48:58.111Z&quot;,<br>  &quot;processedTime&quot; : null<br>}, {<br>  &quot;id&quot; : &quot;1&quot;,<br>  &quot;status&quot; : &quot;NEW&quot;,<br>  &quot;price&quot; : 481.096669522158,<br>  &quot;quantity&quot; : 89,<br>  &quot;createdTime&quot; : &quot;2021-06-06T06:48:58.112Z&quot;,<br>  &quot;processedTime&quot; : null<br>} ]</pre><p>Next, let’s go ahead and deploy the Kafka-consumer using the below command.</p><pre>kubectl apply -f kafka-consumer/deployment.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m8-LU2kGGsrQyDqJ--i6mw.png" /></figure><p>Like I mentioned earlier KEDA uses scalers as event sources that KEDA can scale based on. We will be using the <a href="https://keda.sh/docs/1.4/scalers/apache-kafka/">Kafka scaler</a> here which can be used to scale applications based on Apache Kafka Topic. Since we have already deployed the KEDA related resources, now we are going to create a custom resource ScaledObject “<a href="https://github.com/leegin/KEDA-kafka/blob/master/kafka-scaled-object.yaml">kafka-scaled-object.yaml</a>”to describe the scaling policy we want to achieve. The KEDA operator will watch this ScaledObject resource and create the HPA (Horizontal Pod Autoscaler) resource to scale the pods in “Kafka-consumer-deployment” deployment according to the lag in the Kafka topic.</p><blockquote>If you have multiple topics that are consumed by a single consumer you can add them to the triggers as below.</blockquote><pre>triggers:  <br>- type: kafka    <br>    metadata:      <br>      bootstrapServers: 10.60.10.253:9092  <br>      consumerGroup: order-shipper<br>      topic: keda-kafka-poc<br>      lagThreshold: &quot;20&quot; <br>      offsetResetPolicy: earliest<br>- type: kafka<br>    metadata:<br>      bootstrapServers: 10.60.10.253:9092<br>      consumerGroup: order-shipper<br>      topic: keda_kafka<br>      lagThreshold: &quot;100&quot;<br>      offsetResetPolicy: earliest</pre><p>Here I have set the minReplica count to 1 since I want at least one pod to run every time. In case if you want to scale down to zero you can configure it accordingly.</p><blockquote>The scaling behavior can be configured through the HPA behavior field. Behaviors are specified separately for scaling up and down in scaleUp or scaleDown section under the behavior field. Unfortunately we can benefit from this feature starting from Kubernetes <a href="https://github.com/kubernetes/enhancements/blob/master/keps/sig-autoscaling/853-configurable-hpa-scale-velocity/README.md">v1.18</a> and not on any version older than this.</blockquote><pre>advanced:<br>    horizontalPodAutoscalerConfig:<br>      behavior:<br>        scaleUp:<br>          stabilizationWindowSeconds: 0<br>          policies:<br>          - type: Percent<br>            value: 50<br>            periodSeconds: 60<br>          - type: Pods<br>            value: 3<br>            periodSeconds: 60<br>          selectPolicy: Max</pre><p>Please make sure to update <em>bootstrapServers, consumerGroup , topic</em> value according to the values of the Kafka cluster used. Let’s create the scaled object now.</p><pre>kubectl apply -f kafka-scaled-object.yaml</pre><p>With this KEDA will automatically create an HPA for scaling as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TInDqfviXnHiFh0eb_K3nQ.png" /></figure><p>Now to see autoscaling in action let’s submit about 100,000 order messages to the queue.</p><pre><a href="http://192.168.64.3:32000/orders/2">http://10.68.64.3:32000/orders/</a>100000</pre><p>Once the lag goes over the threshold that we have mentioned in scaledObject the HPA kicks in and it will start scaling the pods based on the scaling behavior configured. Once the messages are consumed, the pod count will again scale into the minimum pod count specified.</p><blockquote>The number of Pods will not increase beyond the number of partitions of the kafka topic even if you set the max replica to a much higher number.</blockquote><p>Other than Kafka scaler, KEDA offers much more built-in scalers like Rabbitmq, GCP pub/sub, Redis, MYSQL, etc which you can use according to your use-case and scale your deployments or jobs accordingly.</p><p>Thanks to <strong>Zbynek Roubalik</strong> and <strong>Noah Kantrowitz </strong>for advice on Kubernetes #keda slack channel.</p><p><strong>References</strong>:</p><ol><li><a href="https://keda.sh/">https://keda.sh</a></li><li><a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/">https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/</a></li><li><a href="https://keda.sh/docs/2.2/scalers/apache-kafka/">https://keda.sh/docs/2.2/scalers/apache-kafka/</a></li></ol><p>If you enjoyed this story, please click the 👏 button and share to help others find it! Feel free to leave a comment below.</p><p>The Groww Engineering publishes technical stuff, the latest technologies, and better ways to tackle common programming problems. You can subscribe to get them <a href="https://tech.groww.in/">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c6aa27a10a80" width="1" height="1" alt=""><hr><p><a href="https://tech.groww.in/event-driven-autoscaling-in-kubernetes-using-keda-c6aa27a10a80">Event-driven Autoscaling in Kubernetes using KEDA</a> was originally published in <a href="https://tech.groww.in/">The Groww Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Visualize GCP Billing using BigQuery and Data Studio]]></title>
            <link>https://tech.groww.in/visualize-gcp-billing-using-bigquery-and-data-studio-301721cf4120?source=rss-8111d0f31642------2</link>
            <guid isPermaLink="false">https://medium.com/p/301721cf4120</guid>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[infra]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Leegin Bernads T S]]></dc:creator>
            <pubDate>Tue, 18 May 2021 07:54:00 GMT</pubDate>
            <atom:updated>2025-04-21T07:35:51.728Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1NJk7QPSPJuyMmCg.jpg" /></figure><p>One of the most common questions we get from leadership and business teams in the organization is “how can I see what I’m spending over time and which team/service is contributing to a major chunk of the cost”. Up until 2017, your only option was the <a href="https://console.cloud.google.com/billing">Billing tab</a> in your Google Cloud Console. In case if you want to get cost breakup based on teams for different services, it would require a lot of manual effort and calculations to get them.</p><p>But now, by coupling Google Cloud’s <a href="https://support.google.com/cloud/answer/7233314?hl=en">billing-export-to-BigQuery</a> functionality with <a href="http://datastudio.google.com/">Google Data Studio</a>, you can not only get up-to-date billing data/graphs throughout the day, but also you can use labels to slice and dice your GCP bill according to how you organize your business and/or applications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DNsPfcK-dnCUeAv0lzsTPg.png" /></figure><p><strong>Prerequisites</strong>:</p><ul><li><a href="https://cloud.google.com/">Google Cloud Platform</a> (GCP) account</li><li>Billing Admin permissions in GCP</li><li>Enable the BigQuery API for your project</li><li>Data Studio</li></ul><h3>Enable Billing Export to BigQuery</h3><p>The first and foremost thing we’ll need to do is to get your billing data exported into a BigQuery dataset. We will use this dataset as the data source for the graphs and queries used to build the billing report above.</p><p>You can follow the instructions in <a href="https://support.google.com/cloud/answer/7233314?hl=en">Export billing data to BigQuery</a> to set up this. Once you’ve gone through the steps there, it’ll take about a day to see the dataset begin to be populated with billing data. And once you’ve got data in your dataset, you can start running queries against it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BzOLAQT3WtkOpRMNeEIeSQ.png" /></figure><blockquote>Your BigQuery dataset will only reflect billing incurred from the date you set up billing-export, forward. Billing data will not be added retroactively, so you shouldn’t expect to see billing data from before export.</blockquote><h3><strong>Let’s build the reports….</strong></h3><p>You can slice and dice over the GCP bill according to how you organize your business and/or applications. How can I look at my bill and know what Service X is spending? You can achieve this by applying <a href="https://cloud.google.com/resource-manager/docs/using-labels">Labels</a> to your Google Cloud resources. Labels are key/value pairs that help you organize and manage things in GCP, and they’re super useful when it comes to billing. To slice through the billing, the following are some of the useful tags.</p><ul><li><strong>Labels based on the team:</strong> Add labels based on the team to distinguish resources owned by different teams. This can be used in cost accounting or budgeting. For example, <em>team: marketing</em> and <em>team: research</em>.</li><li><strong>Labels based on environment: </strong>Add labels based on environment to distinguish resources running in different environments. For example, <em>env: prod</em> and <em>env: release</em>.</li></ul><p>With the labels in place, the filters on the Billing report will allow us to split out costs for any single service or team or project, all services/teams/projects, or any combination of them.</p><p>Also, you can create custom fields in your data source to visualize things better. These custom fields are created with the help of Big query Regex and they are very useful to organize things based on the organization&#39;s requirement.</p><p>I haven’t covered how we can create the custom dashboards using Data Studio because that will require a dedicated blog.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k1qwdHZhrvOYkQXTrGiaSA.png" /></figure><p>Everything you’ve seen and read about above is freely available for you to make a copy of and work from at this <a href="https://datastudio.google.com/u/2/reporting/1MJ0GHVvcHI6cRHwMKyeSK3r7UoabEHOH/page/WXzW">link</a>. You can make your own copy of the same and start building dashboards for your organization. You can also find a copy of sample custom queries <a href="https://gist.github.com/leegin/cc98fc5a6f7f334f5574ee2b46bb01a7">here</a>.</p><h3><strong>GKE usage metering</strong></h3><p>GKE usage metering is helpful for scenarios like the following:</p><ul><li>Determining the resource consumption of a workload running in a given cluster, by assigning a unique label to the Kubernetes objects associated with the workload.</li><li>Identifying workloads whose resource requests differ significantly from their actual resource consumption so that you can more efficiently allocate resources for each workload.</li></ul><p>You need to set up two different billing export here for metering GKE usage.</p><ul><li>Billing export for BigQuery in Google Kubernetes Engine<strong>.</strong></li><li>Billing export for BigQuery in Billing account data.</li></ul><p>To verify that the GKE metric is enabled for your cluster run the below command in the Google cloud shell.</p><pre>gcloud beta container clusters describe &lt;cluster-name&gt; --format=&quot;value(resourceUsageExportConfig)&quot;</pre><p>If it is not enabled, you can turn it on for the cluster by enabling “<strong><em>GKE usage metering</em></strong>” option. By default, this will be disabled for the cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PK0ntMhYCZULXtF4_lBu4w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TG9CHlSB5mcxE-UT8onCZg.png" /></figure><p>We have created a dashboard that you can copy into your project. When you copy the dashboard, you are prompted to select two data sources you just created.</p><ol><li>Click <a href="https://datastudio.google.com/open/1JsheUOianMrAlIIyR8Uk8-HSiJX2vLHZ">this dashboard link</a>.</li><li>Click the Copy button, in the top right of the screen.</li><li>For the data sources, select the data sources that you created before. Use <a href="https://gist.github.com/leegin/864c2609ad6e7a5e0032edd776480f5b">BreakdownQuery</a> in the first field and <a href="https://gist.github.com/leegin/cc98fc5a6f7f334f5574ee2b46bb01a7">GKEProjectCostFromGCP</a> in the second.</li><li>Click <strong>Create Report</strong>.</li></ol><p>After the Google Data Studio report is created you can visualize the GKE usage based on different dimensions and analyze it.</p><h4>References:</h4><p><a href="https://cloud.google.com/billing/docs/how-to/visualize-data">https://cloud.google.com/billing/docs/how-to/visualize-data</a></p><p><a href="https://cloud.google.com/compute/docs/labeling-resources">https://cloud.google.com/compute/docs/labeling-resources</a></p><p><a href="https://cloud.google.com/billing/docs/how-to/export-data-bigquery-setup?hl=en">https://cloud.google.com/billing/docs/how-to/export-data-bigquery-setup?hl=en</a></p><p><a href="https://support.google.com/datastudio/answer/7175478?hl=en">Copy a report</a></p><p>If you enjoyed this story, please click the 👏 button and share to help others find it! Feel free to leave a comment below.</p><p>Groww Engineering publishes technical pieces, the latest knick-knacks in the coding world, and convenient ways to tackle common programming problems.</p><p>Read more of our articles <a href="https://tech.groww.in/">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=301721cf4120" width="1" height="1" alt=""><hr><p><a href="https://tech.groww.in/visualize-gcp-billing-using-bigquery-and-data-studio-301721cf4120">Visualize GCP Billing using BigQuery and Data Studio</a> was originally published in <a href="https://tech.groww.in/">The Groww Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>