<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ashok Raja on Medium]]></title>
        <description><![CDATA[Stories by Ashok Raja on Medium]]></description>
        <link>https://medium.com/@ashokrajar?source=rss-5f3aafe4f222------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*iVlbrNgmvfugjYdW.jpg</url>
            <title>Stories by Ashok Raja on Medium</title>
            <link>https://medium.com/@ashokrajar?source=rss-5f3aafe4f222------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 23:09:28 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ashokrajar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Scaling your LLM Infra Cost to Zero]]></title>
            <link>https://ashokrajar.medium.com/scaling-your-llm-infra-cost-to-zero-cd6ef7a146b9?source=rss-5f3aafe4f222------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd6ef7a146b9</guid>
            <category><![CDATA[gpu-computing]]></category>
            <category><![CDATA[ollama]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[llmops]]></category>
            <category><![CDATA[runpod]]></category>
            <dc:creator><![CDATA[Ashok Raja]]></dc:creator>
            <pubDate>Sat, 20 Jul 2024 18:35:21 GMT</pubDate>
            <atom:updated>2024-07-20T18:54:22.609Z</atom:updated>
            <content:encoded><![CDATA[<p>Who does’t like to keep their cloud billing as minimal as possible and how many dream of <em>not paying for idle resources</em>, when there is no one using it. Specially when it comes to GPU resources which are way more expensive.</p><p><strong>AI</strong> is now the industry buzz word or should I say <strong><em>The Magic Word.</em></strong> But it doesn’t come cheap, at least not yet.</p><h3><strong><em>What if </em></strong>?</h3><p>I recently stumbled upon this <em>GPU Cloud service </em><a href="https://runpod.io/"><em>runpod.io</em></a><em> . </em>My very first reaction is it’s just <strong><em>WOW wonderful</em></strong>. There are many more such services mushrooming with this AI madness.</p><blockquote>Disclaimer : Don’t get me wrong, this is just my perspective and I am not promoting any of their service or offering. This is purely my experience playing with their cloud.</blockquote><h3>How is it even Possible ?</h3><h4>Serverless !! Serverless !! Serverless !! Serverless !! Serverless !!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/381/1*RKkAJiPyOHWKgGWdyRLsOw.png" /></figure><p>Yes, we can run the LLM as a <em>Serverless Endpoint</em> and pay for only what you use.</p><p>RunPods offer an option where you can use GPU with ServerLess which autoscale based on no. of requests or request queue delay.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*40PTK2aGw1QHgyE9-oQ02Q.png" /></figure><h3>How complicated is it ? Do I need to learn a whole set of new tools/framework ?</h3><p>Thankfully no. You just need these</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/504/1*ik014gpX2Ckj0LYBmMG2wQ.png" /><figcaption>Image Sources : Google with Creative Common Licenses</figcaption></figure><ol><li><strong><em>write wrapper code in Python</em></strong></li><li>Very basic <strong><em>bash shell scripting</em></strong></li><li><strong><em>Docker</em></strong> to containerize your code</li><li>Knowledge on how to run <a href="https://ollama.com/">ollama</a> models. (If you have read so far you I am assuming you must be familiar with it)</li></ol><p>You don’t even need to be champion on these techs just very basic understanding and exprience is enough.</p><h3>Quick Demo</h3><p>I am not going to bore you with writing code etc, you can find all the code required to build this project in my GitHub. Or even the completed final Docker image in my Docker Hub page.</p><p><em>GitHub Code Repo : </em><a href="https://github.com/ashokrajar/runpod-ollama-serverless"><em>https://github.com/ashokrajar/runpod-ollama-serverless</em></a></p><p><em>Docker Hub Final Image Repo : </em><a href="https://hub.docker.com/repository/docker/ashokrajar/runpod-ollama-serverless/tags"><em>https://hub.docker.com/repository/docker/ashokrajar/runpod-ollama-serverless/tags</em></a></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F2AdgBh3YgBU%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D2AdgBh3YgBU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F2AdgBh3YgBU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/6e0b528d12364368c7168ae528620501/href">https://medium.com/media/6e0b528d12364368c7168ae528620501/href</a></iframe><p>Above request costed me only <strong><em>$0.08/request</em></strong>. It can go even further down when multiple requests are made as with Serverless there is <em>cold start time. </em>When multiple requests are made following requests <strong><em>doesn’t require cold start time</em></strong> and it will be <strong><em>even more cheaper</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YKBpytIClnnPXBUi7EXyHA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pgdL1AbGqdp3MeZ-_NfoHg.png" /></figure><blockquote><em>If you have made up your mind to give RunPod.IO a try kindly use my referal link </em><a href="https://runpod.io/?ref=xq02lxa1"><em>https://runpod.io?ref=xq02lxa1</em></a><em> and help me write more such content.</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd6ef7a146b9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scaling your Kubernetes PODS to ZERO and BACK with K8s & KEDA]]></title>
            <link>https://ashokrajar.medium.com/scaling-your-kubernetes-pods-to-zero-and-back-with-k8s-keda-d0f8b4fdb8ab?source=rss-5f3aafe4f222------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0f8b4fdb8ab</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cost-savings]]></category>
            <category><![CDATA[keda]]></category>
            <category><![CDATA[autoscaling]]></category>
            <category><![CDATA[scalability]]></category>
            <dc:creator><![CDATA[Ashok Raja]]></dc:creator>
            <pubDate>Sun, 24 Dec 2023 17:59:37 GMT</pubDate>
            <atom:updated>2023-12-24T17:59:37.500Z</atom:updated>
            <content:encoded><![CDATA[<p>In this post let see how we can <strong><em>dynamically</em></strong> <strong><em>scale</em></strong> your Data Processing or Data Oriented applications to <strong><em>Zero</em></strong> resources and back to the <strong><em>desired capacity</em></strong> only when there is a demand using modern infrastructure tools like <strong><em>K8s</em></strong><em>(Kubernetes</em>) &amp; <strong><em>KEDA</em></strong><em>(Kubernetes Event-driven Autoscaling)</em>.</p><p>Also we will cover the major <strong><em>pitfall</em></strong> of <strong><em>KEDA</em></strong> in <strong><em>Scaling IN/to-Zero </em></strong><em>and how we can handle it.</em></p><blockquote>Note : Installation of K8s, KEDA &amp; Kafka are out of scope for this post as you can find numerous post/videos covering them.</blockquote><h3>Why scale to ZERO ?</h3><ul><li><em>Don’t we all want to </em><strong><em>spend less</em></strong><em> ?</em></li><li><em>Why keep the infrastructure up and running when it’s </em><strong><em>idle</em></strong><em> ?</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*g4vtF8ia9nc9wnk5" /><figcaption>Photo by <a href="https://unsplash.com/@constantinevdokimov?utm_source=medium&amp;utm_medium=referral">Konstantin Evdokimov</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>A Simple Application</h3><p>Let consider a simple demo app for this post. Don’t worry you can easily apply this method/solution to any application of any scale/complexity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-7DiN9_p1paGOBf6_BMRHw.png" /></figure><p>Let’s see how we can scale the Data Processing application to ZERO when there is no data to process and scale to the desired no. of servers/replicas when the messages starts to flow and pile up.</p><blockquote>Code : <a href="https://github.com/ashokrajar/k8s-keda-kafka-demo">https://github.com/ashokrajar/k8s-keda-kafka-demo</a></blockquote><h3>Back to School</h3><p>Before we go in the solution and implementation, let’s take a step back to understand how <strong><em>K8s</em></strong> &amp; <strong><em>KEDA </em></strong><em>works together to </em><strong>scale-in/scale-out </strong>the <strong>pods</strong>(application containers).</p><h4>What is the role of KEDA ?</h4><ul><li>KEDA handovers the actual POD scaling to K8s native HPA based on custom metrics.</li><li>And KEDA publishes the necessary custom metrics listenting to Kafka topics and it’s LAG.</li></ul><h4>How K8s HPA Works ?</h4><p>Kubernetes HPA(Horizontal Pod Scaler) has a <strong><em>complex Algorithm</em></strong> based on various metric which itself calls for a separate blog post.</p><p>To put it simple….</p><blockquote><strong>desiredReplicas</strong> = ceil[<strong>currentReplicas</strong> * ( <strong>currentMetricValue</strong> / <strong>desiredMetricValue</strong> )]</blockquote><p>And the <em>currentMetricsValue</em> is calculated from</p><blockquote><strong><em>currentMetricValue = currentKafkaTopicTotalLag / currentRepicaCount</em></strong></blockquote><p>You can read more about K8s HPA <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#algorithm-details">here</a>. With this simple algorithm in mind let’s jump in to solution and implementation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BLIC16R-D0Q_l7bYluh7KQ.png" /><figcaption>Time lines is defined like T1(currentKafkaTopicTotalLag/currentMetricValue/desiredMetricValue)</figcaption></figure><h3>Enabling KEDA</h3><pre>---<br>apiVersion: keda.sh/v1alpha1<br>kind: ScaledObject<br>metadata:<br>  name: kafka-autoscale-demo<br>  namespace: keda-kafka-demo<br><br>spec:<br>  scaleTargetRef:<br>    name: keda-kafka-demo-consumer<br>  pollingInterval: 5<br>  cooldownPeriod: 120<br>  idleReplicaCount: 0<br>  minReplicaCount:  1<br>  maxReplicaCount: 5<br>  triggers:<br>    - type: kafka<br>      metadata:<br>        bootstrapServers: kafka.ashokraja.in:9092<br>        consumerGroup: k8-keda-kafka-demo-consumer<br>        topic: k8s_keda_autoscale_testing<br>        offsetResetPolicy: latest<br>        lagThreshold: &#39;20&#39;<br>        activationLagThreshold: &#39;0&#39;</pre><h4>Wow Magic happens and everything works as expected in a perfect world !!!</h4><p>The message flow is 1:1 or close to it. What it means is consumers are able to consume the messaged at the same rate the producers are producing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nJt28NhNy_VP2lW9u6tu2A.png" /></figure><h3>Real World Challenge (All Hell lose !!)</h3><p>KEDA’s defaults works great when the messages are consumed and processed without any delay by the consumer and are <strong>committing back</strong> to unblock the Kafka Rebalancing so there are no idle consumers.</p><blockquote>But <strong><em>that is not true</em></strong> for an ETL jobs or a workflow or a pipeline which further has to do a lot scrapping/processing/computing/transformation based on the consumed messages. Messages may be produced in a predictable manner.</blockquote><blockquote>But the consumer are not consuming or the consumers are <strong>committing back</strong> to unblock the Kafka Rebalancing so there are no idle consumers.</blockquote><p>In such scenario <strong><em>K8s/KEDA/HPA detects that the desired number of pods are way too high</em></strong> not having any knowledge of the consumers which are still processing those messages and <strong><em>starts to Scale Down </em></strong>the pods. And since it doesn’t know which pods are still processing the messages it <strong><em>picks the pod randomly</em></strong> and kills it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UKAXtopX8jDHrsunxK51qg.png" /></figure><h3>Every problem has a Solution</h3><h4>Solution 1 : <strong><em>Delay K8s HPA Scale Down as long as you desire</em></strong></h4><p>Ref: <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#example-disable-scale-down">https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#example-disable-scale-down</a></p><pre>advanced:<br>    horizontalPodAutoscalerConfig:<br>      behavior:<br>        scaleDown:<br>          selectPolicy: Disabled</pre><blockquote>Complete Manifest sample:</blockquote><pre>---<br>apiVersion: keda.sh/v1alpha1<br>kind: ScaledObject<br>metadata:<br>  name: kafka-autoscale-demo<br>  namespace: keda-kafka-demo<br><br>spec:<br>  scaleTargetRef:<br>    name: keda-kafka-demo-consumer<br>  pollingInterval: 5<br>  cooldownPeriod: 120<br>  idleReplicaCount: 0<br>  minReplicaCount:  1<br>  maxReplicaCount: 5<br>  advanced:<br>    horizontalPodAutoscalerConfig:<br>      behavior:<br>        scaleDown:<br>          selectPolicy: Disabled<br>  triggers:<br>    - type: kafka<br>      metadata:<br>        bootstrapServers: kafka.ashokraja.in:9092<br>        consumerGroup: k8-keda-kafka-demo-consumer<br>        topic: k8s_keda_autoscale_testing<br>        offsetResetPolicy: latest<br>        lagThreshold: &#39;20&#39;<br>        activationLagThreshold: &#39;0&#39;</pre><p>With this <strong><em>tweak</em></strong> in place the pods are <strong><em>not scaled down unless the </em></strong><strong><em>Lag = 0. </em></strong>Also pay attention to cooldownPeriod = 120sec , tweak this to your desired value depending on how long a consumer is going to process the messages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tT9T57BFAjitSTfepIiuSg.png" /></figure><h4>Solution 2 : use ScaledJobs instead of ScaledObjects</h4><p>For very long running jobs say for some weird reason your consumer will be running for hours or say even days ;).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0f8b4fdb8ab" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Docker/K8s & Linux Netfilter’s conntrack Nightmare]]></title>
            <link>https://ashokrajar.medium.com/docker-k8s-linux-netfilters-conntrack-nightmare-dffa9a34b80c?source=rss-5f3aafe4f222------2</link>
            <guid isPermaLink="false">https://medium.com/p/dffa9a34b80c</guid>
            <category><![CDATA[netfilter]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[scalability]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[linux]]></category>
            <dc:creator><![CDATA[Ashok Raja]]></dc:creator>
            <pubDate>Fri, 12 May 2023 08:34:54 GMT</pubDate>
            <atom:updated>2023-05-13T18:19:32.758Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NVWUR1T8E3PsT5rKWPVTUQ.png" /></figure><p>Recently I happen to deeply learn &amp; investigate about Linux kernels Netfilter modules connection tracking limits and how it can impact the performance of any application running inside a Container.</p><h3>What is Netfilter conntrack ?</h3><p><strong><em>Netfilter</em></strong> is a firewall subsystem which is part of the Linux network stack which is commanly managed by a cli called <strong><em>iptables</em></strong>. In early days Netfilter was implemented as a stateless but quickly became insufficient, so Netfilter implemented a statefull firewall which keeps record of all the connections.</p><pre>sudo conntrack -L<br>tcp 6 1 TIME_WAIT src=150.136.89.255 dst=192.168.2.2 sport=55824 dport=22 src=192.168.2.2 dst=150.136.89.255 sport=22 dport=55824 [ASSURED] use=1<br>udp 17 13 src=127.0.0.1 dst=127.0.0.53 sport=59443 dport=53 [UNREPLIED] src=127.0.0.53 dst=127.0.0.1 sport=53 dport=59443 use=1<br>tcp 6 431999 ESTABLISHED src=192.168.2.30 dst=192.168.2.2 sport=61167 dport=22 src=192.168.2.2 dst=192.168.2.30 sport=22 dport=61167 [ASSURED] use=1<br>udp 17 14 src=192.168.2.2 dst=192.73.248.83 sport=41641 dport=3478 src=192.73.248.83 dst=192.168.2.2 sport=3478 dport=41641 use=1<br>udp 17 13 src=192.168.2.2 dst=102.67.165.90 sport=41641 dport=3478 src=102.67.165.90 dst=192.168.2.2 sport=3478 dport=41641 use=1<br>udp 17 13 src=127.0.0.1 dst=127.0.0.53 sport=35164 dport=53 [UNREPLIED] src=127.0.0.53 dst=127.0.0.1 sport=53 dport=35164 use=1<br>udp 17 13 src=127.0.0.1 dst=127.0.0.53 sport=41767 dport=53 [UNREPLIED] src=127.0.0.53 dst=127.0.0.1 sport=53 dport=41767 use=1<br>.<br>truncate for readabilty</pre><h3>What is the role of Netfilter in the Container World ?</h3><p>We know that containers are nothing but simple processes/code running in an isolated environment inside the Linux Operating System.</p><p>Netfilter provide <strong><em>network isolation</em></strong> for the Docker containers.</p><h3>How can Netfilter impact containerized applications performance ?</h3><p>Netfilter has a lot of configurations which can be refered <a href="https://www.kernel.org/doc/html/v5.8/networking/nf_conntrack-sysctl.html">here</a>.<br>For this post let’s only focus just on one very important configuration <em>nf_conntrack_max</em>.</p><blockquote>nf_conntrack_max : Size of connection tracking table. Default value depends on available memory, but what I have observed is it varies based on the distribution. eg: On a ubuntu system defaults is 65536, some other distribution or cloud provider have different defaults.</blockquote><p>For this article let’s go with Ubuntu’s default <strong><em>65536</em></strong></p><p>What this means is our application container hosted(running) on this machine/vm can make <strong><em>maximum 65536 connections in/out</em></strong>.</p><p>To simplify let’s say you have just one container running on a single VM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kyZ-vhE1MRYl13XYm3sZrw.png" /></figure><p>We might not see the problem yet on this because we might hit the CPU limit before we even consume all the network connections.<br>Also in production with autoscaling kicking in you will have more VMs running your container.</p><p>Let’s take a better example, consider this deployment model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9ILykCJpnHfJCVi6wk4Qwg.png" /></figure><p>In this architecture let’s still keep the assumption that all the components are autoscaled.</p><blockquote>But if you look carefully the <strong>Haproxy Layer</strong> will soon become a <strong>bottle neck</strong> with growing traffic, especially when you have &gt; <strong>millions of requests</strong> pouring in.</blockquote><blockquote><strong>Once this limit is reached server simply drops the imcoming connections.**</strong></blockquote><h3>How to detect ?</h3><p>Linux kernels dmesg to the rescue. You will see the kernel logs flooded with <em>table full, dropping packet</em> messages.</p><pre>sudo dmesg -T<br>[Thu May 11 12:38:28 2023] net_ratelimit: 59 callbacks suppressed<br>[Thu May 11 12:38:28 2023] nf_conntrack: nf_conntrack: table full, dropping packet<br>[Thu May 11 12:38:28 2023] nf_conntrack: nf_conntrack: table full, dropping packet<br>[Thu May 11 12:38:28 2023] nf_conntrack: nf_conntrack: table full, dropping packet<br>[Thu May 11 12:38:28 2023] nf_conntrack: nf_conntrack: table full, dropping packet<br>[Thu May 11 12:38:28 2023] nf_conntrack: nf_conntrack: table full, dropping packet<br>[Thu May 11 12:38:28 2023] nf_conntrack: nf_conntrack: table full, dropping packet<br>[Thu May 11 12:38:28 2023] nf_conntrack: nf_conntrack: table full, dropping packet<br>[Thu May 11 12:38:28 2023] nf_conntrack: nf_conntrack: table full, dropping packet<br>[Thu May 11 12:38:28 2023] nf_conntrack: nf_conntrack: table full, dropping packet<br>[Thu May 11 12:38:28 2023] nf_conntrack: nf_conntrack: table full, dropping packet<br>[Thu May 11 12:38:33 2023] net_ratelimit: 66 callbacks suppressed<br>[Thu May 11 12:38:33 2023] nf_conntrack: nf_conntrack: table full, dropping packet<br>[Thu May 11 12:38:33 2023] nf_conntrack: nf_conntrack: table full, dropping packet<br>[Thu May 11 12:38:33 2023] nf_conntrack: nf_conntrack: table full, dropping packet<br>[Thu May 11 12:38:33 2023] nf_conntrack: nf_conntrack: table full, dropping packet<br>[Thu May 11 12:38:33 2023] nf_conntrack: nf_conntrack: table full, dropping packet<br>[Thu May 11 12:38:33 2023] nf_conntrack: nf_conntrack: table full, dropping packet<br>[Thu May 11 12:38:33 2023] nf_conntrack: nf_conntrack: table full, dropping packet<br>[Thu May 11 12:38:33 2023] nf_conntrack: nf_conntrack: table full, dropping packet<br>[Thu May 11 12:38:33 2023] nf_conntrack: nf_conntrack: table full, dropping packet<br>[Thu May 11 12:38:34 2023] nf_conntrack: nf_conntrack: table full, dropping packet</pre><h3>The Fix ?</h3><p>Haproxy here is just acting as a Layer 4/7 LoadBalancer which will have a very low CPU/Mem resource usage and will not be scaled very often and <strong><em>pretty soon the connections will reach the default Netfilter Connection tracking limit</em></strong>.</p><h4>Fix 1 (The costly fix)</h4><blockquote>Alright why don’t we scale haproxy based on total conenctions ? then we will be burning <strong>more $$$$$</strong> while we have a very simple solution to this very problem.</blockquote><h4>Fix 2 (A better one)</h4><p>Just increase the connection tracking limits, but also keep your system/container memory in the calculation.</p><p>In my experience I have seen we can easily bump up to 600000 on a VM with just 1GB memory and not have any side effects. eg:</p><pre>echo 600000 &gt; /proc/sys/net/netfilter/nf_conntrack_max</pre><p>To make it persist on system reboot, add the bellow config in <em>/etc/sysctl.conf</em></p><pre># Maximum number of allowed connection tracking entries<br>net.netfilter.nf_conntrack_max = 600000</pre><h4>A similar nf_conntrack story</h4><ul><li><a href="https://blog.cloudflare.com/conntrack-tales-one-thousand-and-one-flows/](https://blog.cloudflare.com/conntrack-tales-one-thousand-and-one-flows/)">https://blog.cloudflare.com/conntrack-tales-one-thousand-and-one-flows/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dffa9a34b80c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Observability with InfluxDB Cloud in 3 Steps]]></title>
            <link>https://ashokrajar.medium.com/observability-with-influxdb-cloud-in-3-steps-3e8298255e1b?source=rss-5f3aafe4f222------2</link>
            <guid isPermaLink="false">https://medium.com/p/3e8298255e1b</guid>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[telegraf]]></category>
            <category><![CDATA[influxdb]]></category>
            <category><![CDATA[telemetry]]></category>
            <dc:creator><![CDATA[Ashok Raja]]></dc:creator>
            <pubDate>Sun, 28 Mar 2021 16:51:43 GMT</pubDate>
            <atom:updated>2021-03-28T16:51:43.269Z</atom:updated>
            <content:encoded><![CDATA[<p>In this post let see how we can do Obervability using InfluxDB Cloud. For this experiment we are going to use Two x86 and One ARM system.</p><blockquote>As a <strong>SRE</strong> after setting up my <a href="https://ashokrajar.medium.com/building-a-micro-datacenter-at-home-46482da57b7d">Mini Home DataCenter</a>, my first priority is to setup monitoring and have <strong>Eyes on my InfraStructure and Services</strong>.</blockquote><h4>My customised Observability Dashboard</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*crf_uJQgZOe54cobSIUDfA.png" /></figure><h3>Infra</h3><p><strong>For x86 we have choosen:</strong><br>2 x <em>emini NUC Mini PC<br>Compute : i7 8th Gen 8750H (6 Core), 9M Cache, 2.2GHz upto 4.1GHz Turbo <br>Memory : 16GB RAM<br>Storage : 1 TB HDD, 1TB Software RAID1 (2 SanDisk </em>Portable SSD USB3<em>)</em></p><p><strong>For ARM we have chosen:</strong><br>1 x <em>Raspberry Pi 4<br>Compute : Quad core Cortex-A72 (ARM v8) 64-bit SoC @ 1.5GHz<br>Memory : 4GB RAM<br>Storage : 32B MicroSD, 5TB Segate Portable HDD, 4TB Segate Portale HDD</em></p><p>You can find a more detailed info of this setup in my another post =&gt; <a href="https://ashokrajar.medium.com/building-a-micro-datacenter-at-home-46482da57b7d"><strong>Building a Micro DataCenter at Home</strong></a></p><h3>InfluxDB Cloud 2.0</h3><p>InfluxDB is a scalable time series database from <a href="https://www.influxdata.com/">https://www.influxdata.com</a>. We are going to use their cloud option. We are going with free tier which allows write of 5MB/5 minutes and query of 300MB/4 minutes which is more that enough for our HomeLab setup.</p><h3>Step 1 : Sign Up</h3><p>Sign up for the Influx Cloud 2.0 Free accout here =&gt; <a href="https://cloud2.influxdata.com/signup"><strong>Influx Cloud Get Started</strong></a></p><h4>Create a Bucket</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MAMICwjXqN1Io0iLC1bE2g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V1tsUQu_3EaA6kJAEswK1g.png" /></figure><h4>Create Agent Config and Generate a API Token</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CZ5gZw9kpSzc3hHB681hJg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BA_JpgzxLiODzu4VJBHxiQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xsoxCPzYjiAYEugwN-wXaw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZUWd35GbMVmIVPSieLXxBA.png" /></figure><p>Copy the token and keep it handy for <strong>Step-3.</strong></p><p>Open the configuration URL which in my case is : <a href="https://us-west-2-1.aws.cloud2.influxdata.com/api/v2/telegrafs/074a65d98a3fd000">https://us-west-2-1.aws.cloud2.influxdata.com/api/v2/telegrafs/074a65d98a3fd000</a>. It will prompt you to download.</p><p>Rename the download file from <strong>074a65d98a3fd000</strong> → <strong>telegraf.toml</strong></p><p>Copy the <strong>telegraf.toml </strong>to all the machine using SCP or a any other method of your choice.</p><h3>Step 2 : Install the Telegraf Agent</h3><p>Telegraf is an agent written in Go for collecting metrics and writing them into InfluxDBor other possible outputs.</p><p>There are <a href="https://docs.influxdata.com/telegraf/v1.18/introduction/installation/#installation"><strong>native installation</strong></a> methods available for most of the popular Operating Systems like, Ubuntu/Debain, RHEL/Centos, Suse, BSD, Mac and Windows.</p><p>But for this post we are going to install telegraf from a <a href="https://portal.influxdata.com/downloads/">Binary Release</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*awyBQJpA5pdR-sEtFjxYaQ.png" /></figure><h4>Installing on a x86 Linux</h4><p>All the steps are run as root user</p><pre>shell # wget https://dl.influxdata.com/telegraf/releases/telegraf-1.18.0_linux_amd64.tar.gz<br>tar xf telegraf-1.18.0_linux_amd64.tar.gz</pre><pre>shell # cd telegraf-1.18.0</pre><pre>shell # mv usr/bin/telegraf /usr/bin/telegraf</pre><pre>shell # mv usr/lib/telegraf/scripts/telegraf.service /etc/systemd/system/telegraf.service</pre><pre>shell # systemctl daemon-reload</pre><pre>shell # mkdir /etc/telegraf</pre><h4>Installing on a ARM (Raspberry PI4)</h4><pre>shell # wget https://dl.influxdata.com/telegraf/releases/telegraf-1.18.0_linux_armhf.tar.gz<br>tar xf telegraf-1.18.0_linux_armhf.tar.gz</pre><pre>shell # cd telegraf-1.18.0</pre><pre>shell # mv usr/bin/telegraf /usr/bin/telegraf</pre><pre>shell # mv usr/lib/telegraf/scripts/telegraf.service /etc/systemd/system/telegraf.service</pre><pre>shell # systemctl daemon-reload</pre><pre>shell # mkdir /etc/telegraf</pre><h3>Step 3: Configure the Telegraf Agent</h3><p>Configuration is same for both x86 and ARM based system.</p><pre>shell # echo &quot;INFLUX_TOKEN=&lt;paste the token copied in Step-1&gt;&quot; &gt; /etc/default/telegraf</pre><pre>shell # mv <strong>telegraf.toml </strong>/etc/telegraf/</pre><pre>shell # systemctl start telegraf.service<br>shell # systemctl status telegraf.service</pre><p>That’s it you should now see the metrics pouring in and there already a <strong>default System Dashboard</strong> setup for you already with some cool metrics to Observe.</p><h4>Default System Dashboard created automatically</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LGZkzrnmDp7inDH7LpdKCw.png" /><figcaption>Automatically created default System Dashboard</figcaption></figure><p>You can get creative from here and create some even cooler dashboard.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e8298255e1b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Micro DataCenter at Home]]></title>
            <link>https://ashokrajar.medium.com/building-a-micro-datacenter-at-home-46482da57b7d?source=rss-5f3aafe4f222------2</link>
            <guid isPermaLink="false">https://medium.com/p/46482da57b7d</guid>
            <category><![CDATA[mini-data-center]]></category>
            <category><![CDATA[home-data-center]]></category>
            <category><![CDATA[virtualization]]></category>
            <category><![CDATA[diy]]></category>
            <category><![CDATA[data-center]]></category>
            <dc:creator><![CDATA[Ashok Raja]]></dc:creator>
            <pubDate>Sun, 28 Mar 2021 09:11:09 GMT</pubDate>
            <atom:updated>2021-03-28T09:11:09.732Z</atom:updated>
            <content:encoded><![CDATA[<p>Creating a mini Data Center at home has been my long term dream. Since my ISP gave me a static IP very cheap and finally was able to spend some $$$ to afford it and most importantly got some time to set it up.</p><h3>Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LQ9KsKgoU-Nb4B6jzm1BZA.png" /></figure><h3>Hardware</h3><p><strong><em>1 </em></strong>x<strong><em> emini NUC Mini PC — Metal01</em></strong><em><br>Compute : i7 8th Gen 8750H (6 Core), 9M Cache, 2.2GHz upto 4.1GHz Turbo <br>Memory : 32GB RAM<br>Storage : 1 TB HDD, 1TB Software RAID1 (2 </em>x<em> </em>SSD USB3<em>), 2 </em>x<em> 5TB usb HDD, 1 </em>x<em> 2TB usb HDD</em></p><p><strong><em>1 </em></strong>x<strong><em> emini NUC Mini PC — Metal02</em></strong><em><br>Compute : i7 8th Gen 8750H (6 Core), 9M Cache, 2.2GHz upto 4.1GHz Turbo <br>Memory : 32GB RAM<br>Storage : 1 TB HDD, 1TB Software RAID1 (2 </em>x<em> </em>SSD USB3<em>)</em></p><p><strong><em>1 </em></strong>x<strong><em> Raspberry Pi 4 — Atom</em></strong><em><br>Compute : Quad core Cortex-A72 (ARM v8) 64-bit SoC @ 1.5GHz<br>Memory : 4GB RAM<br>Storage : 32GB MicroSD, 1 </em>x<em> 5TB Segate Portable HDD, 1 </em>x<em> 4TB Segate Portale HDD</em></p><p><strong><em>1 </em></strong>x<strong><em> Apple Mac Mini — Main PC</em></strong><em><br>Compute : </em>6-core Intel Core i7, 12MB shared L3 cache, 3.2GHz<em> upto 4.6GHz Turbo <br>Memory : 32GB RAM<br>Storage : 1 TB SSD, 1 </em>x<em> 2TB NVMe SSD<br>Keyboard : </em><a href="https://www.keychron.com/products/keychron-k2-wireless-mechanical-keyboard"><em>Keychron K2</em></a><em><br>Mouse : </em><a href="https://www.logitech.com/en-us/products/mice/mx-master-3.910-005620.html"><em>Logitech MX Master 3</em></a></p><p><strong><em>Network : 1 </em></strong>x<strong><em> 8 Port Gigabit Switch</em></strong><em><br></em><strong><em>Router : 1 </em></strong>x<strong><em> Asus 5ghz Gigabit Router</em></strong></p><h3>Infra Setup</h3><h4>Router, Switch and Raspberry Pi 4</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*gVBE_CbfXT7ec7LPxeMzMg.png" /></figure><h4>Metal01 &amp; Metal02</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*NeWuaoy_bcRL8VhXvQGF7g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*Yoahnn8nh2f-a3Vp7VONsQ.png" /></figure><h4>Main PC, Monitor &amp; GPU</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*TuLN0NWJXqfvjUOwX1xIag.png" /></figure><h3>Alright What I use it for ??</h3><p>I use this for various purpose. Learning, Experimenting new technology &amp; POCs, Media storage &amp; Streaming 4K videos to Android and Apple TV, Backups etc..</p><h4>Metal01 — Core VMs</h4><ul><li>cmdcenter — a bastion server, Haproxy server to expose some internal service like (VS Code Server, Proxmox KVM UI, Firefly III UI etc)</li><li>Core K8 Cluster — Running Firefly III (Personal Finance Manager)</li><li>Windows 10 — Once in a while need this for some odd reasons</li><li>NAS Server VM — All the storage is connected to this VM and shared via Samba, NFS, AFP.</li></ul><h4>Metal02 — Other VMs</h4><ul><li>Experimental K8 Cluster</li><li>clitools — Multiuser env with all the cloud cli tools installed</li><li>Few other POC VMs</li></ul><h4>Atom</h4><ul><li><a href="https://dev.deluge-torrent.org/">Deluged</a> Torrent Client with a WebUI</li><li><a href="https://www.plex.tv/media-server-downloads/">Plex Media Server</a></li><li>TimeMachine Server</li></ul><h4>Main PC</h4><p>This is my main computer I use daily for all things(Writing Code, Mail, Teams, Slack, Browsing, Editing Pictures I shoot, Writing this Blog etc)</p><ul><li><a href="https://github.com/cdr/code-server">VS Code Server</a> (Web IDE based on Microsoft VS Code)</li><li>Docker</li><li>All the IDEs Installed</li><li>All developer tools and languages Installed</li><li>All messaging and meeting applications</li><li>Various browsers. Though Safari and <a href="https://vivaldi.com/">Vivaldi</a> are my browser of choice.</li><li>Lightroom CC, Photoshop CC</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=46482da57b7d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multi Stage Azure PipeLine for Golang]]></title>
            <link>https://ashokrajar.medium.com/multi-stage-azure-pipeline-for-golang-69261e984621?source=rss-5f3aafe4f222------2</link>
            <guid isPermaLink="false">https://medium.com/p/69261e984621</guid>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[azure-devops]]></category>
            <category><![CDATA[azure-pipelines]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[azure-pipeline-template]]></category>
            <dc:creator><![CDATA[Ashok Raja]]></dc:creator>
            <pubDate>Tue, 28 Apr 2020 21:23:07 GMT</pubDate>
            <atom:updated>2020-04-28T21:23:07.951Z</atom:updated>
            <content:encoded><![CDATA[<p>I have been playing around with Azure DevOps as part of my experiment with various CI/CD tools. I will be sharing my views and comparison of various CI/CD tools in a separate post. For now lets focus on how we can create a templatized <strong>Multi Staged Azure Pipeline.</strong></p><blockquote>GitHub Repo : <a href="https://github.com/ashokrajar/mylabs-go">https://github.com/ashokrajar/mylabs-go</a></blockquote><blockquote>Multi Stage Azure Pipeline Demo : <a href="https://dev.azure.com/ashokrajar/testpad/_build?definitionId=3&amp;_a=summary">https://dev.azure.com/ashokrajar/testpad</a></blockquote><p>We will be building a simple Golang cli application for demo purpose.</p><pre>shell # ./<a href="https://github.com/ashokrajar/mylabs-go">mylabs-go</a><br>Hello Home !!<br>shell # ./mylabs-go version<br>0.2.1<br>shell #</pre><p>Let’s demonstrate how we do <strong>Build =&gt; Test =&gt; Deploy</strong>(In our case it will be <strong>Release Binary Artifact</strong>) using Azure Multi Stage build.</p><p>What all <strong>Tests</strong> will we be running ?</p><ul><li>Unit Tests</li><li>Code Coverage</li><li>Vulnerability Test (we will be using <a href="https://www.shiftleft.io/">ShiftLeft</a>)</li></ul><p>All these tests will be executed in parallel for multiple versions of Golang.</p><h3>Our GOAL</h3><p>Create a dynamically configurable Azure Pipeline for Golang application for multiple platforms.</p><blockquote>Multi Stage Azure Pipeline Demo : <a href="https://dev.azure.com/ashokrajar/testpad/_build?definitionId=3&amp;_a=summary">https://dev.azure.com/ashokrajar/testpad</a></blockquote><h4>A Multi Stage build which looks like this</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y6lh2R9uM6BYVOMr8hLZsg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8FtzjFf39CXh1F-Pz_MzEQ.png" /><figcaption>Pipeline View</figcaption></figure><h4>Embedded Test Results</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H4JfLFrj3WKTkQea0mowsQ.png" /></figure><h4>Embedded Code Coverage Results</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UiTKuLAqWHvOs4bAxXq8fw.png" /></figure><h4>Embedded Vulnerability Scan Reports</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vqmopa_VnpeAHnYMWaTe1g.png" /></figure><p>Alright enough showing off let’s get our hands dirty.</p><h3>Design/Create a Azure Pipeline to achieve our GOAL</h3><h4>Step 1 : Create <strong>folders/files hierarchy</strong></h4><p>For creating reusable template we need a proper <strong>folders/files hierarchy design</strong>.</p><pre>├── azure-pipelines.yml<br>├── go.mod<br>├── main.go<br>├── main_test.go<br>├── shiftleft.yml<br>└── templates<br>    └── azure<br>        ├── jobs<br>        │   ├── build.yml<br>        │   ├── release.yml<br>        │   └── test.yml<br>        └── steps<br>            ├── buildapp.yml<br>            └── setupgo.yml</pre><h4>Step 2 : Create azure-pipeline.yml</h4><p>We have designed a pipeline config which will <strong>trigger the builds</strong> for commits to <strong>master, dev &amp; release/*</strong> branches and also or <strong>pull request to master </strong>branch. At the same time it will build will <strong>not be triggered</strong> for changes to non-project files.</p><pre>trigger:<br>  batch: true  # Ensure batch execution for very active repos<br>  branches:<br>    include:<br>      - master<br>      - dev<br>      - release/*<br>  paths:<br>    exclude:<br>      - README.md<br>      - .gitignore<br><br>pr:<br>  autoCancel: True  # Auto cancel if active pull request updated<br>  branches:<br>    include:<br>      - master<br>  paths:<br>    exclude:<br>        ..... removed for brevity .....</pre><pre>variables:<br>  GOPATH: &#39;$(Pipeline.Workspace)/gowork&#39;<br><br>stages:<br># We will be building stages in following steps</pre><p>Before we start creating multiple stages lets create some reusable templates.</p><h4>Step 3 : Creating Shared Templates</h4><p>Azure provides a powerful templating functionality which let you define reusable content, logic, and parameter.</p><p><strong><em>Template : templates/azure/steps/setupgo.yml</em></strong></p><pre>parameters:<br><strong>  goVersion</strong>: &#39;1.14&#39;<br><br>steps:<br>  - task: GoTool@0<br>    displayName: &#39;Use Go ${{ parameters.goVersion }}&#39;<br>    inputs:<br>      version: ${{ parameters.goVersion }}<br><br>  - script: |<br>      set -e -x<br>      mkdir -p &#39;$(GOPATH)/bin&#39;<br>      echo &#39;##vso[task.prependpath]$(GOROOT)/bin&#39;<br>      echo &#39;##vso[task.prependpath]$(GOPATH)/bin&#39;<br>    displayName: &#39;Create Go Workspace&#39;</pre><p>This will setup the Golang workspace with the default version of <strong>1.14</strong>. Which can be overridden when calling the template from the pipeline config files.</p><p><strong><em>Template : templates/azure/steps/buildapp.yml</em></strong></p><pre>steps:<br>  - task: Go@0<br>    displayName: &#39;Build Application Binary&#39;<br>    inputs:<br>      command: &#39;build&#39;<br>      workingDirectory: &#39;$(System.DefaultWorkingDirectory)&#39;<br>      arguments: &#39;-o $(Build.BinariesDirectory)/mylabs-go&#39;</pre><p>Similarly this template helps build the Golang application.</p><p><strong><em>Template : templates/azure/jobs/build.yml</em></strong></p><pre><strong>parameters:<br>  name: </strong>&#39;&#39;<strong><br>  pool: </strong>&#39;&#39;<br><br>jobs:<br>  - job: ${{ parameters.name }}<br>    pool: ${{ parameters.pool }}<br>    steps:<br>      - template: ../steps/setupgo.yml<br><br>      - template: ../steps/buildapp.yml</pre><p>This template will help executing of the build stage of the the Golang application.</p><p>But Wait ! What ? More templates inherited inside the template ? <em>YES!, </em>we are just make using the templates we designed in the previous steps</p><p><strong>Template : templates/azure/jobs/test.yml</strong></p><pre>jobs:<br>  - job: RunTests<br>    strategy:<br>      matrix:<br>        GoVersion_1_13:<br>          go.version: &#39;1.13&#39;<br>        GoVersion_1_14:<br>          go.version: &#39;1.14&#39;<br><br>    pool:<br>      vmImage: &#39;ubuntu-18.04&#39;<br><br>    steps:<br>      - template: ../steps/setupgo.yml<br>        parameters:<br>          goVersion: &#39;$(go.version)&#39;<br><br>      - script: |<br>          set -e -x<br>          go version<br>          go get -u github.com/jstemmer/go-junit-report<br>          go get github.com/axw/gocov/gocov<br>          go get github.com/AlekSi/gocov-xml<br><br>          curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin v1.24.0<br>          curl https://cdn.shiftleft.io/download/sl &gt; $(go env GOPATH)/bin/sl &amp;&amp; chmod a+rx $(go env GOPATH)/bin/sl<br>        displayName: &#39;Install Dependencies&#39;<br><br>      - script: |<br>          set -e -x<br>          golangci-lint run<br>        displayName: &#39;Run Code Quality Checks&#39;<br><br>      - script: |<br>          set -e -x<br>          go test -v -coverprofile=coverage.txt -covermode count ./... &gt; test_results.txt<br>          go-junit-report &lt; test_results.txt &gt; report.xml<br>        displayName: &#39;Run Unit Tests&#39;<br><br>      - task: PublishTestResults@2<br>        displayName: &#39;Publish Test Results&#39;<br>        inputs:<br>          testRunner: JUnit<br>          testResultsFiles: $(System.DefaultWorkingDirectory)/**/report.xml<br><br>      - script: |<br>          set -e -x<br>          gocov convert coverage.txt &gt; coverage.json<br>          gocov-xml &lt; coverage.json &gt; coverage.xml<br>        displayName: &#39;Run Code Coverage Tests&#39;<br><br>      - task: PublishCodeCoverageResults@1<br>        displayName: &#39;Publish Code Coverage&#39;<br>        inputs:<br>          codeCoverageTool: Cobertura<br>          summaryFileLocation: $(System.DefaultWorkingDirectory)/**/coverage.xml<br><br>      - script: |<br>          curl https://cdn.shiftleft.io/download/sl &gt; $BUILD_SOURCESDIRECTORY/sl &amp;&amp; chmod a+rx $BUILD_SOURCESDIRECTORY/sl<br>          $BUILD_SOURCESDIRECTORY/sl analyze --wait --tag branch=$BUILD_SOURCEBRANCHNAME --tag app.group=MyLabs --tag app.language=go --app MyLabs-G0 --cpg --go ./...<br>        displayName: &#39;Run Vulnerability Checks&#39;<br>        env:<br>          SHIFTLEFT_ORG_ID: $(SHIFTLEFT_ORG_ID)<br>          SHIFTLEFT_ACCESS_TOKEN: $(SHIFTLEFT_ACCESS_TOKEN)<br><br>      - script: |<br>          set -e -x<br>          docker run \<br>          -v &quot;$(Build.SourcesDirectory):/app:cached&quot; \<br>          -v &quot;$(Build.ArtifactStagingDirectory):/reports:cached&quot; \<br>          shiftleft/sast-scan scan --src /app \<br>          --out_dir /reports/CodeAnalysisLogs<br>        displayName: &quot;Perform Vulnerability Scan&quot;<br>        continueOnError: &quot;true&quot;<br><br>      - task: PublishBuildArtifacts@1<br>        displayName: &quot;Publish Vulnerability Scan Results&quot;<br>        inputs:<br>          PathtoPublish: &quot;$(Build.ArtifactStagingDirectory)/CodeAnalysisLogs&quot;<br>          ArtifactName: &quot;CodeAnalysisLogs&quot;<br>          publishLocation: &quot;Container&quot;</pre><p>This template will perform these actions,</p><ul><li>Setup Golang</li><li>Install Dependencies</li><li>Run Code Quality Checks</li><li>Run Unit Tests</li><li>Publish Test Results into Pipeline</li><li>Publish Coverage Results into Pipeline</li><li>Run ShiftLeft Inspect/anAlyse Vulnerability Scan</li><li>Run ShiftLeft SAST Vulnerability Scan</li><li>Publish ShiftLeft SAST Vulnerability Scan Results into Pipeline</li></ul><p>Testing on multiple versions.</p><pre>strategy:<br>      matrix:<br>        GoVersion_1_13:<br>          go.version: &#39;1.13&#39;<br>        GoVersion_1_14:<br>          go.version: &#39;1.14&#39;</pre><p>Also note the strategy we have defined, if you want to support wider version of Golang just add more version, here it’s that simple.</p><p><strong>Template : templates/azure/jobs/release.yml</strong></p><pre>parameters:<br>  name: &#39;&#39;<br>  pool: &#39;&#39;<br><br>jobs:<br>  - job: ${{ parameters.name }}<br>    pool: ${{ parameters.pool }}<br>    steps:<br>      - template: ../steps/setupgo.yml<br><br>      - template: ../steps/buildapp.yml<br><br>      - task: CopyFiles@2<br>        displayName: &#39;Copy binary files to Artifact Stage Directory&#39;<br>        inputs:<br>          sourceFolder: $(Build.BinariesDirectory)<br>          targetFolder: $(Build.ArtifactStagingDirectory)<br><br>      - task: PublishBuildArtifacts@1<br>        displayName: &#39;Publish Build Artifacts&#39;<br>        inputs:<br>          artifactName: $(Agent.OS)<br><br>      - task: Bash@3<br>        displayName: &#39;Get/Set Application/Package Version&#39;<br>        inputs:<br>          targetType: &#39;inline&#39;<br>          script: |<br>            set -e -x<br>            version=`./mylabs-go version`<br>            echo &quot;##vso[task.setvariable variable=MYLABSCLI_VERSION;]$version&quot;<br>          workingDirectory: $(Build.BinariesDirectory)<br>        condition: and(succeeded(), eq(variables[&#39;Build.SourceBranch&#39;], &#39;refs/heads/master&#39;))<br><br>      - task: Bash@3<br>        displayName: &#39;Get/Set OS Specific Package Feed Name&#39;<br>        inputs:<br>          targetType: &#39;inline&#39;<br>          script: |<br>            set -e -x<br>            OS_NAME=`echo &quot;$(Agent.OS)&quot; | tr &quot;[:upper:]&quot; &quot;[:lower:]&quot;`<br>            echo &quot;##vso[task.setvariable variable=FEED_NAME;]$OS_NAME&quot;<br>          workingDirectory: $(Build.BinariesDirectory)<br>        condition: and(succeeded(), eq(variables[&#39;Build.SourceBranch&#39;], &#39;refs/heads/master&#39;))<br><br>      - task: UniversalPackages@0<br>        displayName: &#39;Publish Release Artifacts&#39;<br>        inputs:<br>          command: &#39;publish&#39;<br>          publishDirectory: &#39;$(Build.ArtifactStagingDirectory)&#39;<br>          feedsToUsePublish: &#39;internal&#39;<br>          vstsFeedPublish: &#39;1354bdaa-1b77-41d3-a573-e85080e85d85/90f9f1a3-3b7f-4814-aea6-f06d7842d9af&#39;<br>          vstsFeedPackagePublish: $(FEED_NAME)<br>          versionOption: &#39;custom&#39;<br>          versionPublish: $(MYLABSCLI_VERSION)<br>        condition: and(succeeded(), eq(variables[&#39;Build.SourceBranch&#39;], &#39;refs/heads/master&#39;))</pre><p>This template will do these actions,</p><ul><li>Setup Golang</li><li>Build and produce Golang application binary artifact</li><li>Set environment variable for OS specific Azure Artifact Universal Package Feed</li><li>Get/Set Application version to be published in the Azure Artifact Universal Package Feed</li><li>Push the application binary into Azure Artifact Universal Package</li></ul><p>Alright now we have created a reusable build template how do we use this in the pipeline ?</p><h4>Step 4 : Create Build Stage</h4><h4><em>azure-pipeline.yml</em></h4><pre>stages:<br>  - stage: Build<br>    jobs:<br>      - <strong>template: templates/azure/jobs/build.yml   </strong># Linux Build<br>        parameters:<br>          <strong>name</strong>: &#39;Linux_Build&#39;<br>          <strong>pool</strong>:<br>            vmImage: &#39;ubuntu-18.04&#39;<br>      - <strong>template: templates/azure/jobs/build.yml   </strong># macOS Build<br>        parameters:<br>          <strong>name</strong>: &#39;Mac_Build&#39;<br>          <strong>pool</strong>:<br>            vmImage: &#39;macos-10.14&#39;</pre><p>Now you can see how the <strong>parmeterized template</strong> helped us to reuse the sample for different build binary based on the Operating System.</p><h4>Step 5 : Create Test Stage</h4><h4>azure-pipeline.yml</h4><pre>stages:<br>  ..... removed for brevity .....<em><br>  </em>- stage: Test<br>    jobs:<br><strong>      - template: templates/azure/jobs/test.yml</strong></pre><p>I don’t have to explain here as it’s self explanatory.</p><h4>Final Step : Create Release Stage</h4><h4>azure-pipeline.yml</h4><pre>stages:<br>  ..... removed for brevity .....<br>  - stage: Release<br>    jobs:<br>      - template: cicd/jobs/release.yml<br>        parameters:<br>          name: &#39;Linux_Release&#39;<br>          pool:<br>            vmImage: &#39;ubuntu-18.04&#39;<br>      - template: cicd/jobs/release.yml<br>        parameters:<br>          name: &#39;Mac_Release&#39;<br>          pool:<br>            vmImage: &#39;macos-10.14&#39;</pre><p>This stage will create release for multiple Operating System.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=69261e984621" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Warming up AWS EBS volumes restored from Snapshot]]></title>
            <link>https://ashokrajar.medium.com/warming-up-aws-ebs-volumes-restored-from-snapshot-de83134a899a?source=rss-5f3aafe4f222------2</link>
            <guid isPermaLink="false">https://medium.com/p/de83134a899a</guid>
            <category><![CDATA[warming]]></category>
            <category><![CDATA[ebs-snapshots]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[ebs-volume]]></category>
            <dc:creator><![CDATA[Ashok Raja]]></dc:creator>
            <pubDate>Thu, 06 Apr 2017 22:00:41 GMT</pubDate>
            <atom:updated>2020-04-29T23:47:49.975Z</atom:updated>
            <content:encoded><![CDATA[<h3>Should we pre-warm/initialize a EBS volume ?</h3><p>This post talks about the need of pre-waring the EBS volumes…</p><h3>Newer EBS volumes</h3><p>Newly provisioned EBS volumes doesn’t require a pre-warming and they can attain it’s maximum performance from the moment they are available.</p><h3>EBS volumes created from Snapshot</h3><p>EBS volumes created from snapshots need to be pre-warmed/initialized before it can be used on high IO demanding applications.</p><h4>Why ?</h4><ul><li>Snapshots are nothing but storage blocks backed up into S3.</li><li>When an EBS volume is created from a snapshot, not all the storage blocks are written in to the volume.</li></ul><p>So when an application tries to read the data (storage block),</p><ol><li>The corresponding storage block are first pulled down from S3 and written to the volume.</li><li>Only then it’s available for the application to consume the block, and hence increases the latency of an I/O operation the first time each block is accessed</li></ol><h3>How to pre-warm/initialize the EBS volume</h3><p>Using dd or fio utilities read all the blocks on the volume. The dd command is installed by default in most of the OS, but fio is faster due to it&#39;s capability of multi-threaded reads.</p><h4>Using dd: (slower)</h4><pre>sudo dd if=/dev/xvdf of=/dev/null bs=1M</pre><h4>Using fio: (Considerably faster)</h4><pre>sudo fio --filename=/dev/xvdf --rw=read --bs=128k --iodepth=32 --ioengine=libaio --direct=1 --name=volume-initialize</pre><blockquote>Tip: It took me 12hours to pre-warm a 2TB EBS volume with `fio` utilitly.</blockquote><p><em>Originally published at </em><a href="https://mylife.ashokraja.in/2017/04/06/warming-up-aws-ebs-volumes-restored-from-snaphot/"><em>https://mylife.ashokraja.in</em></a><em> on April 6, 2017.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de83134a899a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>