<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ran Rubin on Medium]]></title>
        <description><![CDATA[Stories by Ran Rubin on Medium]]></description>
        <link>https://medium.com/@ranrubin?source=rss-59f5cda9f551------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*LqU4gujkz-CzgYc-kW-C8w.jpeg</url>
            <title>Stories by Ran Rubin on Medium</title>
            <link>https://medium.com/@ranrubin?source=rss-59f5cda9f551------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 16:04:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ranrubin/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How We Added Jetson Workers to Our Cloud-Native Production Environment]]></title>
            <link>https://medium.com/@ranrubin/how-we-added-jetson-workers-to-our-cloud-native-production-environment-8098ce125b35?source=rss-59f5cda9f551------2</link>
            <guid isPermaLink="false">https://medium.com/p/8098ce125b35</guid>
            <category><![CDATA[k8s]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[nvidia]]></category>
            <category><![CDATA[k3]]></category>
            <category><![CDATA[jetsons]]></category>
            <dc:creator><![CDATA[Ran Rubin]]></dc:creator>
            <pubDate>Fri, 13 May 2022 05:17:36 GMT</pubDate>
            <atom:updated>2022-05-15T06:55:26.462Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JEIKXOyifE3RGWbno5nw-Q.jpeg" /></figure><blockquote>“We decided to create a hybrid K3s cluster with the control plane on AWS EC2 instances and the workers being the edge devices located in our designated on-premise server room.”</blockquote><p>At <a href="https://deci.ai/">Deci</a>, we help AI teams build, optimize, and deploy deep learning models on any hardware and environment with outstanding accuracy and runtime performance. Our Neural Architecture Search (NAS) based <a href="https://deci.ai/platform/">deep learning development platform</a> provides data scientists and ML engineers with a wide variety of tools that simplify and shorten the end-to-end development cycle — from generating custom hardware-aware architectures to deploying with our unified model inference API.</p><p>With more and more companies and AI teams needing to process large amounts of data on the edge and in real-time, we wanted to help them auto compile and quantize modules on edge devices — and thought it would be a good idea to start with the famous <a href="https://developer.nvidia.com/embedded/jetson-modules">Nvidia Jetson</a> family. Here’s how we extended Deci’s offer to auto compile and quantize deep learning modules to edge devices like Jetsons.</p><p>Please note that although this solution is demonstrated on Jetsons, it can be easily extended to any hardware that supports K3s.</p><h3><strong>Hosting the Edge Hardware</strong></h3><p>Working on edge devices introduced a new challenge for our dev team. Because our infrastructure is cloud-native, with K8s acting as our container orchestrator, all of our nodes and hardware were cloud-based. The thing is, our cloud provider (AWS) does not offer on-demand Nvidia Jetson instances, so we will have to host them ourselves.</p><p>We were up for the challenge and set out to build our own on-premise hosting of edge hardware. We had to find a production solution that would enable a seamless connection between the on-premise edge hardware we had and our cloud infrastructure. Moreover, any solution had to be reliable, reproducible, and allow us to scale up and add new hardware easily.</p><h3><strong>Using a Memory-light Version of K8s</strong></h3><p>If we’re talking about scalability and cloud-native cluster management, then <a href="https://kubernetes.io/">K8s</a> is usually the right way to go. K8s is probably the most popular method for container orchestration. However, K8s also consume quite a bit of memory and potentially wouldn’t work well for edge devices, which usually have less memory and CPU power than a typical server. Our first goal was to find a lightweight version of K8s.</p><p>After some exploration and deliberation, we ended up with various options — <a href="https://github.com/k3s-io/k3s">K3s</a> by Rancher, <a href="https://github.com/canonical/microk8s">Microk8s</a> by canonical, or the interesting CNCF-backed project <a href="https://github.com/kubeedge/kubeedge">KubeEdge</a>. We ultimately decided to go with <strong>K3s</strong> because it was the most mature solution out of the three.</p><p><a href="https://rancher.com/docs/k3s/latest/en/">K3s is a lightweight version of Kubernetes</a>. It’s easy to install and uses about half the memory, all in a binary of less than 100 MB. It also supports many IoT and edge devices. Now, we needed to decide how we would connect the edge hardware we had on-premise to all of Deci’s proprietary technologies that live in the cloud. Any solution would have to be seamless and cloud-native to work smoothly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ov4NbTcGCjdZPbQB" /><figcaption>Source: <a href="https://rancher.com/docs/k3s/latest/en/architecture/">Rancher</a></figcaption></figure><h3><strong>The K3s Hybrid Cluster: Servers in the Cloud + Workers On-prem</strong></h3><p>As always, before beginning to build anything, we defined our requirements. We wanted to build an infrastructure and software system that would be:</p><ol><li>Integrated seamlessly with our current application and cloud infrastructure.</li><li>Production-ready, which means highly available, scalable, reproducible, and easy to maintain.</li></ol><p>We decided to create a hybrid K3s cluster with the control plane on AWS EC2 instances and the workers being the edge devices located in our designated on-premise server room. To achieve high availability, we created an external database as part of the control plane (see <a href="https://rancher.com/docs/k3s/latest/en/architecture/">High-Availability K3s Server with an External DB</a>).</p><p>Having the control plane on AWS EC2 instances meant having one leg in the cloud. This would allow us easier integration with our existing cloud infrastructure and let us benefit from the scalability, reliability, and availability of working in the cloud.</p><p>Naturally, we couldn’t help but wonder whether this architecture would create a latency issue between our on-premise worker and the cloud server nodes. Because the jobs we run involve high power computing and are relatively lengthy in any case, we figured the latency involved wouldn’t significantly impact the workloads done on those nodes.</p><p>Below we demonstrate how we connected an edge device with <a href="https://developer.nvidia.com/embedded/jetson-xavier-nx-devkit"><strong>Nvidia’s Jetson Xavier NX</strong></a> to the cluster. Any edge device that can run a K3s agent can be added in a similar way.</p><h3><strong>Building the Cloud Environment</strong></h3><p>We use AWS as our cloud provider, but the solution we present can be implemented with any cloud provider using the appropriate alternative resources.</p><p>We built our cloud environment by creating the following resources:</p><ul><li>A virtual private cloud (VPC) with private and public subnets.</li><li>3 EC2 instances of <a href="https://aws.amazon.com/ec2/instance-types/t3/">t3.medium</a>.</li><li><a href="https://aws.amazon.com/rds/postgresql/">RDS postgres</a> as the external database</li><li>Security groups to allow traffic between the nodes (see <a href="https://rancher.com/docs/k3s/latest/en/installation/installation-requirements/#networking">networking)</a></li><li>Identity access management (<a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html">IAM</a>) roles to allow access to our existing cloud environment</li><li>A <a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html">Network Load balancer</a>, including a target group and listeners</li></ul><p>If you’re thinking of setting up something similar, make sure to deploy the entire cloud infrastructure using an <a href="https://www.redhat.com/en/topics/automation/what-is-infrastructure-as-code-iac">infrastructure as a code</a> tool, such as <a href="https://www.terraform.io/">Terraform</a> which we use at Deci. Terraform allows us to compose, collaborate, reuse, and provision infrastructure as code across our development teams.</p><p>We started by creating a designated VPC for our edge cluster. Next, we created three EC2 instances that we would later configure to be our server nodes. For production purposes, we placed them in different availability zones.</p><p>We used an RDS instance of Postgres as the external DB for our servers. There are <a href="https://rancher.com/docs/k3s/latest/en/installation/datastore/">a few options </a>that you can choose for your external DB, but we like Postgres the best.</p><p>To get true high availability, we added a network load balancer that would direct traffic to the server nodes.</p><p>As mentioned in the introduction, we needed the solution to integrate with our current cloud system. This meant we had to establish appropriate IAM roles that the pods on the cluster would assume at runtime. Because our cluster has nodes in our cloud environment we can use tools like <a href="https://github.com/jtblin/kube2iam">kube2iam </a>to assume the roles.</p><p>We also had to create the supporting security groups that would allow K3s to work properly, as described in the <a href="https://rancher.com/docs/k3s/latest/en/installation/installation-requirements/#networking">networking segment of the official documentation</a>. Since we’re using the default <a href="https://k8s.networkop.co.uk/cni/flannel/">Flannel VXLAN cni</a>, we opened ports 6443, 10250 on TCP, and 8472 on UDP. We also assigned a security group to the DB, to accept TCP traffic from the server nodes’ security group at port 5432.</p><p>It’s also important that you set up a secure network connection between your on-premise setup and your cloud environment. There are many ways to achieve this, but we’re not going to get into that in this article.</p><h3><strong>Configuring the Server Nodes</strong></h3><p>Following the <a href="https://rancher.com/docs/k3s/latest/en/installation/ha/">official Rancher documentation</a>, here’s how to configure the server nodes:</p><ol><li>Update and upgrade the servers.</li><li>Update /etc/hosts file with the hostname as an alias next to localhost and update the other servers with their private IPs.</li></ol><p><strong>Configure the first server node</strong></p><p>On one EC2 instance, the following command can be used to install the K3s server with the PostgreSQL database as the external datastore :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eae4d86a8dc8847944866e64a0cf6758/href">https://medium.com/media/eae4d86a8dc8847944866e64a0cf6758/href</a></iframe><p>In the example above, we launched the server with the following attributes:</p><ul><li>Version — v1.21.9+k3s1,</li><li>Debug flag — helps us debug when needed</li><li>Datastore endpoint — applies to Postgres, according to <a href="https://rancher.com/docs/k3s/latest/en/installation/datastore/#datastore-endpoint-format-and-functionality">datastore endpoint format and functionality</a></li><li>We disabled <a href="https://doc.traefik.io/traefik/providers/kubernetes-ingress/">traefik ingress</a> since our use case did not require it.</li></ul><p>Running the command above configures your node to function as a server node (control plane) in the K3s cluster. Because we added thedebug flag, we were able to verify that the installation went smoothly (or troubleshoot if it didn’t) by looking at /var/log/syslog — if everything is working as it should, the wrote ping debug message is displayed.</p><p>Once you finish configuring the first server node, it’s time to do the same for the other two servers, but with a small change. You need to provide the servers with the token that was generated by the first server installation.</p><p>You can find the token value at /var/lib/rancher/k3s/server/token</p><p><strong>Configuring the other two server nodes</strong></p><p>The following command can be used to install the remaining K3s server nodes with the same PostgreSQL database as the external datastore and the token that was retrieved from the first server.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/790fb34a3dd8f9e7ec59ba1361ef8ed9/href">https://medium.com/media/790fb34a3dd8f9e7ec59ba1361ef8ed9/href</a></iframe><p>Run kubectl get nodes and make sure that all the server nodes are shown and ready. If they are not, check the logs as mentioned earlier.</p><p><strong>You should now have three EC2 nodes that are —</strong></p><ol><li>Configured as K3s server nodes</li><li>Backed by a PostgresSQL database</li><li>Reachable using the network load balancer we configured earlier.</li></ol><p>Notice that we did not provide any <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/">taint for the server nodes, </a>so we could run cluster add-ons software on these server nodes rather than on the edge device worker nodes.</p><p>You should automate the configuration process shown above for the server nodes using an <a href="https://www.redhat.com/en/topics/automation/what-is-an-ansible-playbook#:~:text=An%20Ansible%C2%AE%20playbook%20is,make%20up%20an%20Ansible%20inventory.">Ansible playbook</a>. At Deci, we aspire to automate whatever we can, and having these configurations done by a machine helped us save time and avoid mistakes.</p><h3><strong>Preparing the Nvidia Jetson Xavier NX Developer Kit</strong></h3><p>We set up a mini hosting farm in our offices that can accommodate edge devices. Our setup is composed of a server rack that hosts the devices, Ethernet switches for every environment with network segmentation, and internet and power backups (UPS) to support our production deployment.</p><p>We also verified that we have a secured private network connection between our on-premise network and the cloud.</p><p>Exact instructions for how to set up the Jetson device are beyond the scope of this blog, but you can find that kind of information in some of the resources listed at the end of the article. Another thing you might find interesting is that we mounted an NVMe SSD card to the carrier board to get better performance for the cluster (as opposed to eMMC or MicroSD). We had a great experience working with <a href="https://www.seeedstudio.com/A206-Carrier-Board-for-Jetson-Nano-Xavier-NX-p-5132.html">Seed A206 carrier board</a>, which has a <a href="https://wiki.seeedstudio.com/Tutorial-of-A20X-Carrier-Boards/">comprehensive tutorial</a>.</p><p>As for specific configurations for our cluster, we adjusted the power mode, disabled IPv6, and allowed IPv4 forwarding.</p><p>To configure the Jetson Xavier<strong> </strong>and set the clocks for maximum performance, you can run the following commands —</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a4a379451fcd6744396d721baa43c170/href">https://medium.com/media/a4a379451fcd6744396d721baa43c170/href</a></iframe><p>For the networking configurations, you can run —</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f1504ec05b652155533ed4992bc0fe95/href">https://medium.com/media/f1504ec05b652155533ed4992bc0fe95/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RZ1MiTB9jJCHw93ktr5LXQ.jpeg" /><figcaption>Jetson Xavier NX workers on our server rack</figcaption></figure><h3><strong>Configuring Jetson Xavier as a Worker Node</strong></h3><p>On the hardware, your first step should be to configure /etc/hosts with the server nodes’ IP addresses and hostnames.</p><p>The following command can be used to install the Jetson worker node:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9d2c2556b20edf60c487c6d3f4853055/href">https://medium.com/media/9d2c2556b20edf60c487c6d3f4853055/href</a></iframe><p>In the example above, we launched the server with the following attributes:</p><ul><li>Server IP — the load balancer’s IP or DNS</li><li>Taint — we launch the worker with a taint so it would be schedulable for selected workloads only.</li><li>Node IP — the IP of the Jetson Xavier</li><li>Token — the server token we got from the previous step</li></ul><p>You can use kubectl get nodes to assure the worker was added successfully.</p><p><strong>Ensure network connectivity</strong></p><p>Because we had nodes both on and off the cloud, it’s a good practice to ensure node-to-node connectivity as well as Container Network Interface (CNI) testing. You can do this easily by trying to resolve any DNS using the CoreDNS service on K3s, with the default IP 10.43.0.10 as follows:</p><pre>nslookup google.com 10.43.0.10</pre><p>If the above command times out, you have a networking problem.</p><p><strong>Set </strong><strong>nvidia-container-runtime as the default runtime</strong></p><p><a href="https://containerd.io/">containerd</a> is the default container runtime on K3s (you can use docker as well). We will have to configure containerd to use nvidia-container-runtime as the default runtime, so the K3s pods that run on the Jetson Xavier device will use its GPU.</p><p><a href="https://rancher.com/docs/k3s/latest/en/advanced/#configuring-containerd">K3s allow</a>s us to configure containerd by coping /var/lib/rancher/k3s/agent/etc/containerd/config.toml to a file named config.toml.tmpl in that same directory.</p><p>On config.toml.tmpl file, change the runtime configuration (at the end of the file) with the following block:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f02d66e527a53da43c46b88b22d4e3a1/href">https://medium.com/media/f02d66e527a53da43c46b88b22d4e3a1/href</a></iframe><p>So the config.toml.tmpl file should look something like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eab60dd8bdc812baa6ca0fe5827b7129/href">https://medium.com/media/eab60dd8bdc812baa6ca0fe5827b7129/href</a></iframe><p>Now restart the K3s agent or reboot the machine.</p><p>Similar to the server nodes, we configured the worker nodes using an Ansible playbook.</p><h3>Automated Runtime Optimization with Deci’s Platform</h3><p>What’s our takeaway from this experience? In the grand scheme of things, we’re reminded that the deep learning space is constantly evolving, and it’s important to be innovative and agile, combining imagination and experimentation, to continue providing the tools that the AI community needs.</p><p>The process that we detailed is proof that incorporating edge devices into an integrated cloud environment can be done without all that much effort. As a result, it now only takes a matter of minutes for AI developers to automatically compile and quantize deep learning models on edge devices, specifically on Jetson devices.</p><p>Here’s a quick walkthrough of how it works:</p><ol><li><strong>Pick or upload a model. </strong>Sign up for Deci using your email address. Go to the “Lab” tab to upload your model or pick a pre-trained one from the “Model Zoo” tab.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o8xzbsTggDv7srr2EkQIKA.png" /><figcaption>Hardware-aware model zoo</figcaption></figure><p><strong>2. Select your optimization targets. </strong>On the “Lab” tab, select the model you would like to compile and click “Optimize”. You will be prompted to select your target inference hardware, batch size, and quantization level. Once selected, click “Start Optimization”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2pyPB_rmd5p2ciFq" /><figcaption>Support for a wide range of hardware</figcaption></figure><p><strong>3. Grab a coffee while we compile your model.</strong> It takes only a few minutes to see your compiled and quantized model ready for deployment on the “Lab” tab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tpF131N1J7Bi3xU9CoSOzw.png" /><figcaption>Compilation and quantization in minutes</figcaption></figure><p><strong>4. Compare optimization results across various targets. </strong>On the “Insights” tab you will see all your optimized models. You can easily compare the results to understand the impact of the optimization on your models’ performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qI7Gx6XJrxA4kz54d5xlxA.png" /><figcaption>Optimization results, accuracy and inference performance metrics</figcaption></figure><p>If you’re interested in using Deci’s free tool that delivers the fastest and simplest way to optimize your deep learning models on Jetson devices, <a href="http://console.deci.ai/sign-up">create a free account on the platform</a> today.</p><p>If you’re a DevOps engineer and want to be part of our story, come <a href="https://deci.ai/careers/">work with us</a>!</p><p><strong>References and related links:</strong></p><ul><li>K3s documentation (<a href="https://rancher.com/docs/k3s/latest/en/">https://rancher.com/docs/k3s/latest/en/</a>)</li><li><a href="https://developer.nvidia.com/embedded/learn/get-started-jetson-xavier-nx-devkit">https://developer.nvidia.com/embedded/learn/get-started-jetson-xavier-nx-devkit</a></li><li><a href="https://github.com/k3s-io/k3s-ansible">https://github.com/k3s-io/k3s-ansible</a></li><li><a href="https://www.suse.com/c/running-edge-artificial-intelligence-k3s-cluster-with-nvidia-jetson-nano-boards-src/">https://www.suse.com/c/running-edge-artificial-intelligence-k3s-cluster-with-nvidia-jetson-nano-boards-src/</a></li><li><a href="https://blogs.windriver.com/wind_river_blog/2020/06/nvidia-k8s-device-plugin-for-wind-river-linux/">https://blogs.windriver.com/wind_river_blog/2020/06/nvidia-k8s-device-plugin-for-wind-river-linux/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8098ce125b35" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A robust implementation of ArgoCD’s App Of Apps Pattern]]></title>
            <link>https://medium.com/@ranrubin/a-robust-implementation-of-argocds-app-of-apps-pattern-97f8071e0de?source=rss-59f5cda9f551------2</link>
            <guid isPermaLink="false">https://medium.com/p/97f8071e0de</guid>
            <category><![CDATA[argo-cd]]></category>
            <category><![CDATA[gitops]]></category>
            <category><![CDATA[argoproj]]></category>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[cloud-native]]></category>
            <dc:creator><![CDATA[Ran Rubin]]></dc:creator>
            <pubDate>Wed, 11 May 2022 12:40:21 GMT</pubDate>
            <atom:updated>2022-05-11T12:40:21.005Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d_nWfpG-EF831-yXwx64HQ.png" /></figure><p><a href="https://argo-cd.readthedocs.io/en/stable/">ArgoCD</a> is a great tool that improves the stability and visibility of the continuous deployment process in the organization. When using it properly, it can easily become the #1 tool in your organization for deploying every service.</p><p>ArgoCD’s <a href="https://argo-cd.readthedocs.io/en/stable/operator-manual/cluster-bootstrapping/#app-of-apps-pattern">App of Apps pattern </a>helps us declaratively specify one Argo CD app that consists only of other apps. In this post, I will present an implementation method and a few tips that helped me use this pattern in a convenient, extensible, and stable manner.</p><p>If you are not that familiar with ArgoCD, before reading this article I recommend going over the <a href="https://argo-cd.readthedocs.io/en/stable/">official documentation</a> and then watching this <a href="https://www.youtube.com/watch?v=vpWQeoaiRM4">great video by DevOps Toolkit channel</a>.</p><h3>Using a single YAML file to generate all Application templates</h3><p>As shown in <a href="https://github.com/argoproj/argocd-example-apps/tree/master/apps">the example provided by ArgoCD</a>, a simple implementation of the <a href="https://argo-cd.readthedocs.io/en/stable/operator-manual/cluster-bootstrapping/#app-of-apps-pattern">App of Apps pattern</a> consist of multiple Application templates. With the following layout:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*F-ll8-8oJJahkUv77icNiA.png" /><figcaption><a href="https://argo-cd.readthedocs.io/en/stable/operator-manual/cluster-bootstrapping/#app-of-apps-pattern">https://argo-cd.readthedocs.io/en/stable/operator-manual/cluster-bootstrapping/#app-of-apps-pattern</a></figcaption></figure><p>However, when one will need to update anApplication feature across all applications, they will have to edit each and every Applicationtemplate by itself.</p><p>Instead, I suggest using a single Application template file, that will generate the other Application templates. For example:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aa2969319b3e758f6629383391758144/href">https://medium.com/media/aa2969319b3e758f6629383391758144/href</a></iframe><p>Using this way to deploy all the applications, we avoid code repetition, and make it easier to maintain, while preserving the ability to control and uniquely define each of the Application templates.</p><p>using this template, the values.yaml file of <a href="https://github.com/argoproj/argocd-example-apps/tree/master/apps">the example provided by ArgoCD</a> will look something like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/332ada5eec59335d4b649cb56d3e91be/href">https://medium.com/media/332ada5eec59335d4b649cb56d3e91be/href</a></iframe><h3>Fixing targetRevision to a specific git tag when making template change</h3><p>When it comes to helm charts, the values.yaml file helps us pass and define parameters to the templates. When using the same code for deploying an application on multiple environments (i.e. — development, staging, production), you might want to have a separate values.yaml file for each environment.</p><p>When we use a values-{ENV}.yaml file for every environment, we get better visibility for what values we use on each environment (container images, versions, etc). Another advantage is that it will be easier for our automation to edit these files. Having a designated values file for every environment is a common practice when using helm not a unique practice for the App of Apps pattern.</p><p>However, what happens when we need to update the Helm chart template itself? <br>For example, if we want to explore the possibility of using a StatefulSet instead of aDeployment in one of our charts on our development environment. By changing the Deploymenttemplate, both dev and prod environments would be affected, as they both use HEAD as their targetRevision (a behavior that we might generally want to preserve for a “complete CD” experience).</p><p>To overcome this potential stability issue, whenever you are making a change in the template of the chart, you can update the Application template to use a specific git tag (or SHA if you are slightly less human than the rest of us…). By doing so, every template change affects the Application templates that use HEAD, while those with fixed targetRevisionremain the same.</p><p>An example that simulates setting unique values for a production Application</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bc9c0c73f657a1a64b6ac2dea5f2ab55/href">https://medium.com/media/bc9c0c73f657a1a64b6ac2dea5f2ab55/href</a></iframe><p>The above values file will override the default values file with the following changes:</p><ul><li>helmGuestbook application is set to 1.0.0 tag.</li><li>kustomizeGuestbook application is disabled (we might not need it in our production environment)</li><li>syncWaves application automated deployment is disabled.</li></ul><p>For a complete example based on the Argo-CD example repository, see <a href="https://github.com/ranrubin/argocd-app-of-apps">my GitHub repository</a>.</p><h3>Other tips for working with ArgoCD</h3><ul><li>Keep services’ Applicationdeployments separate from cluster add-ons Applicationdeployments. For example, don’t use the same parent application to deploy Grafana and to deploy your backend.</li><li>Keep services configuration templates in a designated helm chart and deploy it as an Application. For example, there is no logical justification for having the Sensor of Argo-Events be deployed with Argo-Events itself. The deployment of Argo Events will not be updated that often while the Sensor is more likely to be</li></ul><h3>Summary</h3><p>Using ArgoCD’s App of Apps pattern can improve your experience when working with ArgoCD. By following the practices shown above, you can significantly enhance that experience; reduce the amount of code, and increase stability and readability.</p><p>Do you have any other tips and tricks that I did not cover? Please share in the comments!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=97f8071e0de" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My Journey in Converting PyTorch to TensorFlow Lite]]></title>
            <link>https://medium.com/data-science/my-journey-in-converting-pytorch-to-tensorflow-lite-d244376beed?source=rss-59f5cda9f551------2</link>
            <guid isPermaLink="false">https://medium.com/p/d244376beed</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[onnx]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[tflite]]></category>
            <dc:creator><![CDATA[Ran Rubin]]></dc:creator>
            <pubDate>Tue, 29 Sep 2020 02:13:56 GMT</pubDate>
            <atom:updated>2020-09-29T07:23:05.567Z</atom:updated>
            <content:encoded><![CDATA[<h4>Sometimes an MLOps gotta do what an MLOps gotta do</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gtw8Hf9P_JTHwduIdL9I2Q.png" /><figcaption>Taken from <a href="https://commons.wikimedia.org/wiki/Main_Page">Wikimedia</a></figcaption></figure><h3>Intro</h3><p>I recently had to convert a deep learning model (a <a href="https://paperswithcode.com/method/mobilenetv2">MobileNetV2</a> variant) from <a href="https://pytorch.org/">PyTorch</a> to <a href="https://www.tensorflow.org/lite">TensorFlow Lite</a>. It was a long, complicated journey, involved jumping through a lot of hoops to make it work. I found myself collecting pieces of information from Stackoverflow posts and GitHub issues. My goal is to share my experience in an attempt to help someone else who is lost like I was.</p><blockquote><strong>DISCLAIMER: This is not a guide</strong> on how to properly do this conversion. I only wish to share my experience. I might have done it wrong (especially because I have no experience with Tensorflow). If you notice something that I could have done better/differently — please comment and I’ll update the post accordingly.</blockquote><h3>The Mission</h3><p>Convert a deep learning model (a <a href="https://paperswithcode.com/method/mobilenetv2">MobileNetV2</a> variant) from Pytorch to TensorFlow Lite. The conversion process should be:<br>Pytorch →ONNX → Tensorflow → TFLite</p><h3>Tests</h3><p>In order to test the converted models, a set of roughly 1,000 input tensors was generated, and the PyTorch model’s output was calculated for each. That set was later used to test each of the converted models, by comparing their yielded outputs against the original outputs, via a mean error metric, over the entire set. The mean error reflects how different are the converted model outputs compared to the original PyTorch model outputs, over the same input. <br>I decided to treat a model with a mean error smaller than 1e-6 as a successfully converted model.</p><p>It might also be important to note that I added the batch dimension in the tensor, even though it was 1. I had no reason doing so other than a hunch that comes from my previous experience converting PyTorch to <a href="https://developer.qualcomm.com/docs/snpe/index.html">DLC models</a>.</p><h3>Converting Pytorch to ONNX</h3><p>This was definitely the easy part. Mainly thanks to the excellent documentation on PyTorch, for example <a href="https://pytorch.org/docs/stable/onnx.html">here</a> and <a href="https://pytorch.org/tutorials/advanced/super_resolution_with_onnxruntime.html">here</a>.</p><p>Requirements:</p><ul><li>ONNX ==1.7.0</li><li>PyTorch ==1.5.1.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c9a1f11be8c537fa563971399e963686/href">https://medium.com/media/c9a1f11be8c537fa563971399e963686/href</a></iframe><p>The newly created ONNX model was tested on my example inputs and got a mean error of 1.39e-06.</p><p>Notice that you will have to convert the torch.tensor examples into their equivalentnp.array in order to run it through the ONNX model.</p><h3>Converting ONNX to TensorFlow</h3><p>Now that I had my ONNX model, I used <a href="https://github.com/onnx/onnx-tensorflow">onnx-tensorflow</a> (<a href="https://github.com/onnx/onnx-tensorflow/releases/tag/v1.6.0">v1.6.0</a>) library in order to convert to TensorFlow. I have no experience with Tensorflow so I knew that this is where things would become challenging.</p><p>Requirements:</p><ul><li>tensorflow==2.2.0 (Prerequisite of onnx-tensorflow. However, it worked for me with tf-nightly build 2.4.0-dev20200923 as well)</li><li>tensorflow-addons==0.11.2</li><li><a href="https://github.com/onnx/onnx-tensorflow">onnx-tensorflow</a> == <a href="https://github.com/onnx/onnx-tensorflow/releases/tag/v1.6.0">1.6.0</a></li></ul><p>I’m not sure exactly why, but the conversion worked for me on a GPU machine only.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/552aab062ef4ab5d1dc61257253cafa1/href">https://medium.com/media/552aab062ef4ab5d1dc61257253cafa1/href</a></iframe><p>I ran my test over the TensorflowRep object that was created (examples of inferencing with it <a href="https://github.com/onnx/onnx-tensorflow/tree/master/example">here</a>). The run was <strong>super slow </strong>(around 1 hour as opposed to a few seconds!) so it got me worried. However, eventually, the test produced a mean error of 6.29e-07 so I decided to move on.</p><p>The big question at this point was — what was exported? What is this .pb file? <br>After some digging online I realized it’s an instance of <a href="https://www.tensorflow.org/api_docs/python/tf/Graph">tf.Graph</a>. Now <strong><em>all that was left</em></strong><em> </em>to do is to convert it to TensorFlow Lite.</p><h3>Converting TensorFlow to TensorFlow Lite</h3><p>This is where things got really tricky for me. As I understood it, <a href="https://www.tensorflow.org/lite/convert">Tensorflow offers 3 ways to convert TF to TFLite</a>: SavedModel, Keras, and concrete functions. I’m not really familiar with these options, but I already know that what the onnx-tensorflow tool had exported is a frozen graph, so none of the three options helps me :(</p><p>After quite some time exploring on the web, <a href="https://stackoverflow.com/questions/53182177/how-do-you-convert-a-onnx-to-tflite/58576060#58576060">this guy</a> basically saved my day. It turns out that in Tensorflow v1 converting from a frozen graph is supported! I decided to use v1 API for the rest of my code.</p><p>When running the conversion function, a weird issue came up, that had something to do with the protobuf library. Following <a href="https://stackoverflow.com/a/57307176/6018371">this user</a> advice, I was able to move forward.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/102a236bb3a4fc59d03aea756265656a/href">https://medium.com/media/102a236bb3a4fc59d03aea756265656a/href</a></iframe><p>You would think that after all this trouble, running <a href="https://www.tensorflow.org/lite/guide/inference#load_and_run_a_model_in_python">inference </a>on the newly created tflite model could be done peacefully. But my troubles did not end there and more issues came up.</p><p>One of them had to do with something called “<a href="https://www.tensorflow.org/lite/guide/ops_compatibility">ops</a> “(an error message with &quot;ops that can be supported by the flex.”). After some digging, I realized that my model architecture required to <a href="https://www.tensorflow.org/lite/guide/ops_select">explicitly enable some operators</a> before the conversion (see above).</p><p>Then, it turned out that many of the operations that my network uses are still in development, so the TensorFlow version that was running (2.2.0) could not recognize them. This was solved by installing <a href="https://pypi.org/project/tf-nightly/">Tensorflow’s nightly build</a>, specifically tf-nightly==2.4.0.dev20299923.</p><p>Another error I had was &quot;The Conv2D op currently only supports the NHWC tensor format on the CPU. The op was given the format: NCHW”. This was solved with the help of <a href="https://github.com/onnx/onnx-tensorflow/issues/535#issuecomment-683366977">this user’s comment</a>.</p><p>Eventually, this is the inference code used for the tests —</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6be8d8b4a30f8d768fbd157542804de5/href">https://medium.com/media/6be8d8b4a30f8d768fbd157542804de5/href</a></iframe><p>The tests resulted in a mean error of 2.66-07</p><p>I hope that you found my experience useful, good luck!</p><h3>Related Links</h3><p><strong>Official Documentation:</strong></p><ul><li><a href="https://pytorch.org/docs/stable/onnx.html">https://pytorch.org/docs/stable/onnx.html</a></li><li><a href="https://pytorch.org/tutorials/advanced/super_resolution_with_onnxruntime.html">https://pytorch.org/tutorials/advanced/super_resolution_with_onnxruntime.html</a></li><li><a href="https://github.com/microsoft/onnxruntime">https://github.com/microsoft/onnxruntime</a></li><li><a href="https://github.com/onnx/tutorials">https://github.com/onnx/tutorials</a></li><li><a href="https://www.tensorflow.org/lite/guide/ops_compatibility">https://www.tensorflow.org/lite/guide/ops_compatibility</a></li><li><a href="https://www.tensorflow.org/lite/guide/ops_select">https://www.tensorflow.org/lite/guide/ops_select</a></li><li><a href="https://www.tensorflow.org/lite/guide/inference#load_and_run_a_model_in_python">https://www.tensorflow.org/lite/guide/inference#load_and_run_a_model_in_python</a></li></ul><p><strong>Issues and Stackoverflow</strong></p><ul><li><a href="https://stackoverflow.com/questions/53182177/how-do-you-convert-a-onnx-to-tflite/58576060">https://stackoverflow.com/questions/53182177/how-do-you-convert-a-onnx-to-tflite/58576060</a></li><li><a href="https://stackoverflow.com/questions/55475551/toco-from-protos-command-not-found"><em>https://stackoverflow.com/questions/55475551/toco-from-protos-command-not-found</em></a></li><li><a href="https://github.com/onnx/onnx-tensorflow/issues/535#issuecomment-683366977">https://github.com/onnx/onnx-tensorflow/issues/535#issuecomment-683366977</a></li><li><a href="https://github.com/tensorflow/tensorflow/issues/41012">https://github.com/tensorflow/tensorflow/issues/41012</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d244376beed" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/my-journey-in-converting-pytorch-to-tensorflow-lite-d244376beed">My Journey in Converting PyTorch to TensorFlow Lite</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Horizontal Pod Autoscaling (HPA) triggered by Kafka event]]></title>
            <link>https://medium.com/@ranrubin/horizontal-pod-autoscaling-hpa-triggered-by-kafka-event-f30fe99f3948?source=rss-59f5cda9f551------2</link>
            <guid isPermaLink="false">https://medium.com/p/f30fe99f3948</guid>
            <category><![CDATA[autoscaling]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[hpa]]></category>
            <dc:creator><![CDATA[Ran Rubin]]></dc:creator>
            <pubDate>Wed, 07 Aug 2019 07:15:13 GMT</pubDate>
            <atom:updated>2019-08-20T12:50:30.131Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/979/1*x3BRafGKPGsspj8JgFg_hw.png" /></figure><h3>Autoscale your Kafka (consumer) pods using metrics from Prometheus</h3><p>This is a quick guide for autoscaling Kafka pods. These pods (consumer pods) will scale upon a Kafka event, specifically consumer group lag. The consumer group lag metric will be exported to Prometheus by a sidecar in the pod (we will be using <a href="https://github.com/danielqsj/kafka_exporter">kafka_exporter</a> image for the sidecar). <br>The <a href="https://github.com/DirectXMan12/k8s-prometheus-adapter">Prometheus Adapter</a> will transform Prometheus’ metrics into k8s custom metrics API, allowing an hpa pod to be triggered by these metrics and scale a deployment.</p><p>This tutorial was done with a great help from <a href="https://medium.com/@elhayefrat">Elhay Efrat.</a></p><h3>Prerequisites</h3><ol><li>K8s cluster :) I was using v1.14. You need v1.11+</li><li>Prometheus — You must have Prometheus installed on your cluster.</li><li>Kafka — You should have Kafka deployed on your k8s cluster or make the needed, mostly networking, changes in the following code for proper communication between objects.<br>Throughout this guide we will assume that this “cluster” is under ‘kafka’ namespace and “my-kafka-broker<strong>” </strong>is a service that wraps the brokers’ endpoints.</li><li>Api-versions— autoscaling/v2beta, extensions/v1beta1</li><li><a href="https://helm.sh/">Helm — the package manager for k8s</a></li></ol><h3>Intro</h3><p>As mentioned above, this is a quick guide to autoscale your Kafka pods using metrics from Prometheus.</p><p>I decided to use <a href="https://github.com/danielqsj/kafka_exporter">kafka_exporter</a> as my metric exporter. Mainly because the metrics I seek, specifically consumer group lag, are not exported by the <a href="https://github.com/prometheus/jmx_exporter">jmx</a>. You can write your own application that exposes whatever specific metric you desire.</p><p>I decided to use <a href="https://github.com/DirectXMan12/k8s-prometheus-adapter">Prometheus Adapter </a>to create the custom metrics for k8s, mainly because it’s easy to scrape and pass metric data to Prometheus using <a href="https://www.weave.works/docs/cloud/latest/tasks/monitor/configuration-k8s/#per-pod-prometheus-annotations">annotation</a>.</p><p>I decided to focus on “kafka_consumergroup_lag” metric but there are others that kafka_exporter export. Read more about <a href="https://sematext.com/blog/kafka-consumer-lag-offsets-monitoring/">what is Kafka consumer lag</a>.</p><p>I will not deep dive into explaining each item we deploy, but links to very good documentation sources were added.</p><h3>Basic Use Case</h3><p>We want to have an application (on a pod in our k8s cluster) that will consume messages from Kafka. It should read “my-topic” topic and be a part of ”we-consume“ consumer group. In case that the Kafka consumer lag for this topic is more than 5, we want that the consumer pod will automatically scale out.<br>We will achieve the autoscaling by connecting 4 objects.</p><p>On the application side:</p><ol><li><strong>kafka-exporter</strong>: This container runs an image of <a href="https://github.com/danielqsj/kafka_exporter">kafka_exporter</a>. It will interact with the Kafka server and will expose metrics on the pod. It’s a side car in the consumer application pod.</li><li><strong>Kafka consumer application</strong>: The application that consumes Kafka’s messages. Resides in the same pod as the kafka-exporter</li></ol><p>Monitoring side:</p><ol><li><a href="https://github.com/DirectXMan12/k8s-prometheus-adapter"><strong>Prometheus Adapter pod</strong></a><strong> :</strong> Will be deployed using helm chart. It implements Custom Metrics API and creates k8s custom metrics from Prometheus metrics.</li></ol><p>Connects the two:</p><ol><li><a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/"><strong>HPA</strong></a> : The hpa will use the custom metrics that were created by the adapter (in the monitoring side) to scale the consumer application.</li></ol><h3>Deploy our application with kafka_exporter</h3><p>So we have an application that consumes Kafka messages and we want to scale it out when our consumer_lag threshold is breached using <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/">hpa</a>. <br>The hpa scales the pod that it gets the metrics from. For that reason we need the kafka-exporter container to serve as a side car in our consumer pod.</p><p>In a sense this is not optimal because upon pod replication (of the consumer along with the exporter) the kafka-exporter containers will have the same metrics (as long as the number of the topic’s partitions is bigger than the number of consumers, aka pods), thus their replication is redundant. But again, the hpa will scale the pod with the metrics so we must connect the exporter with the application.</p><p>Before we begin, make sure to create a “my-topic” topic with 5 partitions (or as many as you want your hpa to scale out to). I will not cover <a href="https://kafka.apache.org/quickstart">how it’s done</a>.</p><p>Let’s deploy an example <em>kafka-consumer-application</em> deployment. It consists of a kafka_exporter container and our consumer application. <br>Our application is created from a Kafka image and constantly reads from<em> “my-topic”</em> topic as a part of the consumer group <em>”we-consume”</em> every second for 2 seconds.</p><pre># kafka-consumer-application.yaml</pre><pre>kind: <strong>Deployment</strong><br>apiVersion: extensions/v1beta1<br>metadata:<br>  name: kafka-consumer-application<br>  labels:<br>    app_name:  kafka-consumer-application<br>spec:<br>  revisionHistoryLimit: 30<br>  strategy:<br>    type: RollingUpdate<br>    rollingUpdate:<br>      maxUnavailable: 0<br>      maxSurge: 10<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app_name:  kafka-consumer-application<br>  template:<br>    metadata:<br>      labels:<br>        app_name:  kafka-consumer-application<br><strong>      annotations:<br>        prometheus.io/scrape: &#39;true&#39;<br>        prometheus.io/port: &#39;9308&#39;</strong><br>    spec:<br>      containers:<br>        - name: kafka-exporter<br>          image: danielqsj/kafka-exporter:v1.2.0<br>          imagePullPolicy: IfNotPresent<br>          command:<br><strong>            - kafka_exporter<br>            - &quot;--kafka.server=my-kafka-broker.kafka:9092&quot;<br>          ports:<br>            - name: http-metrics<br>              containerPort: 9308</strong><br>          readinessProbe:<br>            httpGet:<br>              path: kafka-application<br>              port: 9308<br>            initialDelaySeconds: 5<br>            timeoutSeconds: 5<br>        - name:  kafka-application<br>          image:  confluentinc/cp-kafka:5.0.1<br>          imagePullPolicy: IfNotPresent<br>          command:<br>            - sh<br>            - -c<br>            - |<br>              while true<br>                do<br>                kafka-console-consumer --bootstrap-server my-kafka-broker-headless.kafka:9092 --topic my-topic --from-beginning --group we-consume --timeout-ms 2000<br>                sleep 1<br>              done</pre><p>Notice that <strong>“my-kafka-broker-headless” is the name of the k8s service</strong> that exposes your Kafka brokers (on port 9092) and “<strong>kafka” is its namespace</strong>.</p><p>The metrics coming from kafka-exporter will be shown under “kafka-consumer-application:9308/metrics” . This is why we give these values in the annotation. Prometheus will scrape the data from our pod according to the values we give in the annotation. Learn more about annotations <a href="https://www.weave.works/docs/cloud/latest/tasks/monitor/configuration-k8s/#per-pod-prometheus-annotations">here</a>.</p><p>Let’s deploy this pod on consumer-deployment namespace</p><pre><strong>$ kubectl apply -f kafka-consumer-application.yaml</strong><strong> -n consumer-test</strong><br><em>deployment.</em>extensions<em>/kafka-consumer-application created</em></pre><pre><strong>$ kubectl get pods -n consumer-test</strong><em><br>kafka-consumer-application-dcc7df96-rwdrb   2/2  Running 0  107s</em></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LWiHk6WEI9hcJe_Kam_iPQ.png" /><figcaption>Our deployment so far</figcaption></figure><h3>Verify metrics are getting to Prometheus</h3><p>Make sure that Prometheus is getting the data from the exporter. You can do so using Grafana if you have it on your cluster — just play with the queries (import <a href="https://grafana.com/grafana/dashboards/7589">this dashboard</a> — if data is shown, you are fine). If you don’t see anything, try to edit one of the graph queries to be plain “kafka_consumergroup_lag”, you should see the metrics.</p><p>You can also verify connection by using Prometheus’s REST API. A possible command is</p><pre>curl &#39;http://localhost:9090/api/vi/series?match[]=kafka_consumergroup_lag&#39;</pre><p>(Here I assume you execute the command from the Prometheus server which has 9090 port exposed. That might not be the case, make sure you know the right endpoint for interacting with the Prometheus server.)</p><p>Now we know that Prometheus is getting the metrics from kafka_exporter which means we we are third of the way through!</p><h3>Figure out your query</h3><p>We must figure out the right query for us. I recommend doing so using Grafana, that way you can look at your queries and test them.</p><p>As mentioned before, we want to check the consumer lag of <em>“my-topic”</em> topic and <em>“we-consume”</em> consumer group. <br>For this use case <a href="https://prometheus.io/docs/prometheus/latest/querying/functions/#aggregation_over_time">the following query </a>is enough:</p><pre>avg_over_time(kafka_consumergroup_lag{topic=&quot;my-topic&quot;,consumergroup=&quot;we-consume&quot;}[1m])</pre><h3>Deploy Prometheus Adapter using Helm chart</h3><p>Now we need to install Prometheus Adapter pod. We will do so using its helm chart and passing our customized values.</p><p>Prometheus Adapter <a href="https://github.com/helm/charts/tree/master/stable/prometheus-adapter">helm chart repository </a>has very good documentation and so is the <a href="https://github.com/DirectXMan12/k8s-prometheus-adapter">actual resource metric API</a>. I strongly recommend reading it. By following their instructions I came up with this file</p><pre># values.yaml</pre><pre>prometheus:<br>  port: 80<br>  url: <a href="http://common-prometheus-server/">http://prometheus-server</a></pre><pre>rules:<br>  default: false<br>  resource: {}<br>  custom:<br>    - seriesQuery: &#39;kafka_consumergroup_lag&#39;<br>      resources:<br>        overrides:<br>          kubernetes_namespace: {resource: &quot;namespace&quot;}<br>          kubernetes_pod_name: {resource: &quot;pod&quot;}<br>      name:<br>       matches: &quot;kafka_consumergroup_lag&quot;<br>       as: &quot;kafka_consumergroup_lag&quot;<br>      metricsQuery: &#39;<strong>avg_over_time(kafka_consumergroup_lag{topic=&quot;my-topic&quot;,consumergroup=&quot;we-consume&quot;}[1m])</strong>&#39;</pre><p>To use the chart ensure that prometheus.url and prometheus.port are configured with the correct Prometheus service endpoint. <br>The easiest place for us to install the chart is in the same namespace as our Prometheus server pod (lets call this namespace prom-ns)<br>Now, install the chart by running</p><pre>helm upgrade -i prometheus-adapter stable/prometheus-adapter --version 1.2.0 -f values.yaml --namespace prom-ns</pre><p>In order to test that this went well, wait a minute or two and run the following command from <strong>one of your master nodes</strong>:</p><pre>kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/<strong>consumer-test</strong>/pod/*/kafka_consumergroup_lag | jq</pre><p>You should get</p><pre>{<br>  &quot;kind&quot;: &quot;MetricValueList&quot;,<br>  &quot;apiVersion&quot;: &quot;custom.metrics.k8s.io/v1beta1&quot;,<br>  &quot;metadata&quot;: {<br>    &quot;selfLink&quot;: &quot;/apis/custom.metrics.k8s.io/v1beta1/namespaces/consumer-test/pod/%2A/kafka_consumergroup_lag&quot;<br>  },<br>  &quot;items&quot;: [<br>    {<br>      &quot;describedObject&quot;: {<br>        &quot;kind&quot;: &quot;Pod&quot;,<br>        &quot;namespace&quot;: &quot;<strong>consumer-test</strong>&quot;,<br>        &quot;name&quot;: &quot;<strong>kafka-consumer-application-7bcb4fb9db-kg2xz</strong>&quot;,<br>        &quot;apiVersion&quot;: &quot;/v1&quot;<br>      },<br>      <strong>&quot;metricName&quot;: &quot;kafka_consumergroup_lag&quot;,</strong><br>      &quot;timestamp&quot;: &quot;2019-07-31T13:02:49Z&quot;,<br>      &quot;value&quot;: &quot;0&quot;<br>    }<br>  ]<br>}</pre><p>You can see in the response that we now have a metric named “kafka_consumergroup_lag” with a current value of 0. <br>If the “items” section is empty — there is something wring with your exporter. If you got Error from server.. that means something went wrong when you deployed the helm chart.</p><p>If you reached this far then you are two thirds of the way done :)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ogN6cxNwM026v-6eQu0D5g.png" /><figcaption>our deployment so far</figcaption></figure><h3>Creating the HPA</h3><p>As mentioned above, the hpa is the object that actually monitors if the metric is too high and scale the application accordingly. We will have to create the hpa in the “consumer-test” namespace in order to be able to scale our application.</p><p>In this example we will scale “kafka-consumer-application” deployment up to a maximum of 10 pods if the average kafka_consumergroup_lag goes above 5.</p><pre># consumer-hpa.yaml</pre><pre>kind: HorizontalPodAutoscaler<br>apiVersion: autoscaling/v2beta1<br>metadata:<br>  name: consumer-hpa<br>spec:<br>  scaleTargetRef:<br>    apiVersion: apps/v1<br><strong>    kind: Deployment<br>    name: kafka-consumer-application</strong><br>  minReplicas: 1<br>  maxReplicas: 10<br>  metrics:<br><strong>  - type: Pods<br>    pods:</strong><br>     <strong> metricName: &quot;kafka_consumergroup_lag&quot;</strong><br>      targetAverageValue: 5</pre><p>Notice that we have to give it the exact metric name as we got when querying the Custom Metrics API.</p><p>Let’s deploy</p><pre><strong>$ kubectl apply -f consumer-hpa.yaml</strong><strong> -n consumer-test</strong><br><em>horizontalpodautoscaler.autoscaling/consumer-hpa created</em></pre><p>It will take a minute or two until the hpa presents data. Then you should see</p><pre><strong>$ kubectl get hpa consumer-hpa -n consumer-test</strong><br><em>consumer-hpa Deployment/kafka-consumer-application 0/5 1 10 1  63s</em></pre><p>That’s it! Everything is connected.</p><p>The following scheme demonstrates the interactions between the items we deployed</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bYU9xthbVsrj99b_-C5-cw.png" /><figcaption>Our deployment end state</figcaption></figure><p>Now feel free to raise the consumer group lag by loading <em>“my-topic”</em> with messages and see the magic happens. When the hpa threshold will be breached the application would scale</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oBQiwCoBx-E1T9VGlVxjiA.png" /><figcaption>The hpa scales the application</figcaption></figure><h3>Resources</h3><ol><li><a href="https://github.com/luxas/kubeadm-workshop#deploying-a-custom-metrics-api-server-and-a-sample-app"><strong>Deploying a custom metrics API Server and a sample app</strong></a></li><li>kafka_exporter repository <a href="https://github.com/danielqsj/kafka_exporter">https://github.com/danielqsj/kafka_exporter</a></li><li>Prometheus Adapter for Kubernetes Metrics APIs <a href="https://github.com/DirectXMan12/k8s-prometheus-adapter">https://github.com/DirectXMan12/k8s-prometheus-adapter</a></li><li>Helm chart for Prometheus Adapter <a href="https://github.com/helm/charts/tree/master/stable/prometheus-adapter">https://github.com/helm/charts/tree/master/stable/prometheus-adapter</a></li><li><a href="https://medium.com/google-cloud/kubernetes-hpa-autoscaling-with-kafka-metrics-88a671497f07">https://medium.com/google-cloud/kubernetes-hpa-autoscaling-with-kafka-metrics-88a671497f07</a></li><li>This nice slideshow: <a href="https://speakerdeck.com/metalmatze/kubernetes-metrics-api?slide=26">https://speakerdeck.com/metalmatze/kubernetes-metrics-api?slide=26</a></li><li>consumer lag: <a href="https://sematext.com/blog/kafka-consumer-lag-offsets-monitoring/">https://sematext.com/blog/kafka-consumer-lag-offsets-monitoring/</a></li><li><a href="https://www.weave.works/docs/cloud/latest/tasks/monitor/configuration-k8s/#per-pod-prometheus-annotations">https://www.weave.works/docs/cloud/latest/tasks/monitor/configuration-k8s/#per-pod-prometheus-annotations</a></li><li><a href="https://github.com/stefanprodan/k8s-prom-hpa">Further reading on autoscaling</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f30fe99f3948" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating Stateful classes with Jenkins Shared-Library]]></title>
            <link>https://medium.com/@ranrubin/creating-stateful-classes-with-jenkins-shared-library-76d4014ef0b2?source=rss-59f5cda9f551------2</link>
            <guid isPermaLink="false">https://medium.com/p/76d4014ef0b2</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[groovy]]></category>
            <category><![CDATA[continuous-delivery]]></category>
            <dc:creator><![CDATA[Ran Rubin]]></dc:creator>
            <pubDate>Sat, 29 Jun 2019 20:39:57 GMT</pubDate>
            <atom:updated>2019-06-29T20:39:57.129Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/665/0*Sx-CkDRPT2VD0Xx-.jpg" /></figure><p>In this article I will introduce a new approach of using shared libraries in Jenkins. This approach allows the users of the Shared Library to give the classes <strong>state </strong>and work in an object oriented manner.</p><p>Prerequisites for this article is to be familiar with Jenkins and the concept of Jenkins Shared libraries. <strong>I will not cover</strong> <strong>what is Shared Library, how to create one and basic usage</strong> — for that please take a look at:</p><ul><li><a href="https://jenkins.io/doc/book/pipeline/shared-libraries/">Jenkins documentation</a></li><li><a href="http://www.aimtheory.com/jenkins/pipeline/continuous-delivery/2017/12/02/jenkins-pipeline-global-shared-library-best-practices.html">aimtheory blog</a></li><li><a href="https://medium.com/faun/a-hacky-hackers-guide-to-jenkins-scripted-pipelines-part-4-dd49fcb0d62">“a hacky hackers guide to Jenkins</a>”</li></ul><p><strong>Additional notes:</strong></p><ol><li>The code that I provide doesn’t necessarily work, but the idea does :)</li><li>I removed function’s documentation from the code snippets. Please refer to the <a href="https://github.com/ranrubin/Jenkins-Shared-Library-Objects">Github repo</a> links in each section to see the full code.</li></ol><h3><strong>Before Jenkins introduced Shared Libraries</strong></h3><blockquote><a href="https://github.com/ranrubin/Jenkins-Shared-Library-Objects/tree/before-shared-library">https://github.com/ranrubin/Jenkins-Shared-Library-Objects/tree/before-shared-library</a></blockquote><p>I’ll start by showing what usually happened before Jenkins introduced shared libraries.</p><p>Let’s say you use <a href="https://www.consul.io/">Consul service mesh</a> frequently in your Jenkins files. If you don’t know what it is — don’t worry, you only need to know that it’s a service that has REST API to interact with.</p><p>Your Jenkins files might look something like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e690aedb94d13c73244e667fbd72fe08/href">https://medium.com/media/e690aedb94d13c73244e667fbd72fe08/href</a></iframe><p>There is a lot of code repetition when you use these operations in many of your Jenkins files. This is not elegant, scalable or convenient.</p><h3><strong>With Shared-Libraries</strong></h3><blockquote><a href="https://github.com/ranrubin/Jenkins-Shared-Library-Objects/tree/with-shared-library">https://github.com/ranrubin/Jenkins-Shared-Library-Objects/tree/with-shared-library</a></blockquote><p>Then, Jenkins came up with the idea of <a href="https://jenkins.io/doc/book/pipeline/shared-libraries/">Shared-Libraries</a> which allows us to share code between different Jenkins jobs. Continuing with our example, we will create a class named <em>Consul</em> which will perform the common operations.</p><p>It may look something like :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b83c64891a61fe99269002a4f2fca3c3/href">https://medium.com/media/b83c64891a61fe99269002a4f2fca3c3/href</a></iframe><p>And the Jenkins file becomes:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/db0faf680961cef555055b5b7bbc6622/href">https://medium.com/media/db0faf680961cef555055b5b7bbc6622/href</a></iframe><p>There is no doubt that things are better now.</p><p>For example, if the API of Consul change and listing all nodes change from “/catalog/nodes” to something else, instead of fixing it in all of our Jenkins files, we merely have to change it in <em>Consul.groovy</em> which all jobs read from.</p><p>However, there is still code repetition here.</p><p>For instance, why do we have to pass the address of consul (denoted by <em>path</em> variable) at every function call in the Jenkins file? We would be better having a Consul instance with data members that preserve state.</p><p>Unfortunately as Ian David Rossi noted on <a href="http://www.aimtheory.com/jenkins/pipeline/continuous-delivery/2018/10/01/jenkins-pipeline-global-shared-library-best-practices-part-2.html">aimtheory blog</a> , you can’t use “pure” groovy classes and Jenkins’s DSL on the same time :(</p><p>So I figured a workaround that imitates a “pure” groovy object and thus allows you to avoid code repetition, preserve state and gives you much more power — power like object composition.</p><h3>Stateful Shared-Libraries</h3><blockquote><a href="https://github.com/ranrubin/Jenkins-Shared-Library-Objects/tree/stateful-shared-library">https://github.com/ranrubin/Jenkins-Shared-Library-Objects/tree/stateful-shared-library</a></blockquote><p>Take a look at the refactored<em> Consul.groovy</em> file:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bf910a83c08fd30fe7a9e895f0b7d00e/href">https://medium.com/media/bf910a83c08fd30fe7a9e895f0b7d00e/href</a></iframe><p>When we instantiate this class, we will call the “construct” function with the values we want to keep. You can see how our Jenkins file looks a bit cleaner now:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3e4d50b35622bc59ed44105e9130db8c/href">https://medium.com/media/3e4d50b35622bc59ed44105e9130db8c/href</a></iframe><p>As opposed to earlier <strong>we don’t pass the path at every single call.</strong></p><p>But this is still not enough.</p><p>Take a look at <em>Consul.groovy</em> again; “register” and “store” functions both use “curl” calls exactly the same way. Wouldn’t it be great to have an object that will manage this?</p><h3>Object Composition</h3><blockquote><a href="https://github.com/ranrubin/Jenkins-Shared-Library-Objects/tree/stateful-advanced">https://github.com/ranrubin/Jenkins-Shared-Library-Objects/tree/stateful-advanced</a></blockquote><p>Let’s call that object “HttpRest”. It will perform simple http interactions by wrapping curl commands. It will look something like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d0a1e0fd368ad79ba4e5ff05706fb2cf/href">https://medium.com/media/d0a1e0fd368ad79ba4e5ff05706fb2cf/href</a></iframe><p>And our <em>Consul.groovy</em>? Well, it looks much more elegant —</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a30ec99ccfd40983720fbbb93114da14/href">https://medium.com/media/a30ec99ccfd40983720fbbb93114da14/href</a></iframe><p>Getting the HttpRest object in our fake constructor help us imitate the idea of <a href="https://en.wikipedia.org/wiki/Object_composition">object composition</a> and gain its power.</p><h3><strong>Final Touch</strong></h3><blockquote><a href="https://github.com/ranrubin/Jenkins-Shared-Library-Objects/tree/stateful-advanced">https://github.com/ranrubin/Jenkins-Shared-Library-Objects/tree/stateful-advanced</a></blockquote><p>There is just one more thing — if we keep instantiating the classes on our Jenkins files with</p><pre>def consul = new org.foo.Consul()</pre><p>We are, in fact, coupling our code to our library directory structure. Imagine that in a few months you want to move <em>Consul.groovy</em> from<em> org/foo</em> to <em>org/boo</em>. That will require you to search and refactor all of your Jenkins files that had instantiate it . HELL.</p><p>This is why we will use the ability of Jenkins Shared Libraries to set global functions under <em>vars</em> directory. Lets add under vars directory the files <em>getHttpRestHandler.groovy</em> and <em>getConsulHandler.groovy</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c9f2750fe28a8b8237523be0592671cd/href">https://medium.com/media/c9f2750fe28a8b8237523be0592671cd/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6848852b1b7a566b2432788509a2d5bf/href">https://medium.com/media/6848852b1b7a566b2432788509a2d5bf/href</a></iframe><p>Our Jenkins file will now look something like</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/947ab9422ee7f2fa163da9ea60620f34/href">https://medium.com/media/947ab9422ee7f2fa163da9ea60620f34/href</a></iframe><p>Notice that with the getters under <em>var</em> we also force the user to call the “construct” function and thus avoid misuse of the classes.</p><p>Also, if you would like consul object to support object composition and the previous method (getting ip and port), you can simply create two constructors with different signatures.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=76d4014ef0b2" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>