<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Daniel Megyesi on Medium]]></title>
        <description><![CDATA[Stories by Daniel Megyesi on Medium]]></description>
        <link>https://medium.com/@dmegyesi?source=rss-1d8963de226e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*M2m4SMYTyBbyydS-soFxug.jpeg</url>
            <title>Stories by Daniel Megyesi on Medium</title>
            <link>https://medium.com/@dmegyesi?source=rss-1d8963de226e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 00:06:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dmegyesi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[My PodDisruptionBudget bible to use with Karpenter and friends]]></title>
            <link>https://medium.com/infrastructure-adventures/my-poddisruptionbudget-bible-to-use-with-karpenter-and-friends-1139226cc0d0?source=rss-1d8963de226e------2</link>
            <guid isPermaLink="false">https://medium.com/p/1139226cc0d0</guid>
            <category><![CDATA[autoscaling]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-eks]]></category>
            <category><![CDATA[karpenter]]></category>
            <dc:creator><![CDATA[Daniel Megyesi]]></dc:creator>
            <pubDate>Thu, 09 Jan 2025 23:35:07 GMT</pubDate>
            <atom:updated>2025-01-09T23:35:07.287Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<p>One of my frequent headaches with Karpenter have been node disruption events — or, more specifically, the lack of them happening, nodes waiting forever to be terminated and burning money while they are underutilized.</p><p>There’s a surprisingly steep learning curve with Kubernetes disruptions, especially when you deploy Helm charts made by someone else, which often include a <a href="https://kubernetes.io/docs/tasks/run-application/configure-pdb/">PodDisruptionBudget</a> (from now on: PDB) resource by default (which is, in general, a good thing).</p><p>However, in the case of Karpenter, there are many footguns here: a node cannot be terminated until there are pods still running on it which cannot be evicted because terminating them would hurt their so-called <em>disruption budget</em>:</p><blockquote>a contract with the Kubernetes control plane about the availability of an application, a guarantee that no matter what, a certain number of healthy, traffic-serving replicas will be always online even when planned or (in a best effort manner) unplanned events happen in the cluster.</blockquote><p><strong>A PDB can only be defined on</strong> pods that have a controller ensuring their state: <strong>Deployment</strong> (which also means: a ReplicaSet or ReplicationController can be specified as well) <strong>and StatefulSet</strong>. Individually created, standalone Pods cannot be referenced in a PDB manifest.</p><p>As usual, the upstream Kubernetes <a href="https://kubernetes.io/docs/concepts/workloads/pods/disruptions/">documentation</a> includes some great details about the different scenarios, use cases and the background behind disruption events, however what I have been missing is a hands-on reference that I can always just open up and (blindly) copy-paste into my Helm values files or in my applications without needing to learn and understand all the different Kubernetes concepts. My goal with this article is to have all of this noted down “for future reference”, thus the not-so-humble title for this article.</p><h3>The basics</h3><p>Let’s get to the chase! I’m going to add here basic concepts that work with any reasonably new Kubernetes version (stable since 1.21 but available much earlier). As the K8s ecosystem is evolving, there are more and more “addons”, new features, parameters added as optional fields, I’m going to mention one separately later and you can cherry-pick some best practices for your use-case that are compatible with your controlplane version.</p><p>A PDB expects 2 main things to be configured:</p><ol><li>reference to the app that you want to scale</li></ol><p>2. the disruption contract by 2 different fields, which are <strong>mutually exclusive:</strong></p><ul><li>minAvailable OR maxUnavailable</li></ul><p><strong>You can’t set both</strong>, so you need to understand which one is the best for your use case.</p><p>As their name suggests, <strong>minAvailable sets an absolute minimum number of replicas of your application that will be guaranteed</strong> by the control plane <strong>to run</strong>, no matter what happens (planned or unplanned events): this can be useful if your app can stay stable with a static replica count even if you have horizontal autoscaling.</p><p><strong>Basically for every other use case where you need to consider autoscaling, you should use </strong><strong>maxUnavailable</strong>: this can be also a static number such as <em>“I don’t want more than 1 unhealthy replica in any given moment”</em> so if your app is at 5 replicas or 37, a mass eviction / disruption will always happen one by one.</p><p>Both parameters also accept percentages, so you can say <em>“I want at least 51% of my replicas up all the time”</em> or <em>“proportional to my horizontal autoscaling, you can terminate up to 10% at the same time”</em>.</p><p>Some valid examples:</p><pre>apiVersion: policy/v1<br>kind: PodDisruptionBudget<br>metadata:<br>  name: zk-pdb<br>spec:<br>  minAvailable: 25%<br>  selector:<br>    matchLabels:<br>      app: zookeeper</pre><pre>apiVersion: policy/v1<br>kind: PodDisruptionBudget<br>metadata:<br>  name: zk-pdb<br>spec:<br>  maxUnavailable: 1<br>  unhealthyPodEvictionPolicy: AlwaysAllow<br>  selector:<br>    matchLabels:<br>      app: zookeeper</pre><h3>Do rolling restarts count as a disruption?</h3><p>The K8s documentation is very clear about this but it’s often missed by people:</p><blockquote><em>Pods which are deleted or unavailable due to a rolling upgrade to an application do count against the disruption budget, but workload resources (such as Deployment and StatefulSet) are not limited by PDBs when doing rolling upgrades.</em></blockquote><p>So it means <strong>if you do a Deployment/StatefulSet rolling restart, it will ignore the PDB and let it happen as quick as possible</strong> (just always waiting for the last new replica to become healthy),</p><p>however if you have a rolling restart in progress AND at the same time you drain a node, it will respect the total number of healthy replicas according to your PDB settings.</p><h3>Use cases</h3><h3>The tl;dr reference</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4gAH_M6fJs511zRilno-yA.jpeg" /><figcaption>The Bing Image Creator’s attempt to illustrate this article. Oil on canvas, gibberish text.</figcaption></figure><p>Hope you will find this article useful and actually read all the explanations below — in case an LLM will parse this article in the future (then it will probably hate me for it because Medium can’t handle tables in Markdown, so it’s a picture…) or you come back for a quick reference, here’s the gist of it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qc_QjEfqYycZc5STh7Yfag.png" /><figcaption>tl;dr</figcaption></figure><p>Additionally, a best practice is to <strong>set </strong><strong>UnhealthyPodEvictPolicy: AlwaysAllow</strong> on K8s 1.27 and above. See explanation below.</p><h3>Application with static 1 replica</h3><p>Be it a Deployment or StatefulSet (I won’t treat them separately from now on, the same rules apply), while technically you can deploy a PodDisruptionBudget for it, it doesn’t make much sense because you can’t ensure 100% availability and a maximum of 1 replica, you have to pick one condition and hurt the other. You will either have downtime or you would have to run a new replica in parallel of having the old one being terminated (which can cause conflicts depending on your app if both of them receive traffic).</p><p><strong>tl;dr never set a disruption budget if you will only have 1 replica running</strong></p><h3>Application with static 2 replicas</h3><p>Here you’re good to go with either minAvailable: 1 or maxUnavailable: 1, they will both have the same effect: you will always have 1 of the 2 pods serving traffic.</p><p><strong>I tend to use </strong><strong>maxUnavailable here</strong> because if for whatever reason I end up setting the replica count to static 1 half a year from now, I don’t have to remember to adapt the PDB, it won’t cause any unexpected surprises.</p><h3>Edge case: Horizontal autoscaling between 1 and 2 replicas</h3><p>This is a mix of the previous 2 scenarios and there isn’t a really good solution here with these primitives. If at the moment your app scales to 1 replica and you set minAvailable: 1, your app will never be evicted, it will burn the money forever on that currently allocated node. If you set maxUnavailable: 1, you can end up with 0 healthy replicas and lose traffic.</p><h3>Application with minimum 2 replicas and horizontal autoscaling</h3><p>This is when things start to get interesting. :) You can <strong>choose what to optimize for</strong>:</p><ul><li>the quickest possible remediation <em>(= eviction of all replicas, for example during a rolling restart) even if it temporarily hurts performance</em></li><li>OR stability/availability</li></ul><p>In the first case you might want to set <strong>minAvailable: 1</strong> (or whatever reasonably small number your app needs to serve traffic even if it gets slow for a few minutes). This <strong>will allow Karpenter</strong> (Cluster Autoscaler, etc.) to kick out and <strong>shuffle your pods in the shortest amount of time possible</strong>, in exchange of hurting the performance (or even <strong>risking an OOM in the only remaining healthy replica</strong>, etc.).</p><p>When you optimize for availability, you will only allow a <strong>slow, controlled pod eviction</strong>, such as <strong>with </strong><strong>maxUnavailable: 1</strong> — your traffic probably won’t really notice it but it can mean a very slow node replacement flow.</p><h3>Unhealthy Pod Eviction Policy</h3><p>This is a cool <a href="https://kubernetes.io/docs/tasks/run-application/configure-pdb/#unhealthy-pod-eviction-policy">feature</a> that was introduced in K8s 1.27 and became stable in K8s 1.31 and <strong>I never set a PDB without this</strong>.</p><p>You can configure how the PDB should behave in case you might not have the desired replica count in a healthy state: for example if you have an app with 2 replicas and <strong>you want to evict a pod that’s in </strong><strong>CrashLoopBackOff forever</strong> (due to an OOM event or due to a node issue): the PDB will never allow it to get evicted because it would hurt your minAvailable: 1 policy by killing the only remaining healthy pod.</p><p>This is when the <strong>UnhealthyPodEvictPolicy: AlwaysAllow</strong> comes to the rescue: it tells the PDB to ignore crashed pods that keep burning money on your infrastructure without actually serving traffic and being useful.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1139226cc0d0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/infrastructure-adventures/my-poddisruptionbudget-bible-to-use-with-karpenter-and-friends-1139226cc0d0">My PodDisruptionBudget bible to use with Karpenter and friends</a> was originally published in <a href="https://medium.com/infrastructure-adventures">Infrastructure adventures</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubeconfig cleanup for unreachable clusters]]></title>
            <link>https://medium.com/infrastructure-adventures/kubeconfig-cleanup-for-unreachable-clusters-0fb496d74a90?source=rss-1d8963de226e------2</link>
            <guid isPermaLink="false">https://medium.com/p/0fb496d74a90</guid>
            <category><![CDATA[notes-to-self]]></category>
            <category><![CDATA[kubeconfig]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[kubectl]]></category>
            <category><![CDATA[aws-eks]]></category>
            <dc:creator><![CDATA[Daniel Megyesi]]></dc:creator>
            <pubDate>Wed, 21 Aug 2024 09:29:34 GMT</pubDate>
            <atom:updated>2024-08-21T09:29:34.147Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<p>There are a bunch of tools, kubectl plugins to clean up a cluttered kubeconfig file. The common thing I found in them is that they just blindly nuke all of them, except for some whitelisted entitites.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QLwO0B7qoXxoVrbz" /><figcaption>Photo by <a href="https://unsplash.com/@allvar?utm_source=medium&amp;utm_medium=referral">Carl Tronders</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>My use case is a bit different, I have a dynamic list with more than 100 live clusters and 20–30 which are already decommissioned or were just temporary test clusters. In this case the cleanup tools fail on me.</p><p>Here’s a dirty bash oneliner to do the trick:</p><pre>for cx in $(kubectl config get-contexts -o name); do echo -e &quot;\n$cx&quot;; if ! kubectl version --context $cx --request-timeout 3s &gt;/dev/null; then kubectl config delete-context $cx; fi; done</pre><p>The key here was the --request-timeout flag, otherwise it can hang for a long time; the “healthcheck” is done by fetching the K8s control plane version.</p><p>It does not clean up the server, user sections from the config file, I only cared about the context list, but it can be easily extended to wipe those as well at the time of deletion.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0fb496d74a90" width="1" height="1" alt=""><hr><p><a href="https://medium.com/infrastructure-adventures/kubeconfig-cleanup-for-unreachable-clusters-0fb496d74a90">Kubeconfig cleanup for unreachable clusters</a> was originally published in <a href="https://medium.com/infrastructure-adventures">Infrastructure adventures</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Things I wish I knew earlier: Alpine Linux cmd package alias]]></title>
            <link>https://medium.com/infrastructure-adventures/things-i-wish-i-knew-earlier-alpine-linux-cmd-package-alias-f9fbe4256eda?source=rss-1d8963de226e------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9fbe4256eda</guid>
            <category><![CDATA[tips]]></category>
            <category><![CDATA[alpine-linux]]></category>
            <category><![CDATA[notes-to-self]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Daniel Megyesi]]></dc:creator>
            <pubDate>Fri, 10 Feb 2023 22:37:28 GMT</pubDate>
            <atom:updated>2023-02-10T22:37:28.918Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<p>I had to Google countless times what is the damn package name in Alpine Linux to install a simple dig or telnet command.</p><p>Seriously, who can remember bind-utils or inetutils-telnet?</p><p>To the rescue: Alpine has the cmd: alias:</p><pre>$ docker run --rm -ti alpine:3.17<br>/ # apk add -u dig<br>fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/main/x86_64/APKINDEX.tar.gz<br>fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/x86_64/APKINDEX.tar.gz<br>ERROR: unable to select packages:<br>  dig (no such package):<br>    required by: world[dig]<br><br>/ # apk add -u cmd:dig<br>(1/15) Upgrading libcrypto3 (3.0.7-r2 -&gt; 3.0.8-r0)<br>(2/15) Upgrading libssl3 (3.0.7-r2 -&gt; 3.0.8-r0)<br>(3/15) Installing fstrm (0.6.1-r1)<br>(4/15) Installing krb5-conf (1.0-r2)<br>(5/15) Installing libcom_err (1.46.5-r4)<br>(6/15) Installing keyutils-libs (1.6.3-r1)<br>(7/15) Installing libverto (0.3.2-r1)<br>(8/15) Installing krb5-libs (1.20.1-r0)<br>(9/15) Installing nghttp2-libs (1.51.0-r0)<br>(10/15) Installing protobuf-c (1.4.1-r1)<br>(11/15) Installing libuv (1.44.2-r0)<br>(12/15) Installing xz-libs (5.2.9-r0)<br>(13/15) Installing libxml2 (2.10.3-r1)<br>(14/15) Installing bind-libs (9.18.11-r0)<br>(15/15) Installing bind-tools (9.18.11-r0)<br>Executing busybox-1.35.0-r29.trigger<br>OK: 19 MiB in 28 packages<br><br>/ # which dig<br>/usr/bin/dig</pre><p>I wish I knew this in the past couple of years; Stackoverflow would have lost so much unnecessary traffic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BR982UCa0KWifTMc" /><figcaption>Photo by <a href="https://unsplash.com/es/@davidmatos?utm_source=medium&amp;utm_medium=referral">David Matos</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><em>Disclaimer: as the </em><a href="https://docs.alpinelinux.org/user-handbook/0.1a/Working/apk.html#_installing_packages"><em>Alpine Package Keeper docs</em></a><em> state that this alias is not foolproof if there are multiple packages containing the same binary, but it gets the job done for the typical use case.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9fbe4256eda" width="1" height="1" alt=""><hr><p><a href="https://medium.com/infrastructure-adventures/things-i-wish-i-knew-earlier-alpine-linux-cmd-package-alias-f9fbe4256eda">Things I wish I knew earlier: Alpine Linux cmd package alias</a> was originally published in <a href="https://medium.com/infrastructure-adventures">Infrastructure adventures</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing Unicron, our big data and machine learning platform]]></title>
            <link>https://medium.com/adevinta-tech-blog/introducing-unicron-our-big-data-and-machine-learning-platform-5c69fb648a41?source=rss-1d8963de226e------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c69fb648a41</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[data-and-machine-learning]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[kubeflow]]></category>
            <dc:creator><![CDATA[Daniel Megyesi]]></dc:creator>
            <pubDate>Tue, 08 Jun 2021 12:29:28 GMT</pubDate>
            <atom:updated>2021-11-04T11:22:23.099Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<h3>Introducing Unicron, our big data and Machine Learning platform</h3><p>There’s always more to learn from Kubernetes and how it can make your life easier. In this post, I’ll walk you through the setup of <a href="https://www.adevinta.com/">Adevinta’s</a> Big Data and Machine Learning platform, Unicron, developed using Kubernetes. This is for the tech part, but what I’d also like to share with you is how technology has affected our organisational culture and the way we work as a team.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RIeD-k8MnUr4hCEWv9WLjw.jpeg" /></figure><h3>The journey from Mesos to Kubernetes</h3><p>Before 2020, Adevinta already had a common big data platform, enabling various data teams across our marketplaces and Central Teams to run batch data processing jobs. One of the motivations behind having a big data platform was economy of scale. By having one centralised platform, managed by a few dedicated engineers, all internal users including Data Scientists and Engineers could focus 100% of their time on “actual data stuff” instead of managing infrastructure or worrying about <a href="https://aws.amazon.com/emr/">Hadoop clusters</a>.</p><p>Fast forward to the beginning of 2021:<strong> our on-call team of four DevOps engineers are already managing a total of 59 Kubernetes clusters</strong> and hosting the centralised platform behind Adevinta’s big data processing and Machine Learning jobs. This number is constantly oscillating based on current business demands. For example, last October we had around 70 live EKS clusters as part of this infrastructure. In this article, we’ll explain what led us to this achievement — not just on a technical, but also on an organisational level.</p><h3>Why did we build a new big data platform?</h3><p>Our previous infrastructure was a multi-tenant, single Mesos cluster, with a corresponding Elastic MapReduce cluster, all running in Amazon’s cloud. The platform was managing terabytes of RAM and many thousands of CPUs in a connected Hadoop setup. Engineers had to create Spark jobs using internal tooling, package up each task as a Docker image and launch it in the shared Mesos infrastructure. We had limited autoscaling capabilities and while we didn’t have serious on-call incidents, it was sometimes challenging to manage the infrastructure because one mistake could affect all tenants at the same time. Additionally, due to the size, we were spending hundreds of thousands of euros per month hosting the system.</p><p>Due to growing needs and internal reorganisations in the company, <strong>we came up with plans</strong> for a new platform, where we could leverage the latest trends in the industry and <strong>offer better capabilities to our internal customers — teams at Adevinta — while also operating at a significantly lower cost</strong>. We dedicated more than two quarters to building the new system from scratch before we welcomed our first production customer with 24/7 on-call support. We started demerging teams from the previous platform in just a matter of months.</p><h3>The new architecture</h3><p>The new system, called Unicron, consists of many independent Kubernetes clusters. <strong>Every customer (data team) gets up to three individual, isolated AWS accounts</strong> with an Elastic Kubernetes Service<strong> (EKS) cluster</strong> in each — <strong>one account for each of their environments </strong>(development, staging and production).</p><p>This way we can manage each customer with more flexibility, as some need smaller machines and run lots of small jobs, whereas others just run one job but use hundreds of gigabytes of memory.<strong> </strong>Some teams have highly critical and time sensitive batch jobs, while others only run workloads once every few days and can sustain outages, so are willing to compromise on lower costs in return.</p><p><strong>Due to the fact clusters are completely isolated from each other, we’ve achieved much better security levels than ever before,</strong> while also having a very clear cost segregation for each team, down to the last penny spent. <strong>The new system has also allowed us to support different reliability requirements and SLAs</strong>.</p><p>We’ve been operating the Unicron platform for almost two years and so far <strong>it has been a success, both financially and technically</strong>. On the one hand, <strong>we’ve reduced Adevinta’s yearly AWS bills by a million dollars</strong>, and on the other, we’ve not experienced any significant incidents or outages and now sleep extremely peacefully thanks to the carefully planned infrastructure and strict quality requirements we’ve set up at a team level. We’ve also been able to onboard more teams and workloads than ever before.</p><h3>The team behind Unicron</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YELfF9gZbmB5qHsrtfzX7A.jpeg" /><figcaption>Our recipe for a collaborative, great team spirit: yearly off-site team building events!</figcaption></figure><p><strong>There’s no good system without a good team behind it</strong>. Before starting to design the new platform, <strong>we spent months</strong> reorganising ourselves, adding new members to our team and <strong>setting up common agreements</strong> — not only on technological choices, but also <strong>on our ways of working together</strong>.</p><p>On top of our usual daily tasks and OKRs, every week we created new proposals in the form of a GitHub issue, where everyone could add comments and opinions. We’d then have a dedicated, time-boxed session to debate each topic. <strong>It was a very democratic process, and we ended up with agreements that are documented </strong>in our internal knowledge base. From time to time, we revisit these documents and change or adapt them if needed. Each time a new member joins, they can go through all of the agreements and understand how we work.</p><p>These agreements are about many different topics, for example:</p><ul><li>Which programming languages are accepted for new projects</li><li>What are the quality requirements for writing Prometheus alerts (we write unit tests for 100% of our on-call alerting rules!)</li><li>How to handle technical debt; what future guarantees do we commit to if we need to compromise on quality when facing time pressure</li><li>How to perform good and meaningful pull request reviews, etc.</li></ul><blockquote><strong>We’ve agreed to dedicate time every quarter to upgrade at least one version of Kubernetes </strong>and take control of version upgrades to keep us up-to-date <strong>and avoid the external time pressure posed by AWS</strong>. We made sure our infrastructure is capable of handling this procedure and that <strong>it’s part of our normal daily tasks with close to zero preparation</strong> required. Last quarter, in just under two months, we successfully upgraded three major versions of Kubernetes to 1.18 during business hours, thanks to this approach.</blockquote><p><strong>When we start working on new projects or bigger changes</strong>, like introducing single-sign on authentication for our services or planning a migration to a new monitoring platform, <strong>we make sure everyone is an equal part of the process and owns the ecosystem</strong> we build and maintain together.</p><p>Today our team consists of 10 people from four nationalities and several domains of expertise, including DevOps and Software Engineers, Data/Machine Learning specialists and Product and Engineering Managers. We all work together towards a common goal: providing the best possible platform to make our internal users’ lives easier.</p><p><strong>After we grew to a certain size, we split our work into two squads</strong>: one squad now works on the core platform, a strong and reliable base, while the other focuses on developing the customer-specific tooling on top of it, to provide better value for big data and Machine Learning capabilities.</p><h3>The technology behind Unicron</h3><h4>Batteries included but swappable</h4><p>The platform quickly evolved from simply being a place for big data jobs and a drop-in replacement for our previous solution to a platform that can answer new business needs. For example, running daily vulnerability scans for Adevinta’s Security team or allowing Adevinta engineers to leverage various Machine Learning technologies.</p><p>By slowly evolving the platform, <strong>we’ve isolated core common components, </strong>for example a control plane, the logging system or autoscaling. <strong>We called this Unicron Core, and on top of this, we offer optional components, for example</strong>:</p><ul><li>Spark jobs</li><li>Luigi scheduler</li><li>Kubeflow or Katib</li><li>Custom Datadog monitoring</li></ul><p>The teams are now able to install these well-tested components in a self-service way, like installing an app through the app store on your phone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*7X8jxjIlDDoJVUIoHfQJlQ.png" /><figcaption>“Layers. Onions have layers. Ogres have layers. Onions have layers. You get it? We both have layers.” <em>(Shrek)</em></figcaption></figure><h4>Here comes Kubernetes</h4><p>We knew we wanted to run everything on top of Kubernetes, but we had internal debates and proofs of concepts to decide whether we wanted to install our own Kubernetes control plane or, for example, <a href="https://github.com/gardener/gardener">use federated clusters</a>. In the end, we decided to outsource all this pain to Amazon for a total of $70/month per cluster and go all-in on EKS. It was worth every penny!</p><p><strong>A common misconception about Kubernetes is that </strong>with the availability of EKS or Google Cloud’s GKE, <strong>you can just install it and then launch your workload</strong>. While this might be true for a sandbox environment as AWS takes care of the (otherwise difficult) control plane installation/upgrades and container networking, <strong>there are a lot of extra pieces you still need to figure out</strong>.</p><p>In order to use Kubernetes in production, these are the actions we undertook as a team:</p><ul><li>install and configure the cluster autoscaler and vertical pod autoscaler — and make sure they were <a href="https://medium.com/infrastructure-adventures/vertical-pod-autoscaler-deep-dive-limitations-and-real-world-examples-9195f8422724">correctly tuned</a></li><li>set up the AWS Application Load Balancers and wildcard ACM certificates to provide a strict HTTPS — only ingress</li><li>set up logging: fluentd, fluentbit shipping logs to S3; querying and searching them with AWS Athena and Jupyter notebooks</li><li>set up monitoring and observability: we use Prometheus and <a href="https://grafana.com/products/cloud/">Grafana</a> for metrics and alerting, and have our own end-to-end test to validate each cluster’s availability and reliability from the customer’s point of view</li><li>deploy <a href="https://github.com/bitnami-labs/sealed-secrets">secrets from git</a> in a secure way</li><li>add smaller, but important pieces like the <a href="https://github.com/cloud-custodian/cloud-custodian">Cloud Custodian</a>, detect if we’re running out of AWS quotas and automatically open a support ticket to raise the number of EC2 instance limits, VPC limits, etc.</li><li>add support for graceful termination for spot instances (or, in fact, even for normal machines, with our in-house node drainer app written in Go)</li><li>set up drivers for GPU or <a href="https://aws.amazon.com/machine-learning/elastic-inference/">Elastic Inference instances</a>, allow connection to <a href="https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints.html">services running in another AWS accounts</a> and VPCs, etc.</li><li>…and finally, the key piece to Unicron: set up <a href="https://argoproj.github.io/projects/argo-cd">ArgoCD</a> and <a href="https://argoproj.github.io/projects/argo/">Argo Workflows</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MO49ys1X8mtcUW8zBuUqAw.png" /><figcaption>The bits and pieces of the Unicron platform</figcaption></figure><h4>The ecosystem</h4><p>We rely heavily on the <a href="https://argoproj.github.io/argo-cd/">Argo</a> ecosystem and therefore on the <strong>GitOps methodology</strong>. Our customers write their own tasks, Spark jobs, etc., in their preferred way and then feed them to our <strong>internal helper tools. These tools generate a Docker image and create a commit to the customer’s git repository</strong>. ArgoCD detects this change in the repo and automatically deploys to the respective cluster. <strong>This also makes it very easy for our customers to replicate the same behaviour across multiple environments by promoting the same commit to a different branch of their repository</strong>.</p><p>Everything in the Unicron ecosystem is <strong>fully decentralised</strong>; each customer lives and works in their own private environment, strictly isolated by IAM roles, VPN restrictions and SSO authentication. All of these lists are <strong>self-service</strong>, meaning the customers can edit their own access lists and commit them to git, then our bells and whistles will take care of the rest.</p><p><strong>The clusters and their EC2 instances are expected to tolerate complete redeployments</strong>, rolling node replacements and live control plane upgrades. They either do these <strong>transparently in the background</strong>, or in case of a redeployment they pick up the work where it was left off before deleting the cluster (redeploying safely stored Kubernetes Secrets from git, launching jobs, etc.).</p><p><strong>There’s no system without technical debt</strong>. Although we’ve set up very strict quality requirements, it’s not always possible or realistic to go all-in and end up with a perfect solution. If we have to compromise on something, we make sure it’s agreed and reviewed by the majority of the team. After documenting the solution, we create a tech debt GitHub issue so we don’t forget to fix it.</p><p><strong>Every quarter we have a dedicated two-three week window to fix technical debt</strong>. We open a discussion to vote on the most pressing issues to be fixed or improved and then get it done step by step, making our systems, life and sleep better every day.</p><h3>The community behind Unicron</h3><p><strong>One of the best things that helped evolve our internal product was setting up a strong community </strong>behind Unicron and its different services. This not only offloads work, but also provides a channel to share ideas and problems, spark innovation, get constant feedback about features and agree on the direction we should follow.</p><p><strong>We built thorough documentation</strong> for all of our infrastructure, with a lot of examples, best practices and recommendations. Additionally, <strong>each time we onboard a new customer, we start with a dedicated workshop</strong> to get them up to speed quickly.</p><p>We discovered we needed two different communities based on the main focus and interests of different teams. So <strong>we now have Machine Learning and big data communities,</strong> where we hold bi-weekly sync sessions to discuss current progress, and internal meetups, where team members or customers can present their latest project or idea. We share the slides and recordings of the sessions so others can also recap.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OUQMdas7VfX6kHROLiDXJg.jpeg" /><figcaption>If we have a lot of people contributing even just one rock at a time, we end up with something amazing!</figcaption></figure><h3>Do you want to be a part of this journey?</h3><p>Thanks to a lot of love, passion and great team spirit, we came up with technical solutions and interesting engineering ideas while building Unicron. We could write page after page to tell you more about it, but wouldn’t it be better if you saw it yourself? Come on and <a href="https://careers.smartrecruiters.com/Adevinta">join the team</a> — we’re hiring! 😉</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c69fb648a41" width="1" height="1" alt=""><hr><p><a href="https://medium.com/adevinta-tech-blog/introducing-unicron-our-big-data-and-machine-learning-platform-5c69fb648a41">Introducing Unicron, our big data and machine learning platform</a> was originally published in <a href="https://medium.com/adevinta-tech-blog">Adevinta Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Vertical Pod Autoscaler deep dive, limitations and real-world examples]]></title>
            <link>https://medium.com/infrastructure-adventures/vertical-pod-autoscaler-deep-dive-limitations-and-real-world-examples-9195f8422724?source=rss-1d8963de226e------2</link>
            <guid isPermaLink="false">https://medium.com/p/9195f8422724</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[deep-dives]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[scaling]]></category>
            <dc:creator><![CDATA[Daniel Megyesi]]></dc:creator>
            <pubDate>Fri, 21 Feb 2020 20:52:39 GMT</pubDate>
            <atom:updated>2020-02-21T20:52:39.849Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<p>Unfortunately, there’s a lack of good, useful examples on the Internet regarding how the Kubernetes <a href="https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler">Vertical Pod Autoscaler</a> (VPA) <em>actually</em> works; since I already had to spend a great deal of my time to figure things out, I hope this will prove useful to others as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Kj-SX_zV5BIp1Mn0" /><figcaption>Photo by <a href="https://unsplash.com/@ronit_sha?utm_source=medium&amp;utm_medium=referral">Ronit Shaked</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I’m assuming you already have the VPA software installed and running on your cluster.</p><p>I’m also assuming you’re familiar with horizontal vs. vertical scaling.</p><h3>Architecture</h3><p>This application adds a custom resource called VerticalPodAutoscaler and registers an admission webhook in the K8s API.</p><p>If this VPA object is defined for the pod, the hook will change its requests and limits in the pod template, according to the VPA recommender.</p><p>There are 3 Deployments running:</p><ul><li><strong>VPA admission hook</strong><br>Every pod submitted to the cluster goes through this webhook automatically which checks whether a VerticalPodAutoscaler object is referencing this pod or one of its parents (a ReplicaSet, a Deployment, etc.)</li><li><strong>VPA recommender<br></strong>Connects to the metrics-server application in the cluster, fetches historical and current usage data (CPU and memory) for each VPA-enabled pod and generates recommendations for scaling up or down the requests and limits of these pods.</li><li><strong>VPA updater<br></strong>Runs every 1 minute. If a pod is not running in the calculated recommendation range, it <strong>evicts the currently running version of this pod</strong>, so it can restart and go through the VPA admission webhook which will change the CPU and memory settings for it, before it can start.</li></ul><p><strong>Only the pod’s runtime settings will be changed.</strong></p><p><strong>The original Deployment specification will be untouched</strong>, so practically there will be a divergence with what the Deployment defines vs. what’s running in reality.</p><p>But this has a nice side-effect: for example, if you’re using Argo CD, it won’t detect any diff in the Deployment specification.</p><h3>Requests vs. limits</h3><p><strong>In Kubernetes, every scheduling decision is always made based on the </strong><strong>resource requests.</strong> Whatever number you put there, the scheduler will use it to allocate place for your pod.</p><p>Optionally, you can configure resource limits, but these are <em>never</em> used by the scheduler. This is just a hard limit for the Kubelet to know when to throttle or kill your pod if it exceeds these cgroup limits.</p><p>So since the only truly important things is the requests parameter, the Vertical Pod Autoscaler will always work with this. Whenever you define vertical autoscaling for your app, <strong>you are defining what the requests should be</strong>.</p><p>It is possible to set a range for the autoscaling: minimum and maximum values, <strong>for the requests</strong>.</p><p>So what happens to the limits parameter of your pod? Of course they will be also adapted, when you touch the requests line. The VPA will <em>proportionally</em> scale limits.</p><p>Here’s an example:</p><p>Your pod’s default settings:</p><pre>requests:<br>  cpu: 50m<br>  memory: 100Mi<br>limits:<br>  cpu: 200m<br>  memory: 250Mi</pre><p>The recommendation engine will determine that you need 120m CPU and 300Mi memory for your pod to work correctly. So it will come up with the following new settings:</p><pre>requests:<br>  cpu: 120m<br>  memory: 300Mi<br>limits:<br>  cpu: 480m<br>  memory: 750Mi</pre><p>As mentioned above, this is <em>proportional</em> scaling: in your default manifest, you had the following requests to limits ratio:</p><ul><li>CPU: 50m → 200m: 1:4 ratio</li><li>memory: 100Mi → 250Mi: 1:2.5 ratio</li></ul><p>So when you get a scaling recommendation, <strong>it will respect and keep the same ratio you originally configured</strong>, and proportionally set the new values based on your original ratio.</p><p>So if you want to ensure your memory limit never goes above 250Mi on your pod, here are some ideas:</p><ul><li>(always) configure minimum and maximum values for the request recommendation</li><li>use 1:1 request-to-limit ratio, so even if you get the maximum request, the limit won’t go above either</li><li>any combination of the above, play with the ratios</li></ul><p>But don’t forget, your limits are almost irrelevant, as the scheduling decision (and therefore, resource contention) will be always done based on the requests. Limits are only useful when there&#39;s resource contention or when you want to avoid uncontrollable memory leaks.</p><h3>How to write VPA manifests</h3><p>Never define more than 1 VPA object for the same Pod / ReplicaSet / Deployment / StatefulSet — the behaviour is unpredictable in such cases.</p><p>Don’t use Horizontal Pod Autoscaler and Vertical Pod Autoscaler on the same pods neither.</p><h4>Recommendation mode (dry run)</h4><p>You need to define a new VPA object with updateMode: off, targeting your application:</p><pre>apiVersion: autoscaling.k8s.io/v1beta2<br>kind: VerticalPodAutoscaler<br>metadata:<br>  name: kube-resource-report-recommender<br>spec:<br>  targetRef:<br>    apiVersion: &quot;apps/v1&quot;<br>    kind:       Deployment<br>    name:       kube-resource-report<br>  updatePolicy:<br>    updateMode: &quot;Off&quot;</pre><p>After 2–3 minutes, you will be able to query the data:</p><pre>$ kubectl describe vpa kube-ops-view-recommender</pre><pre>Name:         kube-resource-report-recommender<br>Namespace:    medium<br>API Version:  autoscaling.k8s.io/v1beta2<br>Kind:         VerticalPodAutoscaler</pre><pre>[...]</pre><pre>Spec:<br>  Target Ref:<br>    API Version:  apps/v1<br>    Kind:         Deployment<br>    Name:         kube-resource-report<br>  Update Policy:<br>    Update Mode:  Off<br>Status:<br>  Conditions:<br>    Last Transition Time:  2020-01-28T16:07:38Z<br>    Status:                True<br>    Type:                  RecommendationProvided<br>  Recommendation:<br>    Container Recommendations:<br>      Container Name:  kube-resource-report<br>      Lower Bound:<br>        Cpu:     12m<br>        Memory:  247917233<br>      Target:<br>        Cpu:     63m<br>        Memory:  380258472<br>      Uncapped Target:<br>        Cpu:     63m<br>        Memory:  380258472<br>      Upper Bound:<br>        Cpu:           137m<br>        Memory:        561393834<br>      Container Name:  nginx<br>      Lower Bound:<br>        Cpu:     12m<br>        Memory:  131072k<br>      Target:<br>        Cpu:     12m<br>        Memory:  131072k<br>      Uncapped Target:<br>        Cpu:     12m<br>        Memory:  131072k<br>      Upper Bound:<br>        Cpu:     16m<br>        Memory:  131072k<br>Events:          &lt;none&gt;</pre><p>You will see the following blocks:</p><p>Container name: A</p><ul><li>Uncapped Target: what would be the resource request configured on your pod if you didn’t configure upper limits in the VPA definition.</li><li><strong>Target: this will be the actual amount configured at the next execution</strong> of the admission webhook. (If it already has this config, no changes will happen (your pod won’t be in a restart/evict loop). Otherwise, the pod will be evicted and restarted using this target setting.)</li><li>Lower Bound: when your pod goes below this usage, it will be evicted and downscaled.</li><li>Upper Bound: when your pod goes above this usage, it will be evicted and upscaled.</li></ul><p>Container name: B</p><ul><li>… same as above, container by container.</li></ul><p><strong>TIP: Use recommendation mode even if you don’t want autoscaling!</strong></p><p>It is useful to have automatic prediction coming from actual data, then you can tune your hard-coded settings to those. <strong>It is not necessary to enable actual autoscaling, even if you keep collecting data and recommendations.</strong></p><h4>Blacklisting containers from scaling</h4><p>Let’s say, I have a Prometheus deployment, which has 2 containers: the Prometheus app itself and a config reloader sidecar. I don’t want to have autoscaling for the sidecar pod:</p><pre>apiVersion: autoscaling.k8s.io/v1beta2<br>kind: VerticalPodAutoscaler<br>metadata:<br>  name: prometheus-recommender<br>spec:<br>  targetRef:<br>    apiVersion: &quot;apps/v1&quot;<br>    kind:       Deployment<br>    name:       prometheus-server<br>  updatePolicy:<br>    updateMode: &quot;Off&quot;<br>  resourcePolicy:<br>    containerPolicies:<br>    - containerName: &quot;configmap-reload&quot;<br>      mode: &quot;Off&quot;</pre><p>You can also set containerName: &quot;*&quot; and use as a catch-all, configure some other parameters for everything, then individually blacklist the rest what you don&#39;t want to autoscale.</p><h4>Configuring minimum and maximum values</h4><p>Continuing with the previous example, now I want to define minimum and maximum limits for the autoscaler:</p><pre>apiVersion: autoscaling.k8s.io/v1beta2<br>kind: VerticalPodAutoscaler<br>metadata:<br>  name: prometheus-recommender<br>spec:<br>  targetRef:<br>    apiVersion: &quot;apps/v1&quot;<br>    kind:       Deployment<br>    name:       prometheus-server<br>  updatePolicy:<br>    updateMode: &quot;Off&quot;<br>  resourcePolicy:<br>    containerPolicies:<br>    - containerName: &quot;prometheus&quot;<br>      minAllowed:<br>        cpu: &quot;300m&quot;<br>        memory: &quot;512Mi&quot;<br>      maxAllowed:<br>        cpu: &quot;1800m&quot;<br>        memory: &quot;3600Mi&quot;<br>    - containerName: &quot;configmap-reload&quot;<br>      mode: &quot;Off&quot;</pre><blockquote>Reminder: the VPA minimum and maximum range settings are always for the <em>requests</em> parameter of your pod.</blockquote><h4>Enable autoscaling</h4><p>Use updateMode: Auto. You can use all the rest of the settings (minimum, maximum, container selector, etc.) already in recommendation mode. (So the only difference is dry-run vs. actually doing the changes.)</p><h3>Limitations</h3><ul><li><strong>The minimum memory allocation the VPA can recommend is 250 MiB.</strong><br>If your requests is smaller, it will be automatically increased to fit this minimum. This minimum value can be globally <a href="https://github.com/kubernetes/autoscaler/issues/2125#issuecomment-503987877">configured</a> though. This setting basically guides you not to use vertical autoscaler for too small applications.</li><li>The Vertical Pod Autoscaler cannot be used on individual pods which don’t have an owner (= they are not part of a Deployment, etc.).</li><li>By default you do not have the Prometheus integration enabled, which allows to watch for OOM events and fetch more historical performance data.<br>Following the example above, we want to use VPA to scale Prometheus itself, so unless you have persistent storage configured for the metrics store of Prometheus, once it gets restarted, we would lose all the historical, useful data which VPA expects for making scaling decisions.</li><li><strong>The Service name must be vpa-webhook </strong><em>(and all the components must be deployed in kube-system if you generate the certs with the built-in scripts)</em>, due to some <a href="https://github.com/kubernetes/autoscaler/blob/vertical-pod-autoscaler-0.6.3/vertical-pod-autoscaler/pkg/admission-controller/config.go#L72">hardcoded</a> things.</li></ul><blockquote>Once <a href="https://github.com/kubernetes/autoscaler/pull/2455">this PR</a> is released in a &gt;0.6.3 version, we can use registerByURL+ webhookAddress, webhookPort flags in the admission-controller deployment.<br>Make sure the SSL certificate accepts your URL as a CN!</blockquote><ul><li><strong>You can only enable vertical scaling on components which have at least 2 healthy replicas running.</strong> In case you have some apps which are not able to run in parallel, yet you want to autosize them, there’s a nice trick you can do in the vpa-updater Deployment:</li></ul><pre>[...]<br>    spec:<br>      containers:<br>      - name: updater<br>        args:<br>        # These 2 lines are default: <a href="https://github.com/kubernetes/autoscaler/blob/vertical-pod-autoscaler-0.6.3/vertical-pod-autoscaler/pkg/updater/Dockerfile#L22">https://github.com/kubernetes/autoscaler/blob/vertical-pod-autoscaler-0.6.3/vertical-pod-autoscaler/pkg/updater/Dockerfile#L22</a><br>        - --v=4<br>        - --stderrthreshold=info<br>        # Allow Deployments with only 1 replica to be restarted with new settings<br>        - --min-replicas=1</pre><pre>[...]</pre><ul><li>Something we learnt the hard way: once you deploy and <strong>enable the Vertical Pod Autoscaler in your cluster, everything goes through its webhook</strong>. Every single pod creation/restart/… event!</li></ul><p>So guess what happens when your VPA Admission Webhook deployment is down or extremely slow… <strong>None of your pods will be able to start, even if they don’t use vertical autoscaling at all!</strong> <em>(This is not specific to VPA, it’s true for any Admission Webhooks.)</em></p><p>There was <a href="https://github.com/kubernetes/kubernetes/issues/67662">a bug in Kubernetes</a> which prevented the controller-manager to start your pods if one of these webhooks stopped responding — it did not handle timeouts correctly. Luckily now it is fixed, however it is only backported until Kubernetes 1.15 or above. If you use something older than this (I’m looking at you, latest EKS 1.14!), you might have a problem.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FMT9NBm8FLNGPmuDTRe%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FMT9NBm8FLNGPmuDTRe%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FMT9NBm8FLNGPmuDTRe%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/b2f69dd8928c3015ccd2a9c8e135f785/href">https://medium.com/media/b2f69dd8928c3015ccd2a9c8e135f785/href</a></iframe><h3>Metrics, monitoring</h3><p>If you enable scraping on the following ports (define them in the Deployment + add the Prometheus annotation), you can have some pretty nice internal metrics available:</p><ul><li>admission-controller: tcp/8944</li><li>vpa-updater: tcp/8943</li><li>vpa-recommender: tcp/8942</li></ul><p>These are the metrics I found useful, as of writing this article:</p><p><strong>Recommender</strong></p><pre># HELP vpa_recommender_aggregate_container_states_count Number of aggregate container states being tracked by the recommender<br># TYPE vpa_recommender_aggregate_container_states_count gauge</pre><pre># HELP vpa_recommender_execution_latency_seconds Time spent in various parts of VPA Recommender main loop.<br># TYPE vpa_recommender_execution_latency_seconds histogram</pre><pre># HELP vpa_recommender_recommendation_latency_seconds Time elapsed from creating a valid VPA configuration to the first recommendation.<br># TYPE vpa_recommender_recommendation_latency_seconds histogram</pre><pre># HELP vpa_recommender_vpa_objects_count Number of VPA objects present in the cluster.<br># TYPE vpa_recommender_vpa_objects_count gauge</pre><p><strong>Updater</strong></p><pre># HELP vpa_updater_evicted_pods_total Number of Pods evicted by Updater to apply a new recommendation.<br># TYPE vpa_updater_evicted_pods_total counter</pre><pre># HELP vpa_updater_execution_latency_seconds Time spent in various parts of VPA Updater main loop.<br># TYPE vpa_updater_execution_latency_seconds histogram</pre><p><strong>Admission controller</strong></p><p>So far nothing implemented, it just listens with an empty default Prometheus endpoint.</p><p>I hope some of the above mentioned things helped to clarify how the Vertical Pod Autoscaler works in the background and the hands-on examples proved useful.</p><p>I personally really love VPA and I think it’s a very powerful tool, if used correctly. If you have ever had the chance to integrate it with Prometheus and have some first-hand experience, would be happy to hear about your insights!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9195f8422724" width="1" height="1" alt=""><hr><p><a href="https://medium.com/infrastructure-adventures/vertical-pod-autoscaler-deep-dive-limitations-and-real-world-examples-9195f8422724">Vertical Pod Autoscaler deep dive, limitations and real-world examples</a> was originally published in <a href="https://medium.com/infrastructure-adventures">Infrastructure adventures</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hands-on: Lustre FSx plugin for EKS, shared storage for machine learning workloads on K8s]]></title>
            <link>https://medium.com/infrastructure-adventures/hands-on-lustre-fsx-plugin-for-eks-shared-storage-for-machine-learning-workloads-on-k8s-c66d90d2b06a?source=rss-1d8963de226e------2</link>
            <guid isPermaLink="false">https://medium.com/p/c66d90d2b06a</guid>
            <category><![CDATA[lustre]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[kubeflow]]></category>
            <dc:creator><![CDATA[Daniel Megyesi]]></dc:creator>
            <pubDate>Fri, 21 Feb 2020 18:55:21 GMT</pubDate>
            <atom:updated>2020-02-21T18:55:21.486Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<h3>Hands-on with the Lustre FSx plugin for EKS, a shared file system provisioner for machine learning workloads on Kubernetes</h3><p>End of 2019 AWS <a href="https://aws.amazon.com/about-aws/whats-new/2019/12/amazon-eks-announces-beta-release-amazon-fsx-lustre-csi-driver/">released</a> a Lustre storage driver plugin for their managed Kubernetes solution, EKS. (<em>Technically, it should work on any Kubernetes installation of 1.14 or above, running in AWS.</em>)</p><p>Their managed Lustre filesystem, FSx is a pretty nice solution to quickly deploy a high performance, POSIX capable filesystem which allows concurrent writes from multiple clients. This is a very helpful tool in machine learning workloads, when you need to re-use the same training data, especially since you can attach this Lustre FS totally transparently to an S3 storage in the background, so you will see the S3 bucket’s contents as a standard filesystem <em>(with concurrent writes!)</em>.</p><p>This sounds very nice on paper, and according to AWS, it’s ready for production <em>(despite its beta state)</em>, so I decided to do a quick review and analysis regarding the ease of deployment, operation and to find out the possible limitations, before we commit to potentially offering this to our internal customers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*C0-iMk02lnXtVnpd" /><figcaption>Photo by <a href="https://unsplash.com/@paul_nic?utm_source=medium&amp;utm_medium=referral">Paolo Nicolello</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Analysis and findings</h3><p>This is more or less a brain dump, please excuse me for the format, I didn’t want the article to be even longer with too much explanation. I assume you’re familiar with the basic AWS components.</p><h4>To get it up and running, on an existing Kubernetes cluster</h4><ul><li>create: new <strong>IAM policy</strong> for FSx + new <strong>IAM role</strong> which need to be attached with Kiam to both the controller Deployment and the per-node agent DaemonSet</li></ul><blockquote>Although I’m talking about <a href="https://github.com/uswitch/kiam">Kiam</a> here, as a generic and more popular solution, by default AWS demoes this with their native Kubernetes-to-IAM implementation: <a href="https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html">IAM Roles for Service Accounts</a>.</blockquote><ul><li>create: new <strong>security group</strong> (SG) with tcp/988 bidirectional, attach it to worker nodes and the auto provisioned Lustre FS</li><li><strong>deploy</strong>: <a href="https://github.com/kubernetes-sigs/aws-fsx-csi-driver/tree/master/deploy/kubernetes/base">this</a> is a Kustomize deployment, kubectl apply -kinstalls it; only replacement happening is Docker image addresses</li></ul><blockquote>Sorry about the unspecific URL on <em>master</em>, the plugin is under heavy development, they don’t put git tags properly at the time of writing this article.</blockquote><ul><li>after deploy:<br> — patch image URL to use eu-west-1 (your region)<br> — patch Deployment to use IAM role annotation<br> — patch DaemonSet to use IAM role annotation</li><li>patch kube-system namespace to allow Kiam to assume roles inside the same AWS account<br> — alternatively, move all the FSx stuff under your own (Kiam-enabled) namespace, instead of the default hardcoded kube-system</li></ul><h4>Components</h4><ul><li><strong>controller</strong> runs as an HA deployment with leader election; watches for PersistentVolumeClaims</li><li><strong>node agent</strong> runs on every potential worker node, in privileged mode, mounting system disk + host network</li><li><strong>StorageClass</strong>: defines the Lustre endpoint: can be dynamic as well, no need to pre-create the FS <em>(as you need it in their </em><a href="https://docs.aws.amazon.com/eks/latest/userguide/efs-csi.html"><em>other storage plugin</em></a><em>, the managed NFS solution, AWS EFS</em>)</li><li>must specify in the StorageClass (SC) definition:<br> — single zone subnet ID → we need to create 1 SC per subnet<br> — security group ID we created above<br> — if using S3 as background storage: S3 bucket URL + folder → we need to create 1 SC per subnet AND per S3 folder path</li><li><strong>PersistentVolumeClaim</strong>: references a StorageClass; need to define Lustre FS size: 1200 GB, 2400 GB or n x 3600 GB</li></ul><blockquote>0.2 GB/s IO speed is provisioned per each 1000 GB (can scale up to hundreds of GB/s, if you can afford the cost)</blockquote><h4>Monitoring</h4><ul><li>there are a lot of Cloudwatch metrics for FSx, can be fetched with <a href="https://github.com/prometheus/cloudwatch_exporter">Cloudwatch Exporter</a></li><li>there are no (Prometheus) metrics for the app components itself</li></ul><h4>Billing, costs</h4><ul><li>$0.154/GB/month <em>(eu-west-1) → 1200 GB is the minimum storage</em>, for each FS created → <strong>$185/month/each 1200 GB provisioned</strong> (~ $6/day)</li><li>it is proportionally billed; you can delete, re-create the FS anytime, takes ~5 minutes for 1200 GB.</li></ul><blockquote>If it’s attached to a backing S3 bucket, you will keep the data even if you delete, re-create the filesystem at any time.</blockquote><ul><li>additional cost: cross-AZ traffic, if you attach the FS from a node of a different AZ</li></ul><blockquote>The managed Lustre filesystem lives in a single AZ.</blockquote><ul><li>additional cost: S3 storage cost if you want to persist the data written to this file system</li></ul><h4>How it actually works</h4><ul><li>you create a <strong>StorageClass</strong> which <strong>defines</strong> in <strong>which AZ</strong> do you spawn the Lustre filesystem, and whether it has an <strong>S3 bucket and folder</strong> behind it</li><li>you create the <strong>PersistentVolumeClaim</strong> (PVC) which defines the<strong> filesystem size</strong> and some mount options <em>(noatime, ...)</em></li></ul><p><strong>1 PVC = 1 Lustre FSx instance.</strong></p><ul><li>you can create multiple PVCs (= filesystem instances) to the same StorageClass (and same S3 bucket folder)</li></ul><blockquote>The sync to S3 is not real time, you specifically need to trigger it in a controlled fashion.</blockquote><ul><li>you can attach the same PVC to multiple containers in RW-RW mode</li><li>once the PVC is created, the Controller notices the event and starts provisioning the file system (it takes ~5m for a 1200 GB size)</li><li>PVC will be in Pending state; when the underlying FS is created, a new PersistentVolume object appears and is bound to the PVC, thus ready to be mounted</li><li><em>a single ENI is created in the VPC in the specified subnet + a DNS alias for it</em><br> — this is the customer-facing endpoint of the AWS-managed Lustre filesystem<br> — this ENI is attached to the security group specified in the StorageClass definition</li><li>you create a pod which wants to mount this PVC</li><li>pod gets scheduled to a worker node</li><li>worker node’s FSx <strong>DaemonSet</strong> realizes there’s a Lustre mount request; it will talk to the AWS API to <strong>mount this FS to the underlying EC2 host</strong><br> — once the mount is done, the pod will be started and gets the FS mounted through the EC2 host</li><li>pod sees the remote filesystem as a normal directory, can create files inside, they will be automatically saved to the Lustre FS</li><li><strong>there is NO automated S3 sync!</strong></li><li>whatever you write to Lustre, it will never go to S3, until you manually press the Sync button in the AWS console, in the AWS CLI, or by triggering a special CLI tool which you add to your Docker image<br> — when asking to sync, a <em>Data Repository Task</em> object will be created and it will eventually do an “rsync” between Lustre and S3, only touching the actually modified files<br> — <strong>if your pod terminates early and cannot finish/do an S3 sync, you don’t lose data</strong>, it will persist in Lustre, as long as you don’t delete the filesystem itself (the PVC); you can do S3 sync anytime</li></ul><p>Technically you can see the files already existing in S3 in the Lustre FS as normal files and directories, however it did not work for me; I was able to write new files and sync to S3, but anything I put in S3 never appeared on the Lustre side.</p><blockquote>Could have been caching issue inside the EC2 DaemonSet or in the EC2 host itself, in the Controller or anywhere else.</blockquote><blockquote>I was approaching the end of the week and this information is already good enough to make a decision whether we want to spend more time with it or not. If you have RW-RW anyway and you are only going to write to S3 once, and then process the dataset many-many times, you might just want to use S3 as a backup/checkpoint storage, to avoid reprocessing large datasets from scratch.</blockquote><h3>Conclusion</h3><p>Although I just spent a couple of hours testing/validating this storage driver plugin, it seems to be good enough to start playing with it and see some actual workload running on it.<strong> It is definitely a big improvement over their EFS plugin, where you had to pre-provision the NFS filesystem, fetch its ID and then hardcode it inside your StorageClass definition.</strong></p><p>I’m not particularly happy about the lack of internal metrics in the components and privilege handling is a bit underdeveloped yet: triggering an S3 export requires users to run a container in privileged mode with CAP_SYS_ADMIN capability.</p><p>The installation method has the stereotype AWS project traits: curl github.com | kubectl apply - or| bash. Maybe it’s time I contribute a Helm chart to the community.</p><p>Managing your own distributed, concurrent-write capable, scalable filesystem is a big enough pain to <strong>appreciate the work behind this project</strong> and just pay up to a few hundred dollars a month to AWS and use their service, spawn a new filesystem whenever you need it, delete it when your task is finished, etc. — all of it from inside Kubernetes!</p><p>Looking forward to how it will develop over the next few months!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c66d90d2b06a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/infrastructure-adventures/hands-on-lustre-fsx-plugin-for-eks-shared-storage-for-machine-learning-workloads-on-k8s-c66d90d2b06a">Hands-on: Lustre FSx plugin for EKS, shared storage for machine learning workloads on K8s</a> was originally published in <a href="https://medium.com/infrastructure-adventures">Infrastructure adventures</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Centralized SSH login to Google Compute Engine instances]]></title>
            <link>https://medium.com/infrastructure-adventures/centralized-ssh-login-to-google-compute-engine-instances-d00f8654f379?source=rss-1d8963de226e------2</link>
            <guid isPermaLink="false">https://medium.com/p/d00f8654f379</guid>
            <category><![CDATA[tips]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[notes-to-self]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Daniel Megyesi]]></dc:creator>
            <pubDate>Mon, 20 Aug 2018 00:07:39 GMT</pubDate>
            <atom:updated>2018-08-20T00:12:09.619Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<blockquote>This article is part of my “short notes for myself, maybe useful for others too” <a href="https://medium.com/infrastructure-adventures/notes-to-self/home">series</a>.</blockquote><blockquote>Related documentation:</blockquote><p><a href="https://cloud.google.com/compute/docs/instances/managing-instance-access">Managing Instance Access Using OS Login | Compute Engine Documentation | Google Cloud</a></p><h3>Requirements</h3><h4>Enable global OS login feature</h4><p>You can do it either globally on the project, then it’s valid for every instance:</p><pre># gcloud CLI<br><strong>gcloud compute project-info add-metadata</strong> --metadata enable-oslogin=TRUE<br></pre><pre># Terraform equivalent<br>resource &quot;google_compute_project_metadata_item&quot; &quot;oslogin&quot; {<br>  project = &quot;&lt;project ID&gt;&quot;<br><strong>  key     = &quot;enable-oslogin&quot;<br>  value   = &quot;TRUE&quot;</strong><br>}</pre><p>You can also disable 1-by-1 for specific VMs, and keep it on by default for everything else.</p><p>Or just enable it machine by machine:</p><pre>gcloud compute instances add-metadata [INSTANCE_NAME] --metadata enable-oslogin=TRUE</pre><h4>Upload your SSH key to the project</h4><pre>gcloud compute os-login ssh-keys add --key-file .ssh/id_rsa.pub</pre><h4>Grant permissions to certain users or groups</h4><p>There are 2 roles available:</p><ul><li>roles/compute.osLogin for non-root access</li><li>roles/compute.osAdminLogin for users who get sudo</li></ul><pre># gcloud CLI<strong><br>gcloud projects add-iam-policy-binding</strong> $PROJECT \<br>    --member=<strong>$ACCOUNT_EMAIL</strong> \<br>    --role=<strong>roles/</strong><strong>compute.osAdminLogin</strong></pre><pre><br># Terraform equivalent<br><strong>resource &quot;google_project_iam_member&quot;</strong> &quot;role-binding&quot; {<br>  project = &quot;&lt;project ID&gt;&quot;<br><strong>  role    = &quot;roles/</strong><strong>compute.osAdminLogin&quot;<br>  member  = &quot;&lt;ACCOUNT_EMAIL&gt;&quot;</strong><br>}</pre><h3>Usage</h3><p>Your user name will be always your full e-mail address, all dots and special characters, the @ sign replaced by an underscore.</p><p>Example:</p><p>daniel@megye.si becomes -&gt;daniel_megye_si</p><h4>With direct connection to the internal network</h4><p>If you have</p><ul><li>Cloud VPN</li><li>Cloud Interconnect</li><li>or a jump server</li></ul><p>to reach the internal network directly:</p><pre>ssh daniel_megye_si@&lt;VM IP&gt;<br>ssh daniel_megye_si@&lt;VM FQDN&gt;</pre><pre># gcloud CLI<br>gcloud compute ssh <strong>--internal-ip</strong> &lt;INSTANCE_NAME&gt;</pre><h4>Every other case</h4><p><strong>You’re going to need to have public IP on each machine</strong> + firewall rule allowing SSH from Internet.</p><pre>gcloud compute ssh &lt;INSTANCE_NAME&gt;</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d00f8654f379" width="1" height="1" alt=""><hr><p><a href="https://medium.com/infrastructure-adventures/centralized-ssh-login-to-google-compute-engine-instances-d00f8654f379">Centralized SSH login to Google Compute Engine instances</a> was originally published in <a href="https://medium.com/infrastructure-adventures">Infrastructure adventures</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Working with multiple environments in gcloud CLI]]></title>
            <link>https://medium.com/infrastructure-adventures/working-with-multiple-environment-in-gcloud-cli-93b2d4e8cf1e?source=rss-1d8963de226e------2</link>
            <guid isPermaLink="false">https://medium.com/p/93b2d4e8cf1e</guid>
            <category><![CDATA[tips]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[notes-to-self]]></category>
            <dc:creator><![CDATA[Daniel Megyesi]]></dc:creator>
            <pubDate>Sun, 19 Aug 2018 23:43:10 GMT</pubDate>
            <atom:updated>2018-08-20T00:10:59.665Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<blockquote>This article is part of my “short notes for myself, maybe useful for others too” <a href="https://medium.com/infrastructure-adventures/notes-to-self/home">series</a>.</blockquote><p>…aka how to manage different environments and projects in Google Cloud and switch between them quickly, each with their own settings and defaults.</p><h3>Login and create contexts</h3><pre># First time login</pre><pre>gcloud init<br></pre><pre># Create contexts</pre><pre>gcloud config configurations create prod</pre><pre>gcloud config configurations create staging</pre><pre>gcloud config configurations create sandbox</pre><h3>Configure each environment</h3><p>Either you can do gcloud init and gcloud config … for every environment one by one:</p><h4>Version 1. The elegant way</h4><p>You need to login to your Google account for each environment and configure project name, etc.</p><pre># Select environment<br>gcloud config configurations activate <strong><em>sandbox</em></strong></pre><pre># Login<br>gcloud init</pre><pre><br># Defaults<br>gcloud config set project <em>&lt;PROJECT_NAME&gt;</em></pre><pre>gcloud config set compute/region <em>europe-west3</em></pre><pre>gcloud config set compute/zone <em>europe-west1-b</em></pre><h4>Version 2. The quick way</h4><p>Create the config file for each environment by hand; change:</p><ul><li>file name (config_sandbox)</li><li>project ID</li><li>account address</li></ul><pre><strong># file contents of: config_sandbox</strong></pre><pre>cat &gt; ~/.config/gcloud/configurations/config_sandbox &lt;&lt; EOF</pre><pre>[core]</pre><pre>account = daniel@megye.si</pre><pre>project = <em>&lt;PROJECT NAME&gt;</em></pre><pre>[compute]</pre><pre>zone = <em>europe-west1-b</em></pre><pre>region = <em>europe-west1</em></pre><pre>EOF</pre><h4>Validate</h4><pre>gcloud config configurations list</pre><h3>Switch between environments</h3><pre>gcloud config configurations activate sandbox</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=93b2d4e8cf1e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/infrastructure-adventures/working-with-multiple-environment-in-gcloud-cli-93b2d4e8cf1e">Working with multiple environments in gcloud CLI</a> was originally published in <a href="https://medium.com/infrastructure-adventures">Infrastructure adventures</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Docker Registry login with Google Cloud service accounts]]></title>
            <link>https://medium.com/infrastructure-adventures/docker-registry-login-with-google-cloud-service-accounts-6bd528bed134?source=rss-1d8963de226e------2</link>
            <guid isPermaLink="false">https://medium.com/p/6bd528bed134</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[tips]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[notes-to-self]]></category>
            <dc:creator><![CDATA[Daniel Megyesi]]></dc:creator>
            <pubDate>Sun, 19 Aug 2018 23:30:56 GMT</pubDate>
            <atom:updated>2018-08-20T00:11:51.615Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<blockquote>This article is part of my “short notes for myself, maybe useful for others too” <a href="https://medium.com/infrastructure-adventures/notes-to-self/home">series</a>.</blockquote><p>…aka how to push Docker images to Google Container Registry without having the gcloud CLI tool installed, only using Docker commands.</p><h3>Requirements</h3><ul><li>Container Registry API enabled (containerregistry.googleapis.com)</li><li>Service account created which we will use to push Docker images</li><li>Correct permissions for the service account</li><li>Service account JSON key downloaded</li></ul><h3>Create a service account</h3><pre># gcloud CLI<br><strong>gcloud iam service-accounts create</strong> <strong>$ACCOUNT_NAME</strong> --display-name=&quot;&lt;what this account does&gt;&quot;</pre><pre><br># Terraform equivalent<br><strong>resource &quot;google_service_account&quot;</strong> &quot;service_account&quot; {<br>  <strong>account_id   = &quot;&lt;ACCOUNT NAME&gt;&quot;</strong><br>  display_name = &quot;&lt;what this account does&gt;&quot;<br>  project      = &quot;&lt;project ID&gt;&quot;<br>}</pre><h3>Grant permissions to the service account</h3><blockquote>Related documentation:</blockquote><p><a href="https://cloud.google.com/container-registry/docs/access-control">Configuring Access Control | Container Registry | Google Cloud</a></p><p>You have two options:</p><ul><li>IAM role: roles/storage.admin on the project, but then this account can edit/delete any buckets in the entire project</li></ul><pre># gcloud CLI<strong><br>gcloud projects add-iam-policy-binding</strong> $PROJECT \<br>    --member=serviceAccount:<strong>$ACCOUNT_EMAIL</strong> \<br>    --role=<strong>roles/storage.admin</strong></pre><pre><br># Terraform equivalent<br><strong>resource &quot;google_project_iam_member&quot;</strong> &quot;role-binding&quot; {<br>  project = &quot;&lt;project ID&gt;&quot;<br><strong>  role    = &quot;roles/storage.admin&quot;<br>  member  = &quot;serviceAccount:&lt;ACCOUNT_EMAIL&gt;&quot;</strong><br>}</pre><ul><li>the same role on the specific bucket: [REGION].artifacts.[PROJECT-ID].appspot.com, but the bucket has to already exist at the time of granting the permissions (= you must push a Docker image first)</li></ul><pre># gcloud CLI<br><strong>gsutil iam ch serviceAccount:&lt;ACCOUNT_EMAIL&gt;:admin gs://eu.artifacts.&lt;project ID&gt;.appspot.com</strong></pre><pre><br># Terraform: not really automate-able, due to the push requirement</pre><h3>Download service account JSON key</h3><pre># gcloud CLI<strong><br>gcloud iam service-accounts keys create</strong> &lt;output filename&gt; <strong>--</strong><a href="/service/mailto:iam-account=sa-dehydrated-git@personal-201021.iam.gserviceaccount.com"><strong>iam-account=&lt;</strong></a><strong>ACCOUNT_EMAIL&gt;</strong></pre><pre><br># Terraform equivalent<br># please note this will store the JSON key in the TF state!</pre><pre><strong>resource &quot;google_service_account_key&quot;</strong> &quot;key&quot; {<br>  <strong>service_account_id = &quot;&lt;ACCOUNT EMAIL&gt;&quot;</strong><br>}</pre><pre>output &quot;service_account_json_key&quot; {<br>  value = &quot;${<strong>google_service_account_key</strong>.key<strong>.private_key</strong>}&quot;<br>}</pre><h3>Docker login from CI pipeline or command line</h3><blockquote>Related documentation:</blockquote><p><a href="https://cloud.google.com/container-registry/docs/advanced-authentication#json_key_file">Authentication Methods | Container Registry | Google Cloud</a></p><p>This works both in a CI pipeline or in just the CLI by hand. You need to have the service account JSON key.</p><pre># If using US-based registry</pre><pre>docker login -u _json_key --password-stdin https://gcr.io &lt; account.json</pre><pre># If using EU-based registry</pre><pre>docker login -u _json_key --password-stdin https://eu.gcr.io &lt; account.json</pre><p>Now you’re ready to push your images to eu.gcr.io/&lt;project name&gt;/&lt;image name&gt; format.</p><h4>Gitlab CI setup</h4><ul><li>set up CI secret variable with the account.json file base64 encoded (cat account.json | base64)</li><li>echo $SERVICE_ACCOUNT_KEY | base64 --decode &gt; account.json</li><li>and the rest as above</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6bd528bed134" width="1" height="1" alt=""><hr><p><a href="https://medium.com/infrastructure-adventures/docker-registry-login-with-google-cloud-service-accounts-6bd528bed134">Docker Registry login with Google Cloud service accounts</a> was originally published in <a href="https://medium.com/infrastructure-adventures">Infrastructure adventures</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Self-renewing Let's Encrypt wildcard certificates in K8s for internal domains: Part II - certs]]></title>
            <link>https://medium.com/infrastructure-adventures/self-renewing-lets-encrypt-wildcard-certificates-in-k8s-for-internal-domains-part-ii-certs-66c613a2279f?source=rss-1d8963de226e------2</link>
            <guid isPermaLink="false">https://medium.com/p/66c613a2279f</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[ssl]]></category>
            <category><![CDATA[lets-encrypt]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Daniel Megyesi]]></dc:creator>
            <pubDate>Sun, 19 Aug 2018 17:23:48 GMT</pubDate>
            <atom:updated>2018-08-19T17:38:23.146Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<h3>Self-renewing Let’s Encrypt wildcard certificates in Kubernetes for internal domains: Part II — certificate provisioning</h3><p>Let’s pick up the thread where we left off; here’s what we have done <a href="https://medium.com/infrastructure-adventures/self-renewing-lets-encrypt-wildcard-certificates-in-kubernetes-for-internal-domains-part-i-dns-ea788eb3dbf9">in the previous article</a>:</p><ul><li>we <strong>built a</strong> <strong>PowerDNS server Docker image</strong></li><li>(bootstrapped a Kubernetes cluster)</li><li>we have <strong>deployed</strong> this DNS server with its HTTP API enabled <strong>in K8s, published to the Internet</strong></li><li><strong>configured</strong> our <strong>DNS provider</strong> to use our fake DNS sub-zone for Let’s Encrypt validation</li></ul><p>So today we only have a few steps left to finish the self-renewing SSL certificate infrastructure:</p><ul><li>a minor touch: <strong>set up the CI/CD pipeline to redeploy</strong> the new DNS server image <strong>to Kubernetes</strong> <strong>after each commit</strong>, automatically</li><li><strong>create a Docker image of the Let’s Encrypt client</strong></li><li><strong>automatically provision and renew certificates</strong>, and save the files to git</li><li>each time there’s a new cert, <strong>automatically deploy them</strong> as a Secret object <strong>to Kubernetes</strong>, ready to be consumed by the user applications</li></ul><p>This is now a fairly easy part, so let’s get to it!</p><h3>CI pipeline to redeploy the DNS server for each new build</h3><p>In the previous article, we have created the PowerDNS Docker image and configured the Google Cloud builder to create this image and push to the registry after each commit. What’s still missing, is to actually deploy it to the Kubernetes cluster, with the latest version always.</p><p>In order to achieve this, we only need to add a few extra lines to the cloudbuild.yaml file in the git repository:</p><pre># ... here we had the Docker push commands to the Registry</pre><pre><strong>- name: &#39;gcr.io/cloud-builders/kubectl&#39;<br>  args:<br>  - &#39;--namespace=infra&#39;<br>  - &#39;set&#39;<br>  - &#39;image&#39;<br>  - &#39;deployment/powerdns-letsencrypt&#39;<br>  - &#39;pdns=gcr.io/$PROJECT_ID/$REPO_NAME:$COMMIT_SHA&#39;<br>  env:<br>  - &#39;CLOUDSDK_COMPUTE_ZONE=europe-west3&#39;<br>  - &#39;CLOUDSDK_CONTAINER_CLUSTER=cluster-1&#39;</strong><br></pre><pre># last line stays untouched:<br>images: [&#39;gcr.io/$PROJECT_ID/$REPO_NAME:$COMMIT_SHA&#39;, &#39;gcr.io/$PROJECT_ID/$REPO_NAME:$BRANCH_NAME&#39;]</pre><p>So what exactly are we doing here?</p><ul><li>we use the official kubectl image from Google</li><li>and just <strong>simply trigger a rolling update to the Deployment object</strong> of the DNS server: kubectl set image deployment/$DEPLOYMENT_NAME $CONTAINER=$NEW_DOCKER_IMAGE</li></ul><p><strong>The rolling update will make sure that if the new build fails, it automatically keeps the previous, working copy online</strong>. Nothing to be configured by us to achieve this.</p><p>You can notice that we need to supply some environment variables to kubectl to find the GKE cluster’s address (compute region and cluster name).</p><p>If you haven’t done it yet, you are going to <strong>have to give permissions to the cloud builder’s built-in service account</strong> (<em>&lt;your project’s numeric ID&gt;@cloudbuild.gserviceaccount.com</em>) <strong>to connect to the Google Kubernetes Engine service</strong>, by adding the Kubernetes Engine Developer permissions. You can do this either in the Google Cloud Dashboard in the IAM menu or use the command line tool:</p><pre>PROJECT=&quot;$(gcloud projects describe \<br>    $(gcloud config get-value core/project -q) --format=&#39;get(projectNumber)&#39;)&quot;</pre><pre><strong>gcloud projects add-iam-policy-binding</strong> $PROJECT \<br>    --member=serviceAccount:<strong>$</strong><a href="/service/mailto:PROJECT@cloudbuild.gserviceaccount.com"><strong>PROJECT@cloudbuild.gserviceaccount.com</strong></a> \<br>    --role=<strong>roles/container.developer</strong></pre><p>Let’s see it rolling! The Cloud Builder output is the following after I push my new commit:</p><pre>[...]</pre><pre>Starting Step #4<br>Step #4: Already have image (with digest): gcr.io/cloud-builders/kubectl<br>Step #4: Running: gcloud container clusters get-credentials --project=&quot;personal-201021&quot; --zone=&quot;europe-west3&quot; &quot;cluster-1&quot;<br>Step #4: Fetching cluster endpoint and auth data.<br>Step #4: kubeconfig entry generated for cluster-1.<br>Step #4: <strong>Running: kubectl</strong> --namespace=infra <strong>set image deployment</strong>/powerdns-letsencrypt pdns=gcr.io/personal-201021/powerdns-letsencrypt:87d150fe221df27b0148b63bb26b4b55f9e69e3a<strong><br>Step #4: deployment &quot;powerdns-letsencrypt&quot; image updated</strong><br>Finished Step #4</pre><p>We’re all set! Now each time you push a change to your PowerDNS repository, it will create a new build and deploy it to Kubernetes.</p><h3>Creating the Let’s Encrypt client</h3><p>Here comes the most important part, the actual code which generates and creates the SSL certificates.</p><p>We create a Docker image which the following things:</p><ul><li><strong>installs the </strong><a href="https://dehydrated.io/"><strong>Dehydrated</strong></a><strong> Let’s Encrypt client</strong></li><li><strong>adds a </strong><a href="https://github.com/silkeh/pdns_api.sh"><strong>PowerDNS API hook</strong></a> to the previous client</li><li><strong>clones our new git repository</strong> which stores the provisioned certificates</li><li>and finally, starts the client to actually <strong>generate the certs</strong></li></ul><p>It’s a fairly trivial Dockerfile, you can find the full source code at <a href="https://github.com/dmegyesi/letsencrypt-wildcard-dnsapi/tree/master/dehydrated-client">my GitHub repo</a>.</p><h4>Creating and accessing the certificate storage repository</h4><p>If you noticed, we introduced a new component: an additional git repository to store the certificate files. Our Docker container will have to be able to:</p><ul><li>clone the git repo (<em>in the name of a service account</em>)</li><li>and to do this, <strong>pass the Google Cloud credentials to git</strong></li></ul><p>The easiest solution for this is just to <strong>use the official Google Cloud SDK image</strong>, so here’s how my Dockerfile starts:</p><pre>FROM google/cloud-sdk:alpine</pre><p>This will by default include all the gcloud commands, it can handle service account based authentication to our GCP git repositories, etc., no need to write any of this magic by ourselves.</p><p>How do we clone a git repo in Google Cloud?</p><pre># <strong>Set up git defaults</strong>, so it doesn&#39;t complain/fail<br>RUN git config --global push.default simple &amp;&amp; \<br> git config --global user.name dehydrated &amp;&amp; \<br> git config --global user.email <a href="/service/mailto:dehydrated@megye.si">dehydrated@megye.si</a></pre><pre># <strong>Add the GCP service account</strong><br>ADD sa-dehydrated-git-key.json /root/</pre><pre># Configure `gcloud` to use this account and clone the repo<br>RUN <strong>gcloud auth activate-service-account</strong> --key-file /root/sa-dehydrated-git-key.json &amp;&amp; \<br>    gcloud config set project personal-201021 &amp;&amp; \<br>    <strong>gcloud source repos clone</strong> dehydrated-data /etc/dehydrated</pre><p>We’re going to need to provision 2 things here. The new git repo:</p><pre>gcloud source repos create powerdns-letsencrypt</pre><p>And the service account which we include in the Docker image:</p><h4>Granting permissions to the Docker image to push to git</h4><ol><li>Create the account</li></ol><pre><strong>gcloud iam service-accounts create</strong> sa-dehydrated-git --display-name=&quot;Git pusher account for Dehydrated client&quot;</pre><p>2. Give it write permissions to the repo</p><p><strong>Create an IAM policy binding file</strong> (<em>iam-dehydrated-git-push.yaml</em>):</p><pre>bindings:<br>- members:<br>  - serviceAccount:<a href="/service/mailto:sa-dehydrated-git@personal-201021.iam.gserviceaccount.com">sa-dehydrated-git@personal-201021.iam.gserviceaccount.com</a><br>  role: <strong>roles/source.writer</strong></pre><p>And then <strong>apply this policy to the service account</strong>:</p><pre><strong>gcloud source repos set-iam-policy</strong> dehydrated-data iam-dehydrated-git-push.yaml</pre><p>Download a JSON access key to the account, so you can include it in the Docker image:</p><pre><strong>gcloud iam service-accounts keys create</strong> sa-dehydrated-git-key.json --<a href="/service/mailto:iam-account=sa-dehydrated-git@personal-201021.iam.gserviceaccount.com">iam-account=sa-dehydrated-git@personal-201021.iam.gserviceaccount.com</a></pre><h4>Writing the script to provision certificates</h4><p>This part will be just a few lines of simple Bash code.</p><p>Apart from some standard boilerplate, the logic is fairly simple:</p><ul><li>download the latest state of the certificates from git</li><li>issue new certificates</li><li>upload them to git</li></ul><p>Let’s see them in details!</p><pre># Update git repos</pre><pre># In order to make git work, we need to login to the SDK<br>gcloud auth activate-service-account --key-file /root/sa-dehydrated-git-key.json<br>gcloud config set project personal-201021</pre><pre><br># Download the latest state of the certificates<br>cd /etc/dehydrated</pre><pre>git fetch<br>git checkout $BRANCH<br>git reset --hard origin/$BRANCH<br>git pull</pre><p>Then actually issue the certificates, calling the client:</p><pre>dehydrated --register --accept-terms</pre><pre># Issue new certificates<br>dehydrated --cron --keep-going</pre><p>And save to git:</p><pre># Save Letsencrypt data to git<br># git exit codes make the script crazy, let&#39;s disable this mode now:<br>set +e</pre><pre>cd /etc/dehydrated</pre><pre>git add -A . &amp;&amp; \<br>git commit -m &quot;New certificates issued at $(date +%s)&quot; &amp;&amp; \<br>git push</pre><h4>Configuring the Dehydrated client</h4><p>If you check the previous section, you can see we just simply call the Dehydrated script to issue the certificates, there are no fancy parameters supplied.</p><p>We can do this because we already included the 2 necessary files in thedehydrated-data repo (where we actually store the certs):</p><p>1. &lt;repo root&gt;/config:</p><pre># During development always use the staging API! This will create fake certificates.<br># After you verified everything works as expected, remove the word &quot;staging&quot; and you&#39;re production ready!</pre><pre>CA=&quot;<a href="https://acme-staging-v02.api.letsencrypt.org/directory">https://acme-staging-v02.api.letsencrypt.org/directory</a>&quot;</pre><pre><strong>CHALLENGETYPE=&quot;dns-01&quot;</strong></pre><pre><strong>BASEDIR=/etc/dehydrated</strong></pre><pre><strong>HOOK=/usr/local/bin/pdns_api.sh</strong></pre><pre><br># RSA based certificates consume more CPU both on server and client side, so let&#39;s use elliptic curves, it&#39;s 2018!</pre><pre>KEY_ALGO=secp384r1</pre><pre><a href="/service/mailto:CONTACT_EMAIL=letsencrypt@megye.si">CONTACT_EMAIL=letsencrypt@megye.si</a></pre><pre><br># The actual in-Kubernetes configuration (= where to find the PowerDNS Service we created the last time)</pre><pre><strong>PDNS_HOST=powerdns-letsencrypt-api.infra</strong><br>PDNS_PORT=8081<br>PDNS_SERVER=localhost<br>PDNS_VERSION=1<br>PDNS_WAIT=30<br>#<strong>PDNS_KEY</strong>: this comes from secret env. variable in K8s</pre><p>2. &lt;repo root&gt;/domains.txt</p><pre><strong>*.</strong>intranet.megye.si <strong>&gt; intranet.megye.si</strong></pre><p>This file’s format is very important! If you check the <a href="https://github.com/lukas2511/dehydrated/blob/master/docs/domains_txt.md">user manual</a>, basically we just said:</p><ul><li>give me a wildcard certificate (*.&lt;domain&gt;)</li><li>AND save it in the following file name: &lt;domain&gt;</li></ul><p>In case you want to provision single domain names, not wildcard certificates, you would do just:</p><pre>random-subdomain.intranet.megye.si</pre><p>and it would create the certs under the same file name.</p><blockquote>When we put *. in front, we’re including this asterisk in the file name as well, and obviously this breaks the shell scripts, that’s why we use the file name alias described above (the &gt; syntax).</blockquote><p>…and we’re done! Ready to issue certificates. Let’s try it!</p><h3>Installing the client to Kubernetes</h3><p><strong>We need to create and push the client’s Docker image to the registry.</strong> It’s the exact same steps as previously (cloudbuild.yaml), so I don’t want to waste lines on it; as usual, you can find the code in <a href="https://github.com/dmegyesi/letsencrypt-wildcard-dnsapi/tree/master/dehydrated-client">GitHub</a>.</p><p>You have now 2 options to issue/renew wildcard certificates with Let’s Encrypt:</p><ul><li>run the Docker image by hand every 2 months</li><li>put it in Crontab and forget about it forever</li></ul><p>Obviously, we’re going to do the second part, <strong>let’s go for 100% automation</strong>!</p><p>And by Crontab, I mean: Kubernetes Crontab :)</p><h4>Testing the client in Kubernetes</h4><p>Before we create our final setup, let’s see that this actually works!</p><blockquote>In Kubernetes, you can create so called “one shot jobs”, which you only need and want to execute 1 time only, and then never again. (<strong>If you create a regular pod or deployment, it will keep restarting after the container stops. We don’t want and need this.</strong>)</blockquote><p>To achieve this, we define a <strong>Job object</strong> in Kubernetes:</p><pre><strong>apiVersion: batch/v1<br>kind: Job</strong></pre><pre>[...]</pre><pre>      containers:<br>      - name: dehydrated<br>        <strong>image: gcr.io/personal-201021/dehydrated-client:master</strong><br>        imagePullPolicy: Always</pre><pre>        <strong>env:<br>        - name: PDNS_KEY<br>          valueFrom:<br>            secretKeyRef:<br>              name: powerdns-letsencrypt-apikey<br>              key: apikey</strong></pre><p>We specified the freshly baked Docker image and, if you remember from the Dehydrated configuration’s last line, <strong>we still need to supply the PowerDNS API key</strong>. We have created it in a secret in the previous article, so now let’s use it! <strong>We pass the </strong>DNS API <strong>key as an environment variable to the container.</strong></p><p>Amazing, it works! Let’s verify the <strong>git repository: you should have the issued certificates stored and saved there</strong>. The ACME client of Let’s Encrypt uses versioned certificates, so the actual cert.pem and similar files are symlinks to the latest versioned file (cert-1527951445.pem).</p><pre>$ ls -1 certs/intranet.megye.si/<br>cert-1527951445.csr<br>cert-1527951445.pem<br>cert.csr<br>cert.pem<br>chain-1527951445.pem<br>chain.pem<br>fullchain-1527951445.pem<br>fullchain.pem<br>privkey-1527951445.pem<br>privkey.pem</pre><h4>Running the client as a Cron object in Kubernetes</h4><p>Let’s put the icing on the cake and forget about issuing certificates by hand ever again in our life. <strong>Let’s convert the previous Job object to a CronJob object.</strong></p><p>To give you a simple analogy:</p><blockquote>a Job object is a one-shot run of an application — just like a Pod!</blockquote><blockquote>a CronJob object is multiple runs of the same application, with failure detection and auto-restarts — just like a Deployment for Pods!</blockquote><blockquote><strong>Deployment = a template of Pods. CronJob = a template of Jobs.</strong></blockquote><pre>apiVersion: batch/v1<br><strong>kind: CronJob</strong></pre><pre>[...]</pre><pre>spec:<br><strong>  # UTC schedule! in summer time it&#39;s +2 in localtime!<br>  # Run it once a week, each Monday<br>  schedule: &quot;30 2 * * 1&quot;</strong></pre><pre>  # If it fails, don&#39;t start multiple instances! Recreate it!<br>  <strong>concurrencyPolicy: Replace</strong></pre><pre>  # <strong>Keep the stopped Pods in the system, so we can inspect logs later!</strong><br>  <strong>successfulJobsHistoryLimit: 3<br>  failedJobsHistoryLimit: 3<br></strong></pre><pre>  # <strong>Now the template part is actually the exact same syntax as in creating a Deployment</strong><br>  jobTemplate:<br>    spec:<br>      template:<br>        metadata:<br>          labels:<br>            app: letsencrypt-wildcard<br>          name: letsencrypt-wildcard-renewal<br>        spec:</pre><pre>          terminationGracePeriodSeconds: 30</pre><pre>          <strong>restartPolicy: Never</strong></pre><pre>          <strong># This is the same as in the Job</strong><br>          containers:<br>          - name: dehydrated<br>            image: gcr.io/personal-201021/dehydrated-client:master<br>            imagePullPolicy: Always</pre><pre>            env:<br>            - name: PDNS_KEY<br>              valueFrom:<br>                secretKeyRef:<br>                  name: powerdns-letsencrypt-apikey<br>                  key: apikey</pre><p>You can find the entire file in <a href="https://github.com/dmegyesi/letsencrypt-wildcard-dnsapi/tree/master/manifests-infra/dehydrated-client">GitHub</a>.</p><p>So what did we do here? We <strong>created a scheduled task in Kubernetes</strong>, which runs 1x every week.</p><p>We actually need to run it 1x every 3 months, but if I put a new wildcard domain in my domains.txt, I want it to be live as soon as I can!</p><p><em>Let’s Encrypt will check the certificate expiration time every week when you run the client.</em> If it expires in more than 30 days, it won’t do anything. If it’s close to expiration, it will generate you a new certificate with an additional 90 days of extension period.</p><p>You can notice two <em>good-to-know</em> things in the definition, it caused me quite a lot of headache previously:</p><ul><li><strong>the CronJob schedule is in UTC format</strong>! So make sure you adjust it correctly to your time zone.</li><li><strong>you can configure job history </strong>both for successful and failed containers. So when the CronJob was executed, it will keep the Pods and you can simply do a <strong>kubectl logs on them and see what was the outcome</strong>. <strong>It will auto-clean these pods</strong> if they reach the history limit you configured. Neat!</li></ul><h3>The cherry on the cake: install the certificates for the end-user applications</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KQS0jfkkYE79Ia-2" /><figcaption>This is actually not a cherry. Photo by <a href="https://unsplash.com/@alexloup?utm_source=medium&amp;utm_medium=referral">Alex Loup</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Now we have the wildcard certificates created and stored in git. How do we deploy it to Kubernetes?</p><p>It will be the exact same way we did with the auto-deploy of the DNS image: <strong>add a CI pipeline to the</strong> dehydrated-data <strong>repo</strong>, <strong>so</strong> <strong>when</strong> there’s a new commit (= <strong>we issued new certs</strong>), <strong>create a Kubernetes Secret object from them</strong> and send it to the cluster.</p><p>With kubectl, you can generate Secrets from files, using the following syntax: kubectl create secret generic $SECRET_NAME --from-file=$FILE_NAME_IN_SECRET=$FILE_LOCATION_ON_DISK</p><p>The now-familiar cloudbuild.yaml will have the following:</p><pre><strong>steps:<br>- name: &#39;gcr.io/cloud-builders/kubectl&#39;</strong><br>  args:<br>  - &#39;--namespace=infra&#39;<br><strong>  - &#39;create&#39;<br>  - &#39;secret&#39;<br>  - &#39;generic&#39;</strong><br>  - &#39;tls-cert-intranet-wildcard&#39;<br><strong>  - &#39;--from-file=tls.crt=certs/intranet.megye.si/fullchain.pem&#39;<br>  - &#39;--from-file=tls.key=certs/intranet.megye.si/privkey.pem&#39;</strong><br>  env:<br>  - &#39;CLOUDSDK_COMPUTE_ZONE=europe-west3&#39;<br>  - &#39;CLOUDSDK_CONTAINER_CLUSTER=cluster-1&#39;</pre><p>(Of course you need the Cloud Builder service account to have Kubernetes Engine Developer permissions. If you already configured it for the PowerDNS image, there’s nothing to do.)</p><p>…and we’re all set! Now if you have multiple domains, of course you have to set up some kind of a loop. Either a simple for loop for iterating on your domains.txt’s file names or multiple steps manually defined in the YAML file, up to your obsession with automation.</p><h3>Summary</h3><p>Phew, it’s been a long journey! Thank you for taking your time and patience to go through these articles. By following these steps,<strong> you have now:</strong></p><ul><li><strong>unlimited amount of wildcard certificates</strong></li><li><strong>all of them</strong> valid, “<strong>real</strong>” certificates</li><li><strong>privately hosted</strong>, not exposing any company structure to the Internet</li><li><strong>automatically renewing</strong> each 60 days, <strong>without human intervention</strong></li><li><strong>automatically deployed</strong> to Kubernetes, ready <strong>to</strong> be used by <strong>end-user applications</strong> to secure their communications</li><li>and as a bonus, <strong>all</strong> your <strong>Docker images automatically built and deployed by a CI/CD pipeline</strong> in Google Cloud</li></ul><p>Hope you enjoyed this tutorial! Let me know in the comments below if something is not clear enough or you have a better approach for some parts!</p><p>You can find the source code here: <a href="https://github.com/dmegyesi/letsencrypt-wildcard-dnsapi">https://github.com/dmegyesi/letsencrypt-wildcard-dnsapi</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=66c613a2279f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/infrastructure-adventures/self-renewing-lets-encrypt-wildcard-certificates-in-k8s-for-internal-domains-part-ii-certs-66c613a2279f">Self-renewing Let&#39;s Encrypt wildcard certificates in K8s for internal domains: Part II - certs</a> was originally published in <a href="https://medium.com/infrastructure-adventures">Infrastructure adventures</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>