<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alex Moss on Medium]]></title>
        <description><![CDATA[Stories by Alex Moss on Medium]]></description>
        <link>https://medium.com/@alex-moss?source=rss-75d735bef60b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*6D5ldeIzCwxflaPfodh05g.jpeg</url>
            <title>Stories by Alex Moss on Medium</title>
            <link>https://medium.com/@alex-moss?source=rss-75d735bef60b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 23:14:56 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@alex-moss/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[GKE Compute Cost Comparisons in Small Clusters]]></title>
            <link>https://alex-moss.medium.com/gke-compute-cost-comparisons-in-small-clusters-652cf7cf34b3?source=rss-75d735bef60b------2</link>
            <guid isPermaLink="false">https://medium.com/p/652cf7cf34b3</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[gke]]></category>
            <category><![CDATA[gke-autopilot]]></category>
            <category><![CDATA[cost-optimization]]></category>
            <dc:creator><![CDATA[Alex Moss]]></dc:creator>
            <pubDate>Sun, 21 May 2023 21:11:13 GMT</pubDate>
            <atom:updated>2023-05-21T21:11:13.752Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-XzMu5UM6TErKQFx" /><figcaption>Photo by <a href="https://unsplash.com/@micheile?utm_source=medium&amp;utm_medium=referral">micheile henderson</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I’ve been running a short-term experiment across the GKE Compute options within my “Home Lab” <em>(read: GCP-hosted Google Kubernetes Engine)</em> with a view to optimise for a balance between ease of operation and cost efficiency. If that sounds useful to you, then by all means read on!</p><p>Let’s dive right in with the Billing data. The chart below shows the view of costs over the course of my two-month experiment for this small GKE Cluster. I’ll be stepping across this graph to detail each experiment I ran and what led to these outcomes, as well as some of the trade-offs along the way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-ZWy1KryVDAtm0dw" /><figcaption>GCP Spend — Two Months</figcaption></figure><p>Note that the collection of workloads deployed did not fluctuate much over the course of the experiment — we’re talking about 13 or so Deployments with maybe the odd replica dropped here and there, but nothing significant.</p><h3>The As-Is</h3><p>Let’s start by describing the steady state. Prior to the experiments this cluster ticked along happily on 3 x e2-medium Spot Instance machines in a Zonal cluster. The initial part of the graph shows this from a cost perspective:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jgsbhhlSVkShrPrN" /><figcaption>GCP Spend — 3x Spot Nodes, Normal Running</figcaption></figure><p>The remaining costs are standing charges for things like Secret Manager, Network Load Balancing and PD Storage — these do not fluctuate very much throughout the course of the experiment — pennies here and there, as the load on these apps is consistent.</p><h3>Squeeeeeeeeeze Me Seymour</h3><p>My first act was to try to squeeze this down to two nodes — which I managed, ish. I was aggressive with setting of resource requests &amp; limits for my Pods whilst still ensuring everything was running, yielding the following saving:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jAGLFr0vLjfFcoB-" /><figcaption>GCP Spend — 2x Spot Nodes and tight tuning of Pod Resources</figcaption></figure><blockquote><em>The nodes ended up as a custom machine type still on Spot E2, as it was easier to squeeze the CPU than the memory (these apps can tolerate being throttled as they’re very low usage, but of course have minimum levels of memory that they need to function).</em></blockquote><p>This worked for a little while, and as you can see was the lowest cost solution I achieved. However, crucially, the memory pressure created by the very tight tuning on some workloads did ultimately lead to scheduling issues — especially for the chunkier workloads such as the databases behind Plausible, which I <a href="https://alexos.dev/2022/03/26/hosting-plausible-analytics-on-kubernetes/">self-host</a>.</p><p>This instability is what led me to try one of GKE’s funkier features …</p><h3>Nap Time Boys n Girls</h3><p><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-provisioning">Node Auto-provisioning</a> can be enabled per node-pool, and various options exist to set the behaviour you’d like. With appropriate tuning of behaviour this ended up around 25% more expensive than my extremely-squeezed setup, but around 33% cheaper than the three-node rig I started with:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OUS_eQHuhqQzYE85" /><figcaption>GCP Spend — Node Auto-Provisioning + VPA</figcaption></figure><p>There’s a couple of points to elaborate on here:</p><ol><li>Enabling this via Terraform is straight-forward. The block below has most of what you’d need. You have the option of doing this at the cluster level or creating a NAP-enabled node pool alongside your existing, and using taints &amp; tolerations to progressively move your workloads over to the new setup, based on risk. This is handy!</li></ol><pre>cluster_autoscaling {<br><br>  enabled = true<br><br>  resource_limits {<br>    resource_type = &quot;cpu&quot;<br>    minimum = &quot;2&quot;<br>    maximum = &quot;10&quot;<br>  }<br><br>  resource_limits {<br>    resource_type = &quot;memory&quot;<br>    minimum = &quot;10&quot;<br>    maximum = &quot;20&quot;<br>  }<br><br>  autoscaling_profile = &quot;OPTIMIZE_UTILIZATION&quot;      # this makes turn-down more aggressive<br>  <br>auto_provisioning_defaults {<br>    service_account = google_service_account.cluster.email<br>    oauth_scopes = [<br>      &quot;https://www.googleapis.com/auth/compute&quot;,<br>      &quot;https://www.googleapis.com/auth/devstorage.read_only&quot;,<br>      &quot;https://www.googleapis.com/auth/logging.write&quot;,<br>      &quot;https://www.googleapis.com/auth/monitoring.write&quot;,<br>      &quot;https://www.googleapis.com/auth/monitoring&quot;,<br>      &quot;https://www.googleapis.com/auth/trace.append&quot;,<br>      &quot;https://www.googleapis.com/auth/servicecontrol&quot;,<br>      &quot;https://www.googleapis.com/auth/service.management.readonly&quot;,<br>    ]<br>    disk_size = 30<br>  }<br>  <br>}</pre><p>2. A <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/verticalpodautoscaler">Vertical Pod Autoscaling</a>. Rather than continuing with my constrained Pod sizing, I enabled VPA for all workloads in “Auto” mode, which allows the pods to be restarted to set recommended sizes based on observed behaviour (none of these are on the JVM). I actually didn’t monitor this as closely as I thought I would beforehand <em>(perhaps a future blog post on dashboarding this to come!)</em>, just trusting the system to do its thing. The graph above is the result of that.</p><ul><li>Rather than create VerticalPodAutoscaler resources for each workload independently, I used <a href="https://kyverno.io/">Kyverno</a> to generate these. That’s probably also a blog post in its own right, but here’s the snippet of policy to generate these, in case you find it useful:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/30406e2b729a28935f97c9157dae0ffa/href">https://medium.com/media/30406e2b729a28935f97c9157dae0ffa/href</a></iframe><p>3. Some <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-provisioning#support_for_spot_vms">minor trickery</a> is required to ensure that NAP nodes are scheduled as <strong>Spot Instances</strong>, for the saving. This is because you’re no longer in control of that from the cluster perspective, instead having to specify on the workload itself that it can tolerate being on Spot.</p><ul><li>I again used Kyverno for this, see this snippet <em>(note that I also had to allow it target resources in </em><em>kube-system for this to remove all non-Spot nodes - Kyverno does not do this by default)</em>:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/19b69cf3bd298880dd1c0882909bce6f/href">https://medium.com/media/19b69cf3bd298880dd1c0882909bce6f/href</a></iframe><ul><li>Fair warning — this policy wasn’t as foolproof as I’d like. If you look closely at the cost chart above, they rise slightly towards the end coinciding with the appearance of some dark red blocks — this is non-Spot Compute nodes appearing. I did not investigate this closely, but something to be aware of and/or alert for, if you use this extensively.</li></ul><p>4. Finally, ensure that the <a href="https://github.com/kubernetes/autoscaler/issues/3183">safe-to-evict annotation</a> — which I already had set — is present, to help the Cluster Autoscaler turn-down nodes containing pods with ephemeral storage. Again, Kyverno can handle this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/43897dfb64fba0b61a183fd34ab00f68/href">https://medium.com/media/43897dfb64fba0b61a183fd34ab00f68/href</a></iframe><p>This worked well enough and I would’ve been content to stop here, but whilst I was trying out these sort of options, I felt I was in a good position now to try out Autopilot — GKE’s not-so-new option for going “nodeless” entirely.</p><h3>Whose Node Is It Anyway?</h3><p><a href="https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview">Autopilot mode</a> for GKE takes away configuration of the node pools entirely. This is an either/or situation — you need to build a new cluster to switch over to this, so needs careful implementation to move your workloads over to it.</p><p>You can see also from the graph below that this was <strong>not </strong>as cheap as the NAP option — settling into ~30% more after I had gotten the setup stabilised on Spot, in comparison to the fully-implemented NAP setup <em>(you can see from the graph that it took me a little time to get things stable!)</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*brBCp2tkYoAJ92ce" /><figcaption>GCP Spend — Autopilot</figcaption></figure><p>I set this up as a parallel cluster and then moved the workloads over to it bit by bit, due to some of the fundamental differences between Standard &amp; Autopilot and not being willing to take the plunge on all at once. The terraform for the cluster is much simpler — enable_autopilot = true is the heart of it - running a plan will guide you into what’s incompatible in your remaining config. I found I could delete around 50 lines or so of config from the NAP-based one above. Result!</p><p>As well as the need to create a new cluster, some other things to be aware of with Autopilot are:</p><ol><li>Autopilot has a <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-resource-requests#min-max-requests">minimum required</a> CPU and Memory for a Pod as well as ratios between them — and it will raise the values to these as needed, depending on how you’ve configured your Pod spec. I’m sure this is why I ended up with a more expensive overall setup.</li></ol><ul><li>One way you can reduce this is to removal any zonal anti-affinity rules you have set up — if you’re willing to trade-off against the extra resilience this affords you. This allows for lower minimum values to be used.</li><li>I also chose to leave VPA enabled even though my assumption is that this doesn’t allow the pod to be resized below the minimum required for Autopilot 🤷</li></ul><p>2. Autopilot itself doesn’t allow certain things, which may affect what you can run on it. <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-security">See this document</a> for more detail — privileged pods for example may be an issue if you’re using some weird-n-funky workloads (I used to run my Gitlab Runner out of GKE, which would’ve been an issue here if I wanted to use the docker-in-docker service, rather than kaniko).</p><p>I also chose to explicitly specify Spot in my Pod spec this time — based on the observation from the NAP setup not working perfectly via the Kyverno mutation. The docs recommend the following in your .spec.template.spec:</p><pre>spec:<br>  template:<br>    spec:<br>      nodeSelector:<br>        cloud.google.com/gke-spot: &quot;true&quot;<br>      terminationGracePeriodSeconds: 25<br>      # ... rest of pod spec</pre><p>Despite the moderate increase in cost, I was pretty happy with the outcome. It feels leaner and simpler to just simply “not care” about the nodes any more (see <a href="https://cloud.google.com/kubernetes-engine/docs/resources/autopilot-standard-feature-comparison">this list</a> for example of the differences). I wouldn’t be concerned about reverting back to NAP, but for the small increase in cost, I felt the trade-off was acceptable. It also lines up with Google’s default behaviour going forward for GKE — I believe it became the default mode for new clusters from GKE v1.25 onwards.</p><blockquote><em>It does make me wonder whether Kubernetes really benefited from exposing the underlying concept of </em><em>Node in the first place - certainly in a cloud-provider world at any rate. Would it have been as popular as it is now? A good question to ask one of the original maintainers perhaps, if you were ever to find yourself in a room with them 😄</em></blockquote><h3>Conclusions</h3><p>My conclusions from this <em>(caveat: brief!)</em> bit of experimentation?</p><p>If you really, really need to minimise the damage to your bank balance, and are willing to risk the impact to service availability by getting things wrong or pushing things too hard, then right-sizing your workloads and packing them as tightly as possible gets you the biggest saving.</p><p>If you want to lower your cognitive load, then just don’t worry about the nodes at all and enable GKE in Autopilot Mode. Check the feature compatibility needs against your workloads before doing so however, especially if you need to run privileged pods (ideally, you should not!). Even though the minimum pod spec sizes are generous for small apps, this seems to <strong>nearly </strong>balance out by avoiding the cost of “other stuff” on the node, but is slightly more expensive than you could achieve if you used …</p><p>… Node Auto-provisioning. This fulfils a nice sweet-spot if you have workloads that need the full-fat GKE Standard features, but you don’t want to think about the Nodes <em>that</em> much. I combined this with Vertical Pod Autoscaling to right-size my Pods for me, combining with a fairly tight overall ceiling on the size of compute available so that I didn’t get a nasty surprise. Getting this configuration right out the gate is non-trivial, however.</p><p>And of course one final point — if you choose either of the automatic compute options NAP or Autopilot, be sure to follow the guidance on scheduling onto <strong>Spot </strong>Instances to maximise your savings — this is significant!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=652cf7cf34b3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating an up-to-date Python Distroless Container Image]]></title>
            <link>https://alex-moss.medium.com/creating-an-up-to-date-python-distroless-container-image-e3da728d7a80?source=rss-75d735bef60b------2</link>
            <guid isPermaLink="false">https://medium.com/p/e3da728d7a80</guid>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[vulnerability-management]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[distroless]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Alex Moss]]></dc:creator>
            <pubDate>Sun, 10 Jul 2022 15:41:21 GMT</pubDate>
            <atom:updated>2022-07-10T15:42:52.752Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0Z1Yzbnmro8DileB" /><figcaption>Photo by <a href="https://unsplash.com/es/@prateekkatyal?utm_source=medium&amp;utm_medium=referral">Prateek Katyal</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><em>In this blog post I’ll walk through the creation of a Python Docker image based on the Distroless container </em><a href="https://github.com/GoogleContainerTools/distroless"><em>published by Google</em></a><em>, but with an up-to-date version of Python and operating system updates — unlike their </em><a href="https://github.com/GoogleContainerTools/distroless/tree/main/experimental/python3"><em>experimental</em></a><em> version. This image still has the same security and operational benefits — such as no shell or unnecessary OS libraries to reduce the security attack surface, as well as preserving the tiny image size.</em></p><p>If you don’t want the background and just want to see how I built it, you can head straight to the source code <a href="https://github.com/alexdmoss/distroless-python">on Github</a>. If you are interested in a bit more of the detail, then I’ve written up a <a href="https://alexos.dev/2022/07/08/creating-an-up-to-date-distroless-python-image/">more thorough post on my own blog</a>.</p><h3>Demystifying Distroless</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*odGdtJL4ZyTNJkxl" /><figcaption>Photo by <a href="https://unsplash.com/@etiennegirardet?utm_source=medium&amp;utm_medium=referral">Etienne Girardet</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>A little bit like the term “serverless”, the term “distroless” is a trendy misnomer in my opinion. The Linux distribution is still in there, but what we really mean is that the image has been stripped down to just the bare minimum we can get away with — just enough to run your application. In particular, there’s no shell.</p><p>Why do we care about this? Well, it leads to a smaller attack surface. There are far fewer OS libraries and tools available to be exploited to help a baddie break into your running container, and even if they do (say, through your app itself), there’s less handy bits of software lying around in the image to take advantage of to break out onto the host or discover more dangerous intel.</p><p>Whilst distributions like Alpine Linux are excellent at helping with this too, you still have a shell to (ab)use. A former colleague of mine has written <a href="https://www.equalexperts.com/blog/tech-focus/docker-security-battle-of-the-base-image/">a great article</a> illustrating the sorts of differences you’d see as a would-be attacker on a container with and without a shell.</p><h3>The Python Predicament</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dYWw2eEpDOCGe_0_" /><figcaption>Photo by <a href="https://unsplash.com/@davidclode?utm_source=medium&amp;utm_medium=referral">David Clode</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This base image choice is made even more complex with Python. Whilst common languages like <a href="https://github.com/GoogleContainerTools/distroless/blob/main/java/README.md">Java</a> and <a href="https://github.com/GoogleContainerTools/distroless/blob/main/nodejs/README.md">Node</a> have well-established Distroless variants that are updated frequently, the <a href="https://github.com/GoogleContainerTools/distroless/tree/main/experimental/python3">Python one</a> continues to be marked as experimental and appears to change more rarely. It ships with whatever is in upstream Debian, leading to <a href="https://github.com/GoogleContainerTools/distroless/issues/1003">issues like this one</a>.</p><p>To illustrate this, here’s a vulnerability scan from <a href="https://github.com/aquasecurity/trivy">trivy</a>, showing that the CVE mentioned in that Github issue is still present 3+ months later:</p><pre>&gt; trivy image -s=HIGH,CRITICAL gcr.io/distroless/python3:latest</pre><pre>2022-06-26T13:49:19.271+0100	INFO	Detecting Debian vulnerabilities...</pre><pre>gcr.io/distroless/python3:latest (debian 11.3)</pre><pre>Total: 10 (HIGH: 7, CRITICAL: 3)</pre><p>You might think then that perhaps with Python specifically it’d be better to use Alpine? Indeed that’s something I’ve done a fair few times myself. Whilst this does help, it creates new problems. Its different choice of C library can create <strong>horrendous</strong> build times when certain (common) packages are in use (no pip wheels available, forcing a build from source) — grpcio and pandas are notable examples in my world.</p><blockquote>This isn’t the only issue you could face too — as <a href="https://pythonspeed.com/articles/alpine-docker-python/">this excellent blog post</a> elaborates on, you can encounter subtle bugs and performance issues as a result of the choice too.</blockquote><p>I have a strong suspicion that the most common choice out there therefore becomes python:*-slim-bullseye. This image works very well, but does suffer from OS library vulnerabilities. Here’s a scan from the latest 3.9 image at the time I was writing up my findings:</p><pre>&gt; trivy image -s=HIGH,CRITICAL python:3.9-slim-bullseye</pre><pre>2022-06-26T13:59:34.109+0100	INFO	Detecting Debian vulnerabilities...</pre><pre>python:3.9-slim-bullseye (debian 11.3)</pre><pre>Total: 16 (HIGH: 13, CRITICAL: 3)</pre><p>Even if you take the view that all of these vulnerabilities are not actually as severe as they’re being classified (<a href="https://security-tracker.debian.org/">Debian’s tracker</a> is pretty good, imho), <em>simply going through the process</em> of reviewing the vulnerability, updating the image if possible, suppressing the vulnerability from your scanning tool if not a concern, and so on is all<em> extremely </em><a href="https://sre.google/sre-book/eliminating-toil/"><em>toilsome work</em></a>, especially at scale.</p><p>So, instead, let’s see how you can have your cake and eat it too (mostly), with your own distroless python image.</p><h3>Building a Better Base</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CXm0mwXMAy4p6HKx" /><figcaption>Photo by <a href="https://unsplash.com/@rssemfam?utm_source=medium&amp;utm_medium=referral">Russ Ward</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I initially began as probably most people would — taking a copy of the <a href="https://github.com/GoogleContainerTools/distroless">distroless repo itself</a> and hacking about with it. The images are built using Google’s <a href="https://bazel.build/">Bazel</a>. I’d not used this before and to be honest I found the learning curve rather steep. A short while in I decided to pivot to technology I was more familiar with — Docker itself.</p><blockquote>As both the Python image we like and the Distroless base image are both based on Debian, we can “cheat” with a <a href="https://docs.docker.com/develop/develop-images/multistage-build/">multi-layer build</a> instead, avoiding a lot of that complexity.</blockquote><p>I ended up with a Dockerfile that looks a bit like this structure:</p><pre>FROM FULLY_FEATURED_PYTHON_UPSTREAM_IMAGE as base</pre><pre>### do nothing, if you like. Or do something. Up to you #casual</pre><pre>FROM GOOGLE_DISTROLESS_IMAGE</pre><pre># copy the files we need from the python base into distroless<br>COPY --from=base /path/on/base /path/on/distroless</pre><pre>### do anything else you find useful here too. If you want to</pre><pre># run python<br>ENTRYPOINT [&quot;/usr/local/bin/python&quot;]</pre><p>The COPY --from is key - and in reality a <em>little</em> more complicated than I’ve made it look. We need to copy Python and its dependencies into distroless, as well as any useful compiled libraries that we’ll need for other Python packages. For me, this involved some trial and error — my knowledge of Python’s internals is not up to the task yet!</p><blockquote>You can fake it by copying the whole of e.g. /lib and /usr just to see if it works, but this: a) results in a 300Mb+ image, and b) is kinda defeating the point of keeping the number of libs to a minimum! Google’s Python is 50.2Mb at time of writing, and my image clocks in at 57.7Mb</blockquote><p>I also chose to use the <a href="https://github.com/GoogleContainerTools/distroless/tree/main/cc">C version</a> of Google’s distroless rather than base, as so much of Python depends on it anyway. I pinched this idea from the Python variant that Google build.</p><p>The <a href="https://github.com/alexdmoss/distroless-python/blob/main/distroless.Dockerfile">final version of my Dockerfile</a> therefore looks a bit more complicated than the above — I’ll talk through this in a bit more detail below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f3ce1083c6db15244f9b6e6f6ae6378a/href">https://medium.com/media/f3ce1083c6db15244f9b6e6f6ae6378a/href</a></iframe><p>I <strong>elaborate on the detail</strong> behind most of the lines here in <a href="https://alexos.dev/2022/07/08/creating-an-up-to-date-distroless-python-image/">my more detailed blog post</a>.</p><p>It’s less deterministic than the Bazel approach, but for my needs it has been working reliably so far and made vulnerabilities much easier to manage.</p><p>Want to use the images? The distroless image itself can be found <a href="https://hub.docker.com/r/al3xos/python-distroless">here</a>:</p><ul><li>docker pull al3xos/python-distroless:3.9-debian11 (there is also a 3.10 tag)</li></ul><p>And if you find the <a href="https://hub.docker.com/r/al3xos/python-builder">builder</a> image useful too (it is for installing your dependencies in an earlier layer):</p><ul><li>docker pull al3xos/python-builder:3.9-debian11 (again, 3.10 also available at time of writing)</li></ul><h3>Testing The Thing</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NhL3NFMoaTkFBIlq" /><figcaption>Photo by <a href="https://unsplash.com/@jeswinthomas?utm_source=medium&amp;utm_medium=referral">Jeswin Thomas</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Speaking of actually using the images — I also added some <a href="https://github.com/alexdmoss/distroless-python/blob/main/EXAMPLES.md"><strong>examples</strong></a> that serve as the smoke tests for the published images above.</p><p>They range from a simple hello-world to using common libraries I use (such as <a href="https://github.com/kubernetes-client/python">kubernetes-client</a> and <a href="https://pandas.pydata.org/">pandas</a>), as well as running more complex frameworks like <a href="https://flask.palletsprojects.com/en/2.1.x/">Flask</a> and <a href="https://fastapi.tiangolo.com/">FastAPI</a>. Again, <a href="https://alexos.dev/2022/07/08/creating-an-up-to-date-distroless-python-image/">more detail in my other post</a>.</p><h3>Summary</h3><p>So there we have it — a Distroless Python image that uses Google’s distroless as a base, but layers in an up-to-date version of Python and its dependencies that are under your control, to tailor to your needs. Whilst this is something I’ve only recently put together, I’m now using it in a few places and it seems to be working well — whilst still delivering on the tiny image size (faster container startup matters!) and in particular leaner attack surface and less toil in managing OS vulnerabilities.</p><p>I hope you find some inspiration from this post to use it yourself or create your own equivalent 😄 🐍</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e3da728d7a80" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hosting Plausible Analytics on Kubernetes]]></title>
            <link>https://alex-moss.medium.com/hosting-plausible-analytics-on-kubernetes-2cef98974142?source=rss-75d735bef60b------2</link>
            <guid isPermaLink="false">https://medium.com/p/2cef98974142</guid>
            <category><![CDATA[visitor-tracking-system]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[plausible]]></category>
            <category><![CDATA[velero]]></category>
            <dc:creator><![CDATA[Alex Moss]]></dc:creator>
            <pubDate>Tue, 21 Jun 2022 13:07:52 GMT</pubDate>
            <atom:updated>2025-06-19T12:21:30.067Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RQ3nS6eAbdw5VA7Y" /><figcaption>Photo by <a href="https://unsplash.com/@justin_morgan?utm_source=medium&amp;utm_medium=referral">Justin Morgan</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><em>In this post I’ll describe some of the steps I’ve taken to spin up a copy of </em><a href="https://plausible.io/"><em>Plausible Analytics</em></a><em> on an existing Kubernetes cluster. I’ll be using Google Kubernetes Engine, but in most regards this should work fine on any provider.</em></p><p>This is actually an article I first penned a few months ago, but held off writing on Medium until I’d given things a little time to bed in. Well, it’s been ticking away quite happily since then — and if you want a more detailed version of the steps involved then <a href="https://alexos.dev/2022/03/26/hosting-plausible-analytics-on-kubernetes/">that article can be found here</a>.</p><h3>Why Plausible?</h3><p>First up — why Plausible? And why self-host it? I’ll keep this brief I promise as it’s not the main point of the article 😇</p><p>I was drawn to it after reading a few articles on the privacy concerns of using Google Analytics — the solution I was using previously to understand who was using my website and how they were using it. Firefox out-right blocks GA for example, and many people run ad-blockers in other browsers that do the same.</p><p>So partly it’s a moral thing, partly it’s an accurate data thing. The lack of any cookies also appealed — no need for annoying permissions banners on my websites! Tangentially I also found it simpler to use for my really basic use-case. Some folks have told me that GA v4 is less invasive but I’ve made the switch now and I’m happy.</p><p>If you’d like to read more about this, then Plausible have their own articles <a href="https://plausible.io/vs-google-analytics">comparing the two</a> and more on <a href="https://plausible.io/privacy-focused-web-analytics">their philosophy</a>. There’s also <a href="https://plausible.io/plausible.io">a demo</a>.</p><p>As for why self-hosting it? Partly confidence in where the data is being held, but mostly because I wanted to learn about how it worked, to be honest 👷‍♂️. If you work for a company looking to use Plausible, I’d personally consider their <a href="https://plausible.io/#pricing">hosted option</a> first if you don’t need to keep the data on your own servers.</p><h3>Running It On Kubernetes</h3><p>Okay, back on topic now I promise! Plausible provide <a href="https://plausible.io/docs/self-hosting">guidance in their docs</a> and <a href="https://github.com/plausible/hosting">their hosting repo</a> has some examples, but as I mentioned above I found I needed to make some tweaks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DbwO97bgo5P8RHGg" /></figure><p><a href="https://github.com/alexdmoss/plausible-on-kubernetes/">My working code can be found on my Github</a>. I’m still using this tool today so that repo should be appropriately fed and watered too. The pipeline is put together using Gitlab (where its mirrored from), but can be adapted quite easily to your chosen CI/CD tool, I’m sure.</p><p>Broadly, we need to solve for the following:</p><ol><li>Get the basic components running — the main app, its Postgres database and its Clickhouse database.</li><li>Come up with a way of generating and storing the relevant secrets for it to work — there’s quite a few of them: the credentials for Plausible itself, Postgres credentials, Clickhouse credentials, email settings, and your Twitter token if you use that.</li><li>Getting the emailed reports working.</li><li>Dealing with the fact that your copy of Plausible is going to be behind some load balancers.</li><li>Data backups and recovery.</li><li>(Optional) Proxying the tracking Javascript <em>(depending on how comfortable you are doing this)</em>.</li></ol><p>I’ll give a brief overview of each step below. As I mentioned, for a more detailed version (including code samples etc) see my more detailed post on my personal blog: <a href="https://alexos.dev/2022/03/26/hosting-plausible-analytics-on-kubernetes/">https://alexos.dev/2022/03/26/hosting-plausible-analytics-on-kubernetes/</a></p><h3>The Basic Components &amp; Secret Handling</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/0*UX3FgETcppAfXVox" /></figure><p>I began with <a href="https://github.com/plausible/hosting/tree/master/kubernetes">their Kubernetes manifests</a> but adapted them using <a href="https://kustomize.io/">kustomize</a>. This is important for controlling resource usage (especially disk space for a tool like this). You can see the structure I adopted in <a href="https://github.com/alexdmoss/plausible-on-kubernetes/tree/main/k8s">this part of my repo</a>. This included dropping the mail server completely — I doubt that would work well on GCP anyway, and I chose to use Sendgrid instead (they have a free tier that’s sufficient for my needs).</p><p>Note the use of a <a href="https://github.com/alexdmoss/plausible-on-kubernetes/blob/main/k8s/base/kustomization.yaml#L10">SecretGenerator</a> in the base to ensure that a new secret is created on any change, so that the pods automatically restart to use it. The secret values themselves are held in Google Secret Manager and patched in at deploy time (other clouds have similar, or you could use Vault).</p><blockquote>I also subbed out the latest tags in the container images to use — this is rarely a good plan in Production! <a href="https://vsupalov.com/docker-latest-tag/">Here’s a good explanation why</a></blockquote><h3>Email Reports</h3><p>As mentioned above, given the constraints on GCP I opted not to muck about trying to persuade a mail server to work within GKE, and instead signed up for Sendgrid. At the scale I’m working with, I’m well within the free tier usage. I’m not going to go through setting up your Sendgrid account itself in detail as writing that down is unlikely to age well and I found it to be pretty straight-forward — the important part is to generate yourself an API Key in there, which we then inject into the Plausible config.</p><p>The following represents the particular combination of variables that I found did the trick:</p><pre>MAILER_EMAIL=THE_EMAIL_ADDRESS_YOU_CONFIGURE_IN_SENDGRID<br>SMTP_HOST_ADDR=smtp.sendgrid.net<br>SMTP_HOST_PORT=465<br>SMTP_HOST_SSL_ENABLED=true<br>SMTP_USER_NAME=apikey<br>SMTP_USER_PWD=$SENDGRID_KEY<br>SMTP_RETRIES=2</pre><p>Take care to substitute the SMTP_USER_PWD securely, as described earlier.</p><p>I found the easiest way to test this was creating an extra user in Plausible and using the forgotten password link.</p><h3>X-Forwarded-For Behind a Load Balancer</h3><p>For Plausible to work behind a reverse proxy load balancer like the Kubernetes nginx-ingress-controller, some further tweaks are needed. You’ll know this is needed if the visitor countries / unique visitor tracking is not working. I found that the following configuration for the nginx-ingress-controller did the trick:</p><pre>hsts: &quot;true&quot;<br>ssl-redirect: &quot;true&quot;<br>use-forwarded-headers: &quot;false&quot;      # not needed as not behind L7 GCLB, but YMMV<br>enable-real-ip: &quot;true&quot;<br>compute-full-forwarded-for: &quot;true&quot;</pre><p>I also needed to edit my Service of Type: LoadBalancer to have spec.externalTrafficPolicy: Local. This affects evenness of load balancing a little, but was required for this to work and I didn’t particularly mind this downside at my scale.</p><h3>Data Backups</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/430/0*ebBbV6um7xCfbEod" /></figure><p>I wanted something in place here to allow me to recover historic data in the two databases in the event of needing to recreate my cluster (or something going wrong with it). I use this cluster to experiment with Kubernetes features fairly regularly, so there’s a certain inevitability in this for me at least 🙈</p><p>The Clickhouse database rules out a cloud-hosted PaaS option like Cloud SQL. A further issue was that, as per <a href="https://github.com/plausible/analytics/discussions/1226">this github discussion</a>, an implementation choice in Plausible means the “standard” utility within Clickhouse can’t be used either ... 💩</p><p>Instead, I ran both databases inside Kubernetes, and used <a href="https://velero.io/">Velero</a> to snapshot the disks periodically. This is <strong>far from perfect </strong>but sufficient for my needs — I can live with a gap in data in the event of catastrophe. Velero is neat in that it is k8s-native but can make use of object storage to snapshot the persistent disks <em>(it’s great at backing up other k8s resources too, should you need that)</em>.</p><blockquote>Velero is taking a snapshot of block storage whilst the disks are in use. This carries risk! At the small scale I’m using Plausible at, this is not as likely to be an issue, but if you’re expecting high load then it might be — you’ll definitely want to set up alerts for failed backups, and test this frequently … <em>😬</em></blockquote><p>The setup of Velero is not part of my public github, but it is pretty easy to do to be honest — and their docs are good. You’ll need to create a storage bucket and configure the VoumeSnapshotLocation and BackupStorageLocation for that, as well as ensuring the credentials are available to the runtime(if on GKE, Workload Identity is a fantastic way to do this securely and with minimal configuration hassle).</p><p>My backup schedule specifically for Plausible looks like this:</p><pre>---<br>apiVersion: velero.io/v1<br>kind: Schedule<br>metadata:<br>  name: plausible-pv-backup<br>spec:<br>  schedule: 0 */1 * * *<br>  template:<br>    includedResources:<br>    - persistentvolumeclaims<br>    - persistentvolumes<br>    includeClusterResources: true<br>    includedNamespaces:<br>    - &#39;plausible&#39;<br>    ttl: 168h0m0s</pre><p>As you can see, I’m not very sensitive about the freshness of the data here — snapshotting hourly is fine by me.</p><p>Restores rely on you setting up the Velero client and using a combination of the velero backup get and velero restore create commands to select and then restore the data, replacing the old disks (or creating new ones on your new cluster, if applicable).</p><h3>Proxying the Request</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*a9WWfWjV2zlm1Kha" /><figcaption><a href="https://unsplash.com/photos/LfaN1gswV5c">PHOTO BY LIANHAO QU ON UNSPLASH</a></figcaption></figure><p>Now then, decision time. Whether you do this or not is a bit of a moral choice. By disguising the Plausible tracking javascript like this, you are being a bit disingenuous with your users — although keep in mind that <a href="https://plausible.io/privacy-focused-web-analytics">their code is respectful</a>. Despite their approach, some browsers / browser extensions are sensitive enough to block the Plausible tracker, assuming it is just as naughty as the Google one. This technique helps you avoid that for more accurate analytics capture, if you’re ok with that.</p><p>In my case, the majority of my sites are served via NGINX, so the <a href="https://plausible.io/docs/proxy/guides/nginx">guidance here</a> covers what I need. You can see one of my <a href="https://gitlab.com/alexos-public/alexos.dev/-/blob/main/config/default.conf?ref_type=heads#L68">examples of this customisation here</a>.</p><blockquote>I’ve also recently started using Caddy for some website hosting. <a href="https://gitlab.com/alexos-public/alexmoss-co-uk/-/blob/main/Caddyfile?ref_type=heads#L41">Here is an example</a> of how I’ve done it using that web server too.</blockquote><p>You then update your tracking code to point to /js/visits.js instead, something like this:</p><pre>&lt;script defer data-api=&quot;/api/event&quot; data-domain=&quot;yourwebsite.com&quot; src=&quot;/js/visits.js&quot;&gt;&lt;/script&gt;</pre><p>You could of course host the Javascript itself within your side code too (skipping the first location block) — although you still need to make sure the /api/event calls are proxied on to your self-hosted Plausible instance to capture the visits. You’d also not pick up upgrades to the tracking code automatically if you did this.</p><h3>Summary</h3><p>I hope you found that run-through useful — I’ve been using it like this for around 6 months now without issue, for around half a dozen sites. I’ll be keeping <a href="https://github.com/alexdmoss/plausible-on-kubernetes">my repo</a> up to date with any tweaks I make along the way, so hopefully that will continue to prove a useful resource for anyone attempting to do what I have.</p><p>Oh, and if you work for a company looking to use Plausible, do consider their <a href="https://plausible.io/#pricing">hosted option</a> if you don’t need to keep the data on your own servers. It looks pretty sensibly priced to me, and helps them to keep improving their product!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2cef98974142" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Docker Desktop Alternatives for M1 Mac]]></title>
            <link>https://alex-moss.medium.com/docker-desktop-alternatives-for-m1-mac-918a2dcda10?source=rss-75d735bef60b------2</link>
            <guid isPermaLink="false">https://medium.com/p/918a2dcda10</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[containerd]]></category>
            <category><![CDATA[m1]]></category>
            <category><![CDATA[mac]]></category>
            <category><![CDATA[apple-silicon]]></category>
            <dc:creator><![CDATA[Alex Moss]]></dc:creator>
            <pubDate>Mon, 03 Jan 2022 17:16:55 GMT</pubDate>
            <atom:updated>2022-12-17T20:53:13.866Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_YiuGMXPuVrqCO2B" /><figcaption>Photo by <a href="https://unsplash.com/@carrier_lost?utm_source=medium&amp;utm_medium=referral">Ian Taylor</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In this blog post I’m going to talk through my recent experiences as I attempted to ditch Docker Desktop — the licensing changes that come into effect at the end of January being the primary motivator.</p><blockquote><em>Without going into any detail about it, let’s just say I’m not a fan of taking something that you’ve made freely available previously and deciding that you now want to charge for it!</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*XlwM2HZZxubzfHN6_pG8dw.png" /><figcaption>Image courtesy of this blog post: <a href="https://start.jcolemorrison.com/is-docker-in-trouble/">https://start.jcolemorrison.com/is-docker-in-trouble/</a></figcaption></figure><p>In the end I tried t̶h̶r̶e̶e̶ five options for Mac — landing on one as my preference as it covered both the need to run on the newer Apple Silicon and allow mounting of volumes on the host OS, which is something I do fairly frequently (mostly to shorten the feedback loop when testing changes that run on an image intended to run in CI).</p><blockquote><strong>Disclaimer</strong>: Most of the steps detailed below were found through following other fantastic blog posts I found out there 👏. These are of course noted wherever I’ve used them, with a few tweaks of my own I’ve made on top of these excellent guides. Hopefully having these options together in one blog post is somewhat helpful in choosing between them too!</blockquote><h3>TL:DR</h3><p><strong>Updated 17/12/2022:</strong> Several months back I made the switch from <a href="https://rancherdesktop.io/">Rancher Desktop</a> to <a href="https://github.com/abiosoft/colima">colima</a> and haven’t looked back. It feels more streamlined and performant with some additional useful configuration options, as well as just less noisy. I’m now updating this post to reflect that <strong>colima is now my recommendation </strong>— although Rancher Desktop remains a perfectly viable choice (particularly if you prefer a bit of GUI action to configure things!).</p><p>The remainder of the article is as it was — charting the various options I tried, but with an elaboration on colima and its benefits below, as the most recent option I’ve switched to.</p><blockquote><strong>Sidebar: </strong>I also had to solve this problem for my Windows 10 + WSL (Ubuntu) setup. If you’d like to know more about that, I’ve covered it in a similar blog post on my own website — you can read it <a href="https://alexos.dev/2022/01/02/docker-desktop-alternatives-for-m1-mac/#docker-with-wsl">here</a>.</blockquote><h3><strong>Recommended Option — </strong>Colima</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*m01BA6TOMc2_aDAGqIR9cA.jpeg" /></figure><p>The installation is incredibly straight-forward, just brew install colima . You need the docker CLI installed too (brew install docker) if you don’t have it already also.</p><p>Following installation, you then issue colima start when you want to start the daemon, and after that completes, you should find that docker commands work as normal. The first time you do this is a little slower due to downloading the image and configuring it, but following that only takes a few seconds on my machine. You can of course use colima stop to shut it down to save resources on your machine if desired between docker sessions.</p><p>I’ve included some<strong> further tips n tricks</strong> for running it with better compatibility with third party tooling, as well as exploring some of its other options in my own blog site. This link should take you directly to the relevant bit: <a href="https://alexos.dev/2022/01/02/docker-desktop-alternatives-for-m1-mac/#recommended-option---colima">https://alexos.dev/2022/01/02/docker-desktop-alternatives-for-m1-mac/#recommended-option---colima</a></p><p>Alternatively, see their source on Github for additional configuration options as needed: <a href="https://github.com/abiosoft/colima">https://github.com/abiosoft/colima</a>.</p><h3>Option 1: Docker + Hyperkit + Minikube</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OB5EXe5FLHOeT9DwMYsADg.jpeg" /><figcaption>Source: <a href="https://pixabay.com/illustrations/blue-crystal-cube-deep-futuristic-5457731/">https://pixabay.com/illustrations/blue-crystal-cube-deep-futuristic-5457731/</a></figcaption></figure><p>I started here. This is a simple near “drop-in” replacement for Docker Desktop, but <strong>does not work on M1 Macs</strong>. I used this on my older Macbook for a little while before replacing it with Rancher Desktop. It’s fully docker compliant, if there is such a thing.</p><p>The instructions that follow are heavily based on <a href="https://itnext.io/goodbye-docker-desktop-hello-minikube-3649f2a1c469">this excellent blog post</a>, which has some additional advice, especially if you want to get more out of the local Kubernetes cluster:</p><pre># pre-req: full install of XCode needed - just the CLI isn&#39;t enough</pre><pre># this steps fails on Apple Silicon: <a href="https://github.com/moby/hyperkit/issues/310">https://github.com/moby/hyperkit/issues/310</a><br>brew install hyperkit</pre><pre>brew install docker                  # don&#39;t use --cask<br>brew install minikube</pre><pre># bring up the daemon - which errors on Apple Silicon<br>minikube start --driver=hyperkit --keep-context</pre><pre># tells docker in your *current shell* to use minikube&#39;s daemon<br>eval $(minikube docker-env) </pre><p>I have the minikube start command set up in a start-docker.sh script I can run when needed, and the minikube docker-env in my shell startup (.zshrc, in my case).</p><p>As you can see, pretty straight-forward standard brew installation stuff — plus a couple of commands to run before you try and do docker things <em>(I personally never had Docker running all the time on startup anyway, as it was such a battery drain)</em>. As it’s still just the same docker CLI, the credentials helper to connect to a private registry also works fine out-the-box.</p><p>However, volume mounts from the host did not … but thankfully the blog post I linked above has captured the solution for this. You can minikube mount to spin up a process to mount your local directory into the minikube VM:</p><pre>minikube mount your-local-directory/:/build &gt;/dev/null 2&gt;&amp;1 &amp;</pre><pre>docker run -v /build:/build --rm -it eu.gcr.io/my-private-registry-project/alex-ubuntu:latest</pre><p>In my opinion, the advantages of this option — and why I kept it as the setup on my older Macbook — are:</p><ul><li>You’re still using the docker CLI, so very good from a compatibility point of view</li><li>There’s minimal extra config/scripts needed — almost a drop-in replacement</li><li>It’s great if you want to do Kubernetes development locally and liked that feature in Docker Desktop</li></ul><p>Downsides:</p><ul><li>Doesn’t currently support Apple Silicon (or at least, using the hyperkit driver does not)</li><li>Volume mounts aren’t seamless (although pretty simple tbh)</li></ul><p>However, I also needed an option that worked with Apple Silicon. My first attempt was with Podman …</p><h3>Option 2: Podman</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sd_1XPRWiTeAoFfd-TlRxA.jpeg" /><figcaption>Source: <a href="https://ios.dz/installation-podman-centos-8/">https://ios.dz/installation-podman-centos-8/</a></figcaption></figure><p>After realising that hyperkit didn’t work on M1, this was the next option I tried. I’d heard good things. It mostly worked fine but, as mentioned earlier, for me the crucial issue was the <strong>lack of ability to mount volumes from the host</strong> OS. I use this option a lot.</p><p>That said, if that’s not important to you or they fix it subsequently, I’ve included the steps below. these were cobbled together from the Podman installation guide itself plus this great <a href="https://marcusnoble.co.uk/2021-09-01-migrating-from-docker-to-podman/">blog post</a> — although I didn’t need most of the complexity involved here (I’m guessing it has been fixed since).</p><p>From their install guide — things are nice and simple:</p><pre>brew install podman<br>podman machine init</pre><pre>podman machine start            # bring up the daemon</pre><p>Your docker equivalents should then work as intended:</p><pre># builds a Dockerfile containing a basic nginx image<br>podman build -t podman-test -f Dockerfile .</pre><pre># run it, exposing port<br>podman run -d -p 8080:80 podman-test</pre><pre># see the appropriate output from nginx<br>curl http://localhost:8080/</pre><p>Other similar docker commands I tend to use also seem present:</p><pre>podman ps<br>podman stop &lt;id&gt;<br>podman exec -i -t &lt;id&gt; /bin/bash<br># a subtlety - requires -i -t rather than allowing -it</pre><p>However, as mentioned earlier this crucially does not work:</p><pre># /build is empty 😞<br>podman run --rm -it -p 8080:80 -v $(pwd):/build podman-test</pre><p>I looked around this topic a bit and there are some suggested workarounds, such as <a href="https://github.com/containers/podman/issues/8016#issuecomment-939353204">this one</a> to mount the directory onto the podman VM first. But these look quite hasslesome <em>(caveat: I didn’t try very hard </em>😉<em>)</em></p><p>I therefore backed away at this point as I had another option to try first … Enter lima + nerdctl …</p><h3>Option 3: Lima + nerdctl</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*a9U6af_NGDuK9s5gcmbsEw.png" /><figcaption>Source: <a href="https://www.pinterest.co.uk/pin/594897432010361218/">https://www.pinterest.co.uk/pin/594897432010361218/</a></figcaption></figure><p>This option <strong>ticked all the boxes</strong> for me and I ran with it for a little while without issue, although with more setup needed than the minikube option. I’m comfortable with that though. I like this because it: a) distances me from Docker Inc. changes to licensing in the future (and a little bit on principle, not gonna lie!), and b) puts me closer to the OCI runtime of our Production Kubernetes clusters (they’re GKE, which just run containerd by default now).</p><p>I followed the great guide <a href="https://medium.com/nttlabs/containerd-and-lima-39e0b64d2a59">in this blog post</a>, which basically boils down to:</p><pre>brew install lima<br>limactl start default        # accepted the defaults to setup the VM</pre><p>Your docker equivalents then look like this (which can of course be aliased):</p><pre># Dockerfile containing a basic nginx image<br>lima nerdctl build -t lima-test -f Dockerfile .</pre><pre>lima nerdctl run -d -p 8080:80 lima-test </pre><pre>curl http://localhost:8080/</pre><p>Other similar docker commands also seem fine, just like podman:</p><pre>lima nerdctl ps<br>lima nerdctl stop &lt;id&gt;<br>lima nerdctl exec -it &lt;id&gt; /bin/bash</pre><p>Crucially, this worked too without any special config or setup needed:</p><pre>lima nerdctl run --rm -it -p 8080:80 -v $(pwd):/build --entrypoint=/bin/bash lima-test</pre><p>That said, there were a couple of other steps I needed to go through to deal with my other requirements.</p><p>Because it’s not Docker, the existing credentials helper I had setup to connect to e.g. Google Container Registry did not automatically work. Instead, these credentials need to be readily available on the intermediary lima VM, rather than the host. To solve this, I opted to jump onto the VM and install gcloud, login as I normally would, then ensure those credentials were available to the root user.</p><p>To do this, we start with limactl shell default which should get you a shell prompt on your default lima VM. We then download and unpack the GCloud SDK:</p><pre># update with your preferred gcloud version<br>wget --no-verbose -O /tmp/google-cloud-sdk.tar.gz \<br>    https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-367.0.0-linux-x86_64.tar.gz &amp;&amp; \<br>    sudo tar -C /opt --keep-old-files -xz -f /tmp/google-cloud-sdk.tar.gz &amp;&amp; \<br>    sudo chown $(whoami):$(whoami) /opt &amp;&amp; \<br>    sudo chown -R $(whoami):$(whoami) /opt/google-cloud-sdk &amp;&amp; \<br>    rm -f /tmp/google-cloud-sdk.tar.gz</pre><p>We then ensure the required binaries are in the path, and login:</p><pre>sudo ln -s /opt/google-cloud-sdk/bin/gcloud /usr/bin/gcloud<br>sudo ln -s /opt/google-cloud-sdk/bin/docker-credential-gcloud /usr/bin/docker-credential-gcloud</pre><pre>gcloud auth login                # login as normal<br>gcloud auth configure-docker     # will warn about docker path, can ignore</pre><p>Unfortunately, we’re not quite there yet — but nearly! Back on the host machine, spinning up my Ubuntu docker image was met with an error message I’ve seen a few times before on Apple Silicon: standard_init_linux.go:228: exec user process caused: exec format error. We need to do a bit of work to give QEMU (the hypervisor behind the scenes) the option to execute non-native images.</p><p>Thankfully the <a href="https://github.com/containerd/nerdctl/blob/master/docs/multi-platform.md">nerdctl docs</a> point you in the right direction on this one, via <a href="https://github.com/tonistiigi/binfmt">this super-useful emulator</a>. We therefore do the following:</p><pre>limactl shell default                       # onto the VM again<br>sudo systemctl start containerd</pre><pre>sudo nerdctl run --privileged --rm tonistiigi/binfmt --install all</pre><p>… et voila! Our ubuntu image built on amd64 in a private container registry with a local host volume mount works without issue 🎉:</p><pre>lima nerdctl run -v $(pwd):/build --rm -it eu.gcr.io/my-private-gcr-project/alex-ubuntu:latest</pre><blockquote><strong>A small note:</strong> if you need that local directory to be writable by the container, you need to edit the file ~/.lima/default/lima.yaml on your host. There’s a mounts: section where you can choose to make your home directory and everything in it writable (dodgy if you run untrusted containers!), or add a block to a separate mount path, similar to the one already there for /tmp (the option I took!).</blockquote><p>All that’s left is to add the limactl start default to your startup script and alias lima nerdctl to something — you can even alias this to docker if you wish (although I personally prefer to be more explicit — I chose to alias it to nerd 🤘).</p><p>In my opinion, the advantages of this option are:</p><ul><li>It works on Apple Silicon! 🎉</li><li>It handles volume mounts seamlessly.</li><li>It distances you from docker itself and any further licensing fun and games down the line.</li><li>For me, it’s closer to my Production container stack.</li></ul><p>The downsides:</p><ul><li>It’s not actually docker — so there’s a risk of hitting compatibility issues in comparison to e.g. how things are behaving in CI, perhaps.</li><li>It’s a little bit of a faff to get working with a private container registry in particular.</li><li>It <em>seems</em> to take a bit longer to startup than Minikube / Docker Desktop. This is pretty anecdotal though, and not particularly impactful to me day-to-day.</li></ul><h3>Option 4 — Rancher Desktop</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yUQnxK4jKfB6jekL" /><figcaption>Photo by <a href="https://unsplash.com/@iamkendall?utm_source=medium&amp;utm_medium=referral">Kendall Ruth</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>Updated 14/01/2022: </strong>So we’re done right? Lima + nerdctl does the trick? As mentioned back at the top — whilst that was my preferred option for a while, I’ve recently discovered <a href="https://rancherdesktop.io/">Rancher Desktop</a>, and it’s latest version (0.7+) introduced Apple Silicon support.</p><p>This used to be the option I used on both my MacOS machines for quite a while, before switching to colima as described near the top of this article. Rancher remains a perfectly fine choice, with good compatibility.</p><p>Why did I use this option over the rest?</p><ul><li>☑️ Its installation process is very simple. <em>I love a good bit of command line fiddling, but if it’s not necessary and the software keeps itself patched, then it doesn’t have to be.</em></li><li>☑️ It works with all my device types — M1 Mac, Intel Mac, Windows 10.</li><li>☑️ It’s fully compatible with the docker CLI. This is good news for existing build scripts (e.g. if you need to run these or your tests locally, and they depend on being able to call docker directly — pretty common, and avoids surprises if you aliased nerdctl).</li><li>☑️ Volume mounts to the host OS work without any special configuration or adjustments to scripts / compose files.</li><li>☑️ It has a local Kubernetes environment, for the odd occasion I find that handy to have.</li></ul><p>The installation process is extremely simple with only a few choices to make (it does most of it behind the scenes — and makes use of lima to do it on Mac). You will be prompted to elevate your access a few times, which is understandable given what you’re setting up here.</p><p>I opted to use the recommended stable Kubernetes release and the dockerd/moby engine — but I really like that it offered me the choice of containerd (nerdctl) in case that becomes handy in the future (and I understand from the docs that they can coexist too).</p><p>I did hit a couple of small issues that were local to my device, and probably a result of my various experiments on this topic! Here’s a couple of quick bits of <strong>troubleshooting advice </strong>if you need it:</p><ul><li>As a fairly new product, its user experience when things don’t work is not stellar. Logs can be found on Mac here: ~/Library/Logs/rancher-desktop/ and provided me with the clues I needed.</li><li>It is particular about permissions. For example, the /opt directory on my Mac was not owned by root— the logs above clued me in to this need, as I was just faced with an Error Starting Kubernetes — limactl exited with code 1 error through the UI.</li><li>You must run it from the /Applications on your Mac. Copy it over if you didn’t pick the .dmg install option.</li><li>You must use versions &gt;0.7.0 for it to work on Apple Silicon.</li><li><em>Aside: On Windows it needs to reboot a few times. Just roll with it </em>😏</li></ul><h3>Summary</h3><p>I hope you found this article useful — I’ve presented a range of options to show my thought process, and am recommending <strong>colima</strong> (<a href="https://github.com/abiosoft/colima">https://github.com/abiosoft/colima</a>) as the pick, with <strong>Rancher Desktop</strong> (<a href="https://rancherdesktop.io/">https://rancherdesktop.io/</a>) as a viable alternative.</p><p>Hopefully you found this run through the steps useful for your particular setup. As always with these things — and indeed in my own experience following the existing advice out there — it may not work flawlessly on your kit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*HtvbdVcSdaBlaE28lnWfVQ.jpeg" /></figure><p>If you find any issues, do let me know via the comments — I’d be interested to keep this post up to date with any additional advice over time too!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=918a2dcda10" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Developer-Friendly Runbooks: A Guide]]></title>
            <link>https://alex-moss.medium.com/developer-friendly-runbooks-a-guide-9e2526df2fcc?source=rss-75d735bef60b------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e2526df2fcc</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[operability]]></category>
            <category><![CDATA[hugo]]></category>
            <category><![CDATA[runbooks]]></category>
            <category><![CDATA[it-operations]]></category>
            <dc:creator><![CDATA[Alex Moss]]></dc:creator>
            <pubDate>Fri, 24 Sep 2021 21:36:22 GMT</pubDate>
            <atom:updated>2021-09-24T21:36:22.885Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JKsOCVbVBVaiigQc" /><figcaption>Photo by <a href="https://unsplash.com/@cookiethepom?utm_source=medium&amp;utm_medium=referral">Cookie the Pom</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>When</strong> things go wrong — and yes, they will go wrong — it’s extremely helpful to have easy access to a set of runbooks to guide the unfortunate engineer through the steps needed to mitigate the problem as swiftly as possible. In this post I’m going to describe the approach we use for this where I work, which we’ve found to work very well.</p><p>This post is therefore intended to be a guide to creating a site to host runbooks, as well as the sort of content it’s useful to include within them. Read on if that sounds interesting to you!</p><blockquote>To help illustrate some of the points I’m making below, I’ve created my own simplified version of the site — i.e. a lot less content. You can visit it here: <a href="https://runbooks.alexos.dev/">runbooks.alexos.dev</a>, and the source code for it lives <a href="https://github.com/alexdmoss/runbooks">here</a>.</blockquote><h3>What do we need from a Runbooks repository?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bd6KUoiCB3YElWGW" /><figcaption>Photo by <a href="https://unsplash.com/@syinq?utm_source=medium&amp;utm_medium=referral">Susan Q Yin</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>For a collection of runbooks to be successful, there are three essential things in my opinion for that repository to have from the outset:</p><ol><li>It needs to be <strong>easy to access</strong>. The user is often accessing this stuff in less than ideal circumstances. It needs to be straight-forward to get to, easy to navigate / search for the thing you need, and the pages need to load fast over a potentially limited connection.</li><li>It needs to <strong>work when others things aren’t working</strong>. I can’t imagine anything worse than being called out for a problem and finding the thing on fire is also the place where all the instructions on how to put it out are kept!</li><li>It needs to be <strong>easy to update</strong>, in particular by the people who know how to fix things. By that I mean — it mustn’t require some software that not everyone will have, or hide the ability to update it behind complicated access controls or connectivity that aren’t part of the engineer’s normal day job.</li></ol><p>There are some additional things that I think are also really helpful, but not strictly necessary:</p><ul><li>Hosting all the runbooks from a <strong>shared location</strong>. In reality this might be a collection for a single (complex) website, or grouped by business area perhaps. The benefits of this are numerous and not always obvious — “new” engineers always know where to look, and it provides inspiration through visibility of how other teams have approached it. It also helps ensure the contents are self-sustaining — standing on the shoulders of giants to cut down the effort for those that follow, and fostering continual improvement. It also helps in spotting obsolete content for removal.</li><li><strong>Keep the tech simple</strong>. You want something that’s easy to host and unlikely to break, and not need too much specialist knowledge to maintain. In some ways this goes for the contents of the runbooks themselves too — although they need to be detailed enough to be useful, of course 😄</li><li>As a content author, always try to think about <strong>whether you can remove a runbook </strong>— or parts of it — through automation. Can the contents be simplified with a script to gather the relevant diagnostic data? Can something be done to (safely) automatically heal the service when an alert fires, instead of requiring a human to get out of bed and do something?</li></ul><h3>What technology should I use, and where should I host it?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9YnMf29Z2h57fR2e" /><figcaption>Photo by <a href="https://unsplash.com/@introspectivedsgn?utm_source=medium&amp;utm_medium=referral">Erik Mclean</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>As mentioned above — the important thing here is that it works when other things aren’t working. You want a <strong>simple and reliable tech stack</strong>, and hosting that does not have many moving parts of dependencies, and is <strong>separate from where the stuff you need to fix lives</strong>.</p><p>I’ve found a lot of success using static sites generated from markdown — and for this I recommend <a href="https://gohugo.io/">Hugo</a>. As mine (and my work’s) primary hosting technology is Kubernetes I <strong>do not</strong> deploy this to kube. The fact that it’s just a bunch of HTML / other static assets means my choices of where to host are lengthy — I could in fact use an entirely different cloud provider, or Gitlab/Github Pages for example.</p><p>The static content satisfies my desire for the pages load really fast (unless you overdose on the frontend wizardry) and keeps the moving parts (and therefore risk of breakage) to a minimum.</p><p>The use of markdown satisfies my desire to ensure it’s easy to update. The vast majority of engineers are going to be familiar with markdown — and even if they’re not its an extremely simple syntax to pick up / copy from someone else. Text files are easily stored in git, which fits nicely with an engineer’s normal workflow, too.</p><blockquote>A tangential perk of git is that it’s also common and/or easy for those people on call to have a local copy of the runbooks cloned on their machine. This can (usually) be pulled even if something bad does happen to the runbook hosting when it’s needed. Although as noted above this shouldn’t happen ... *<em>cough*</em> shouldn’t *<em>cough*</em>.</blockquote><p>What Hugo brings to the mix is the templating of these markdown files into something that’s easy to use through the theme you create for it. For <a href="https://runbooks.alexos.dev/">runbooks.alexos.dev</a> I’ve customised the <a href="https://themes.gohugo.io/themes/hugo-theme-learn/">Hugo Learn Theme</a>, which is itself based on <a href="http://learn.getgrav.org/">Netlify’s Grav Learn Theme</a>. It’s worked well at work and I don’t recall hearing any complaints in the three or so years we’ve been running with it. It’s also been fairly easy to extend with new functionality when the odd request has come in.</p><h3>A little more on the hosting …</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qYUYgUOP_RlpSyMq" /><figcaption>Photo by <a href="https://unsplash.com/@jamietempleton?utm_source=medium&amp;utm_medium=referral">Jamie Templeton</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>At work, due to the need to still have some authentication &amp; authorisation to get to the content, we stick with the same cloud provider, but use one of their serverless compute options — <strong>Google AppEngine</strong> — instead of Kubernetes.</p><p>Whist AppEngine is arguably a little overkill (compared to say, using a storage bucket), it does give some freebies that we’ve become used to on Kubernetes — automatic SSL certificates, and most crucially automatic integration (through <strong>Google’s Identity Aware Proxy</strong> product) with our corporate identity service. This makes it trivially simple to make our runbooks accessible to large numbers of employees with almost no effort or ongoing maintenance.</p><p>As with most serverless platforms, getting a build &amp; deployment pipeline up and running is also super-simple — see my <a href="https://github.com/alexdmoss/runbooks/blob/main/.gitlab-ci.yml">two-stage Gitlab CI</a>, for example — which more or less boils down to running hugo to generate the HTML and gcloud app deploy to publish it.</p><p>It’s incredibly low cost too 💸, which is a bonus.</p><blockquote>I’m planning to write more about how to host static sites on a variety of GCP hosting options in some future posts — so stay tuned if that’s of interest!</blockquote><h3>So, what sort of content goes into the runbook site?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0HGZ8dVVqq0_DHFI" /><figcaption>Photo by <a href="https://unsplash.com/@eugen1980?utm_source=medium&amp;utm_medium=referral">Eugen Str</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I don’t think it’s crucial or really beneficial to be super-opinionated about what goes into a runbook personally — I think it’s better to trust the owning team’s engineers to put the things they need to know about in there <em>(although I would </em><strong><em>strongly</em></strong><em> encourage </em><a href="https://alexos.dev/2020/03/29/team-nimbus-and-the-agents-of-chaos/"><em>testing your runbooks</em></a><em> to make sure you’ve got that right!)</em>.</p><p>That said, what is useful is having a bit of a template (for mine, see <a href="https://runbooks.alexos.dev/examples/02-runbook-master-template/">here</a> and <a href="https://runbooks.alexos.dev/examples/01-service-index-template/">here</a>) to get things started and inspire those who are staring at the proverbial blank piece of paper.</p><p>What follows are some examples of what I think is good to include — linking to some examples where applicable to illustrate further:</p><ul><li><strong>Key information about the owning team</strong> — <a href="https://runbooks.alexos.dev/application-2/#team-details">Example</a>. Team Name, channels to contact you by, key team members like your PO/PM if applicable. Whilst you might think you know these things, keep in mind that your runbook might be in use by other teams.</li><li><strong>A summary of what the service actually does</strong> — <a href="https://runbooks.alexos.dev/application-1/#service-information">Example-1</a> &amp; <a href="https://runbooks.alexos.dev/application-3/components/">Example-2</a>. This doesn’t have to be down to the minute detail of course — things change! — but a bit of an overview (especially again for those outside your team).</li><li><strong>Important links in an emergency — </strong><a href="https://runbooks.alexos.dev/application-2/#useful-support-links">Example</a>. Your key dashboards, application logs, support rota if applicable, perhaps where your code lives. A low-key useful thing I’d add is any A/B tests or experiments you might be running. New stuff breaks.</li><li><strong>Your upstreams and downstreams </strong>— <a href="https://runbooks.alexos.dev/application-2/general-information/dependencies/">Example</a>. Try to <a href="https://reflectoring.io/upstream-downstream/">align on this terminology</a> if you can 😄. Regardless, listing our who your consumers/clients are, and what other services you depend on to work (internal and external) is useful info in complex architectures.</li><li><strong>Instructions for responding to Alerts </strong>— <a href="https://runbooks.alexos.dev/application-1/#content-renderer-microservice-has-all-pods-down">Example</a>. Deep-link to these from wherever your alerts fire from if at all possible to cut down your time-to-respond. Having to navigate to them is just wasteful. Also as you tune and/or remove your alerts, it’s a good prompt to tidy up the runbook too.</li><li><strong>General How-To’s </strong>— <a href="https://runbooks.alexos.dev/application-3/processes/resolving-dead-letters/">Example-1</a> &amp; <a href="https://runbooks.alexos.dev/application-3/processes/scaling-resources/">Example-2</a>. In other words, these are instructions on how to repair or remediate The Thing, as opposed to instructions on how to respond to an alert.</li><li><strong>How to release and rollback </strong>— <a href="https://runbooks.alexos.dev/application-2/investigating-issues/release-and-rollback/">Example-1</a> <a href="https://runbooks.alexos.dev/application-3/processes/build-and-deployment/">Example-2</a>. Whilst ideally standard, rollbacks are not usually used frequently. Also don’t underestimate the value in having this written down for others — I’ve certainly been in the position of describing this over the phone, and having a guide for them to follow would’ve been handy, believe me!</li><li><strong>Debugging Guide</strong> — <a href="https://runbooks.alexos.dev/application-2/investigating-issues/">Example</a>. Particularly helpful for newer team members — a few tips on where to look for clues is often appreciated. This can be things like where the logs are and how to increase the verbosity, or how to connect to and look at the running services and view their events, and so on.</li><li><strong>Anticipated Failure Scenarios</strong> — <a href="https://runbooks.alexos.dev/application-2/investigating-issues/anticipated-failure-scenarios/">Example</a>. If you’ve spent the time thinking about what could go wrong — a useful exercise in my opinion — then including those in your runbooks is a good idea.</li><li><strong>Third Party Support instructions </strong>— <a href="https://runbooks.alexos.dev/application-3/third-party-support/">Example</a>. If your service depends on third party systems, then how to get in touch with them is extremely useful. The majority of incidents I see at work tend to be this — and often there’s little you can do except chase (although you absolutely should be doing something like <a href="https://martinfowler.com/bliki/CircuitBreaker.html">this</a> if you can too).</li><li><strong>Callout and internal support procedures</strong> — <a href="https://runbooks.alexos.dev/application-3/processes/callouts/">Example-1</a> &amp; <a href="https://runbooks.alexos.dev/application-3/processes/hitperson/">Example-2</a>. How you as a team have agreed to work and prompts for the things you need to do — especially during a major incident where you’ll naturally be distracted trying to fix The Thing.</li></ul><figure><img alt="The I.T Crowd is a masterpiece" src="https://cdn-images-1.medium.com/max/617/1*AAZATwCFBqeHysI5xYfx8g.png" /><figcaption>The I.T Crowd is a masterpiece</figcaption></figure><p>I hope you found something of use in this post. Whilst hosting a static site with some instructions in it might not be the most technologically-complex thing in the world — I hope the post above shows that putting a little thought into it in advance is worthwhile when that site is the thing you really, really need when all kinds of <a href="https://alexos.dev/2020/03/29/team-nimbus-and-the-agents-of-chaos/">chaos is unleashed</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e2526df2fcc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Some musings from KubeCon EU 2021 (Virtual Edition)]]></title>
            <link>https://alex-moss.medium.com/some-musings-from-kubecon-eu-2021-virtual-edition-30b3136f0095?source=rss-75d735bef60b------2</link>
            <guid isPermaLink="false">https://medium.com/p/30b3136f0095</guid>
            <category><![CDATA[conference]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[kubecon]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cncf]]></category>
            <dc:creator><![CDATA[Alex Moss]]></dc:creator>
            <pubDate>Wed, 12 May 2021 18:55:23 GMT</pubDate>
            <atom:updated>2022-06-19T22:40:57.267Z</atom:updated>
            <content:encoded><![CDATA[<p><em>I’ve just finished binging on a whole host of talks from KubeCon EU 2021, and thought I’d scribble down some of my immediate thoughts — from the talks I “went” to, as well as more generally about virtual conferencing these days. Which is a bit of a change of pace, let’s be honest.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ignSVImbisyXUsHd" /><figcaption>Photo by <a href="https://unsplash.com/@jonasjacobsson?utm_source=medium&amp;utm_medium=referral">Jonas Jacobsson</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I should caveat this post with the obvious — my experiences are heavily based around how <em>I</em> chose to take in the conference — the sessions I picked, the things I got involved with and the things I didn’t. Conferences for me tend to be focused on learning about or picking up tips on bits of technology or techniques that I already have some idea about in the first place, rather than totally new things that I don’t think I’m going to have any practical use for in my current line of work. I’m also not that great at the “networking” thing 😅</p><p>Anyway, let’s get into it. Here’s how this is going to go down …</p><ol><li>Overall Thoughts on the conference content.</li><li>Technical Sessions — some more specific thoughts around the main themes of the talks I went for.</li><li>Virtual Conferences and whether they’re worth the money.</li></ol><h3>Overall Thoughts</h3><p>I managed to take in quite a large number of talks (26!) — more than I would if I were there in person (I tend to top out at about 6–7 per day in the flesh, and I think that’s generally considered quite a lot!). Some of the themes that came out for me were:</p><ul><li>It was good to have a sense that at work we are on the right path — many of the things I heard about were things we are aware of and either already doing or looking to adopt. It feels like we’re on the curve now, rather than trailblazing or (perish the thought) lagging behind. It’s nice to have the reassurance 😌</li><li>It’s not really <strong>Kube</strong>Con any more, it feels a lot more like the rest of its name, i.e. CNCFCon (which happens to include Kubernetes). I guess this is a reflection that Kubernetes is kinda the slam-dunk on which the whole CNCF thing is based around. I went to a couple of talks that were about the core features of kube, but frankly not that many. I’m not sure if that’s a reflection of the fact that it’s seen more as a given these days, or perhaps that it’s changing more rarely <em>(probably not the latter, if Kubernetes release notes lately are anything to go by!)</em>.</li><li>With the above in mind, it’s nice to see some of the newer sandbox/incubator projects getting more air time (even in Keynotes). I haven’t had this vibe before, where the feel was more of the already big-hitters. The example I have in my head would be something like <a href="https://keda.sh/">KEDA</a>.</li><li>By the end, I swear if someone mentioned <strong>GitOps</strong> one more time I’m going to … 😠. Seriously, this was definitely the buzzword bingo winner of the talks I ended up going to at least. Perhaps I’m going to notice this more because I really haven’t drunk that particular Kool-Aid — to me GitOps is just a pattern of Continuous Deployment and nothing fancier than that (especially when it’s push-based), and how you choose to do it is not as important as the fact you are doing it at all.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/1*LC5u-iYca91xlgm24uiYiw.jpeg" /></figure><h3>Technical Sessions</h3><p><strong>Multi-Tenancy and Multi-Cluster</strong>. Little bit of a hot topic at work — we’re exploring options to run multiple copies of our large multi-tenant cluster. I took in a few sessions around this topic, enough to make me feel like quite a few others are looking into it too. There was a decent talk explaining some of the thinking going on in SIG multi-cluster and why solving some of these problems is hard (TL:DR — broad range of expectations / use-cases is hard to design for). It was interesting hearing them talk about some of the mistakes of the past, avoiding the rush/temptation to build something too early but also follow the sensible approach of slicing off useful functionality that can be effectively described and implemented, rather than solving the whole thing in one go (hello Agile!).</p><blockquote>There was also a great related talk about the new <a href="https://kubernetes.io/blog/2021/04/22/evolving-kubernetes-networking-with-the-gateway-api/">Gateway API</a> which ticked a lot of boxes for me, solving some meaningful problems like control of traffic distribution and making the separation between Product Developers and Platform Developers more explicit and managable. Me likey 👍</blockquote><p><strong>Sidecars.</strong> Came up a few times. One talk in particular talked about a company with 7+ “standard” sidecars across their fleet, which feels like overkill. Sure, they were solving a lot of operational concerns so the separation from the “business logic” makes a lot of sense, but really, 7? 😮. That seems like a bit of an organisational problem (i.e. unnecessary separation of ownership, but I am speculating). Surely you’d start looking at service meshes (which solved a lot of the ones they listed) or different patterns to get the result you need? We seem to be doing okay with one sidecar and occasionally a second, plus a few DaemonSets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/1*IZXBLynZO3M8zEqTf32Blw.jpeg" /><figcaption><a href="https://devrant.com/rants/3775630/saw-this-on-linkedin">https://devrant.com/rants/3775630/saw-this-on-linkedin</a></figcaption></figure><p><strong>Service Mesh</strong>. Most of these talks were naturally Linkerd-focused as a CNCF project, so less directly useful “this is how to do X” given I am about to embark on an Istio journey. Still, pretty interesting to see some of the use-cases, and frankly a bit of reinforcement of my confirmation bias before we start on a technically complex stream of work to get it rolled out. In these talks I also picked up a couple of interesting definitions of what cloud-native could actually mean:</p><blockquote>“Cloud Native means all your resources and dependencies are network attached”</blockquote><p>Which is a riff on this quote from Duncan Winn (Director of SRE at Google Cloud), which I rather like to be honest!</p><blockquote>“Cloud Native is a term describing software designed to run at scale reliably and predictably, on top of potentially unreliable cloud-based infrastructure”</blockquote><p><strong>Security &amp; Networking.</strong> I found myself somewhat unexpectedly going to a few security &amp; networking related sessions. Always a topic worth staying on top of, and newer/better tools for helping monitoring for security issues is of particular interest lately. There were a few sessions advocating the benefits of eBPF and Cilium and some of the awesome tooling you can build on top of that. It did look rather handy. Interestingly this is also the technology behind GKE’s <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/dataplane-v2">dataplane v2</a> which is new but I have my eye on as a shop running on GKE already. Coincidentally, that technology went GA <a href="https://cloud.google.com/blog/products/containers-kubernetes/bringing-ebpf-and-cilium-to-google-kubernetes-engine">a few days ago</a>.</p><p><strong>Autoscaling.</strong> After some recent interesting challenges at work that can be summed up as “how can I make Kubernetes autoscale fast enough to deal with very sudden bursts of traffic” 📈, I went to a few talks on this to see if there were any new tricks. The answer to that seems to be no, but this in of itself was useful to know, and I learnt about some of the ways other companies have dealt with the problem. One presentation (from StockX) talked through an example we’ve talked about at work before — the idea of a proactive scaler that can be told about upcoming activities (like a marketing push or new product launch) and upscale based on preconfigured known-good configuration (and bring it back down again). That this works for other companies is interesting — but for us, getting to the point of knowing its coming and telling someone about it is the problem statement 😔</p><blockquote>Also on the autoscaling point — great to see decent coverage for <a href="https://keda.sh/">KEDA</a>, as we’ve recently started using this ourselves after all sorts of problems with our previous custom-metrics autoscaling. So far our experience has been positive, and I really like the potential for scale-to-zero in Kubernetes too</blockquote><p>This wasn’t all I covered — I also took in sessions on chaos engineering tools like Litmus, as well as Jaeger, Falco, Gatekeeper, and some positive reinforcement about recent decisions we’ve taken on how to scale up our telemetry stack. Some of those could turn into blog posts in their own right later!</p><h3>Virtual Conferences</h3><p>As mentioned towards the top, one of the things I find with Virtual Conferences is I’m able to take in more talks overall. Whilst they’re all recorded anyway when you go in person, I find that the ability to dip in and out of sessions to see if they’re something I’m going to get value out of is really valuable. Sometimes for me the intro is too basic when I’m already familiar with the product so being able to skip along to the new stuff or demo suits me. Other times it goes into more detail than I can handle and I can bow out without the need to awkwardly shuffle past people or distract the presenter 😣</p><blockquote>In person I’m way too polite to just up and leave, even if the talk isn’t at all what I was hoping for — although I probably should!</blockquote><p>Where virtual conferences loose out for me — and this is despite what is clearly a great amount of effort put in by the organisers — is the social interaction. For me personally, I’m not a “chat to random people about shared interests in the corridor” kind of person, although after a few sweeps around the Expo Hall I will start stopping in on booths for vendors/products I’m interested in. Perhaps for future conferences I’ll use the pre-recorded vendor presentations as a hook to jump onto Slack to ask a question, but certainly for this KubeCon and the other virtual conferences I attended last year, I ended up just skipping out on this part of the conference experience, more or less. This is similar with some of the other trimmings — there were loads of “virtual events” and wellbeing stuff on the event platform this time round (games, trivia, etc.) — if it helps others, I’m all for it of course but this just isn’t my bag.</p><p>So was it worth the money? Well objectively you’d probably expect to say no — the talks are online for free a few days later (I’m writing this on the Wednesday after the conference — the talks are going to be on YouTube on Friday). Except for me, it actually is. Why’s that?</p><ol><li>It was only a tenner for the early bird. That’s three posh coffees ☕, if they were actually still open. I don’t know how many coffees it would take to make it not feel worthwhile, but this wasn’t a hard sell for me.</li><li>The platform used for these has gotten a lot better (certainly in comparison to KubeCon last year at least). It’s much easier to navigate, find similar courses, catch up where you left off, see the Q&amp;A and so on. With some effort I’m sure this could be replicated from YouTube playlists and the conference website, but this is a handy convenience in my opinion. There were even prompts to help you reach out to the presenters when they were online, which I didn’t use but is a nice touch.</li><li>The most important reason though — it forces me to actually get some value 💸. This is the really powerful one for me. Think like gym membership — coz you know you’re paying for it, and it’s over a certain time window, you’re motivated to get your value out of it. In practice this means I politely declined or rescheduled work meetings to take in the talks, I reviewed what I’d covered at the end of each day to help me shape the talks I’d hit the following day, and so on. The same sort of things I’d do if I were there in person.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/459/1*1bLdGmXimjqRcbBKtzJIoQ.png" /><figcaption><a href="http://www.picturequotes.com/dedication-quote-1-picture-quote-598551">http://www.picturequotes.com/dedication-quote-1-picture-quote-598551</a></figcaption></figure><p>In other words, it creates a great excuse to carve out the time for learning 🎓. As long as the price of admittance remains low, I think it’s well worth the few days a couple of times a year to get the value out of it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=30b3136f0095" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to connect to multiple Google Kubernetes clusters easily in parallel]]></title>
            <link>https://alex-moss.medium.com/how-to-connect-to-multiple-google-kubernetes-clusters-easily-in-parallel-2127f5cf9df4?source=rss-75d735bef60b------2</link>
            <guid isPermaLink="false">https://medium.com/p/2127f5cf9df4</guid>
            <category><![CDATA[google-kubernetes-engine]]></category>
            <category><![CDATA[kubectl]]></category>
            <category><![CDATA[gke]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Alex Moss]]></dc:creator>
            <pubDate>Sat, 27 Feb 2021 21:21:35 GMT</pubDate>
            <atom:updated>2022-06-19T22:42:27.249Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eSav5b5NqSd8Ait_" /><figcaption>Photo by <a href="https://unsplash.com/@alvarordesign?utm_source=medium&amp;utm_medium=referral">Alvaro Reyes</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><em>This was first posted on </em><a href="https://alexos.dev/"><em>my personal website</em></a><em> on 2nd Feb 2021. That article also includes some screen captures, so </em><a href="https://alexos.dev/2021/02/02/how-to-connect-to-google-kubernetes-clusters-in-parallel/"><em>head there</em></a><em> if you want to see this stuff in action!</em></p><p>In this post I’m going to talk through the approach I use to switch between multiple Google Kubernetes Engine clusters on the command line. I’d expect a lot of the stuff in here has some benefit for non-GCP Kubernetes clusters too, but the ones I use on a day-to-day basis are all hosted there.</p><p>Key outcomes for me were:</p><ol><li>To be able to switch from one cluster to another with only a small number of commands, even if I have to authenticate with different user accounts (as primarily a GCP user, this means different email addresses / GCP projects).</li><li>To be connected to different Kubernetes in different windows — for example tailing logs in a Prod &amp; Non-Prod cluster at the same time — and for that connection to persist across multiple kubectl commands.</li></ol><blockquote><strong>Spoiler Alert:</strong> I solved this with a bit of fiddling of ~/.kube/config plus the marvellous <a href="https://blog.sbstp.ca/introducing-kubie/">kubie</a>.</blockquote><h3>Background</h3><p>I switched to this approach probably around six months ago. At work we have a relatively small number of clusters so up until that point I was pretty comfortable using what I think is the most common approach of <a href="https://github.com/ahmetb/kubectx">kubectx + kubens</a> and this worked well enough. However I found that I was increasingly getting an inconsistent experience when switching between the clusters I use for work and others I was using for running personal websites <em>(like </em><a href="https://mosstech.io/"><em>my blog</em></a><em>)</em> and for fiddling with things — so I started looking for alternatives.</p><p>For the purposes of explaining things, lets assume a hypothetical setup like this:</p><ul><li>A collection of GKE clusters across several GCP projects at work. Access to these is through a common Production email account, but they are split by project/cluster.</li><li>A sandbox GKE cluster in a different Google organisation where the Prod email account doesn’t have access.</li></ul><p>The hierarchy would therefore look a bit like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/26b5986ee681f3371a5305c596368ef0/href">https://medium.com/media/26b5986ee681f3371a5305c596368ef0/href</a></iframe><blockquote>I feel I should point out here that, at work, we do not name our clusters after types of cheese. I just really fancied some cheese when writing this, okay?</blockquote><p>Enough background — on with how I set things up.</p><h3>First, Multiple Google Accounts</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*M1e2-eqhNdg_wJ-1" /><figcaption>Photo by <a href="https://unsplash.com/@euwars?utm_source=medium&amp;utm_medium=referral">Farzad Nazifi</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>My approach here was massively inspired by <a href="https://medium.com/google-cloud/kubernetes-engine-kubectl-config-b6270d2b656c">this blog post</a> by Googler Daz Wilkin. I’m not going to repeat what is explained really well there already so go have a read if you want to understand why the following works!</p><blockquote><em>For brand new setup, you will need to </em><em>gcloud init once to set up the default configuration. It will also be necessary to </em><em>gcloud auth login on each account used at least once, and this may need refreshing once in a while (but not often enough for me to really notice).</em></blockquote><p>I threw away my pre-saved gcloud configurations — not gonna need ‘em! All I have in ~/.config/gcloud/ is a config_default, which gets updated with a simple bash script when I need to switch between Google Accounts/Projects.</p><p>The switching script looks <a href="https://gist.github.com/alexdmoss/8ac9eea1f55063e2c99f8b7bbe9564cd">like this</a>— with the bits wrapped in &lt;&lt; &gt;&gt; to be replaced. I alias this so I would simply do switch dev for a pre-saved project, or switch gcp-project user-email to activate a new/rarely-used project.ond, Multiple Kubernetes Contexts</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/555667b2cb6563955f006c497694b12e/href">https://medium.com/media/555667b2cb6563955f006c497694b12e/href</a></iframe><p>Here, my ~/.kube/config file does not exist and I set up new configs under ~/.kube/configs/ whenever I have a new cluster I need to deal with. For the number I have to worry about, this is quite manageable, but it could be automated if you had a frequently-changing enough list to be worthwhile. The steps look like this (this must be done in a brand new shell not using kubie — see below!):</p><ol><li>Auth to the new cluster as normal: gcloud container clusters get-credentials ${cluster} — project=${project} — zone=${zone}. This adds an entry to your blank ~/.kube/config.</li><li>Copy a template config file (see below) into ~/.kube/configs/ with a unique name.</li><li>Take the values for clusters.cluster.certificate-authority-data and clusters.cluster.server from the no-longer blank ~/.kube/config and put them into your new file created from the template.</li><li>Update the name: fields for the cluster to reflect what you want it to be known as when you list your contexts — clusters.name , contexts.name and contexts.context.cluster. It does not have to match exactly the cluster name if you want to save typing.</li><li>Delete ~/.kube/config (unless you want to have a default cluster for when not using kubie — but you’ll need to keep this file tidy to avoid confusion!).</li></ol><p>The template I mentioned for this looks as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5d16b6f447d16dc73360ea1f5a8bfe4f/href">https://medium.com/media/5d16b6f447d16dc73360ea1f5a8bfe4f/href</a></iframe><h3>Finally, Loading Parallel Kubernetes Contexts</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Z87shueSDWsHl9O1" /><figcaption>Photo by <a href="https://unsplash.com/@grizzlybear?utm_source=medium&amp;utm_medium=referral">Jonathan Petersson</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>To make use of this shiny new config, we bring in <a href="https://blog.sbstp.ca/introducing-kubie/">kubie</a>. This tool works in a similar way to kubectx + kubens— you specify kubie ctx to set your current cluster, and kubie ns to select a namespace. The difference being, that when you run kubie ctx you spawn a new shell within your terminal window, with the context loaded to that.</p><p>What that means in practice is you can for example have a terminal on the left of your screen connect to prod and a terminal on the right of your screen connected to dev, and both continue to work independently from each other. This is really marvelous.</p><blockquote>I have sufficient muscle memory that I had to alias kctx=&#39;kubie ctx&#39;and alias kns=&#39;kubie ns&#39; to save re-learning / more typing!</blockquote><p>There’s also a kubie exec to run just one command using a different context without swapping out the whole shell if you prefer — for example kubie exec cheddar kube-system kubectl get pods. This is really handy if you want to use this in scripts across multiple clusters.</p><p>There’s way more info/options available — see <a href="https://github.com/sbstp/kubie">the project on github</a> for more ideas.</p><h3>How This Works in Practice</h3><p>If working with two clusters and a shared user account, then I simply issue kctx brie and kctx cheddar in separate terminals and I’m away.</p><p>If the second cluster needs a separate user account, then I would switch sandbox alex@dev.work first, then kctx chutney, and I’m sorted. The only thing I need to keep in mind here is that my <strong>gcloud</strong> context has switched globally (no equivalent of kubie here), so any gcloud SDK commands are going to be against the sandbox project in both terminal windows (unless I switch again) — but my kubectl commands are fine (I suspect until the refresh token expires, but in practice I’ve never had an issue).</p><p>If you’d like to see this working in practice, I recorded a couple of screen captures in the equivalent article to this on my own blog: <a href="https://alexos.dev/2021/02/02/how-to-connect-to-google-kubernetes-clusters-in-parallel/">https://alexos.dev/2021/02/02/how-to-connect-to-google-kubernetes-clusters-in-parallel/</a></p><p>And that’s a wrap — hopefully this inspires you to give kubie a try!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2127f5cf9df4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Things I’ve Learnt After A Year As A Lead Engineer]]></title>
            <link>https://alex-moss.medium.com/things-ive-learnt-after-a-year-as-a-lead-engineer-ae59a9a02fb3?source=rss-75d735bef60b------2</link>
            <guid isPermaLink="false">https://medium.com/p/ae59a9a02fb3</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[continuous-learning]]></category>
            <category><![CDATA[technical-leadership]]></category>
            <dc:creator><![CDATA[Alex Moss]]></dc:creator>
            <pubDate>Sun, 03 Jan 2021 21:47:04 GMT</pubDate>
            <atom:updated>2022-06-19T22:43:32.736Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WHe_XtK1O3ro6jHl" /><figcaption>Photo by <a href="https://unsplash.com/@timmossholder?utm_source=medium&amp;utm_medium=referral">Tim Mossholder</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><em>It’s probably natural to get a bit contemplative as one year rolls into another. In this post I’m going to spend a little time writing about some of the things I’ve learnt after a year moving into a “Senior Engineer” position, and then onto some of the things I hope to change going forwards, based on what I’ve learnt.</em></p><p>As a result, this post is going to be a bit introspective and not very technical — hopefully it won’t come across as too self-indulgent! I personally think it’s sometimes interesting to read about things from this perspective, but turn back now if that doesn’t sound like your cup of tea! 🍵</p><h3>The Background</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4c-J-mKD6wf84av4" /><figcaption>Photo by <a href="https://unsplash.com/@neonbrand?utm_source=medium&amp;utm_medium=referral">NeONBRAND</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>To begin with, I’m going to need to do a brief bit of scene-setting. I work for a large UK-based retailer and mostly tackle things in E-Commerce. At the start of 2020 I had recently switched from a job with “Architect” in its title into a position with “Engineer” instead. This was a deliberate choice as I missed the hands-on detailed involvement you get when actually building something. My career choices to date have always been focused on what tend now to be called “Platforms” — as well as building stuff, I also like working on systems that are large &amp; foundational, and love being able to see what else is going on across lots of different areas … and then influencing the bigger picture. Cake, meet Eat It 🍰</p><p>The year just past is 2020, but I didn’t mention Covid-19, which is clearly still a horrific pandemic which has very much affected how we work. Oh, and I became a father for the first time, which was both awesome and life-changing! On top of that, our house nearly burnt down, leading to the loss of most of our belongings …</p><blockquote>Yeah, this latest season certainly had some interesting plot twists</blockquote><p>Those things do matter for this tale, as they affected how I work, and had a significant impact personally on the amount of time I could devote to my profession.</p><h3>What Have I Learnt</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MF2StaDFSXsnnXLP" /><figcaption>Photo by <a href="https://unsplash.com/@sctgrhm?utm_source=medium&amp;utm_medium=referral">Scott Graham</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>So with that in mind, what are the new things I’ve learnt or have been reinforced for me over the course of 2020?</p><p>I went into the year thinking I’d write more code. I was now an engineer after all — that’s what we do! And I did, but only a little and to be brutally honest not nearly as much as I wanted to. Even though I’d moved from Architecture into Engineering, I couldn’t escape the fact that I had still been around in the company for a comparatively long time, built up a reputation, been involved in some big ticket stuff, and so on. These are things that — despite having a reasonably flat structure in the company — lead to words like “senior” or “lead” being thrown around. And with that responsibility comes a few inescapable truisms that I had underestimated:</p><p><strong>1. Your opinion matters</strong>. People tended to be more interested in my thoughts on things than I had previously realised. This was probably the most significant thing of all that I learnt and tends to go hand in hand with: <strong>you will go to lots of meetings</strong>. This is the nature of the beast <em>(although don’t be afraid to challenge when you don’t think those meetings are valuable of course)</em>. You need to have a good handle on what is going on in your company and your field so that your opinion is actually an <strong>informed</strong> and trusted one. Being able to do this instead of just trying to blag it takes up a lot of time.</p><p><strong>2. Someone needs to make a decision.</strong> If people think of you as the lead or a senior person, then think about whether that someone should be you. I would increasingly hear from people that they appreciated it when I would facilitate and, when needed, actually make decisions on things. It is super-awesome-sauce when you can get consensus as a group, but sometimes someone has to make a call on the direction to go, or when to compromise on quality to get something out the door or fixed <em>(oh, hi there technical debt 👋)</em>, and own that decision (whether it turns out to be a good or a bad one). This outcome also contributes to lots of meetings too, of course.</p><p><strong>3. You can’t know everything</strong>. You need to understand how the pieces fit together; you need to know the general trends in your area of specialism; you need to know the common pitfalls to avoid — but if you think you can know all the details about all the things then you are more than likely making a rod for your own back. A much better option is to trust your team.</p><blockquote>If they’re as awesome as my team then they will know a lot more about the details than you do, and be willing to share that expertise</blockquote><p>Listen to their advice, facilitate good healthy discussions about the topic at hand, and if consensus can’t be reached then make a call on what to do and be clear on <strong>why</strong> you’ve gone in a certain direction. This comes with so many benefits — your job becomes significantly easier <strong>and</strong> you build up a relationship of trust with your team, who hopefully feel valued and that they can contribute to the product as a result.</p><p><strong>4. It doesn’t have to be you</strong>. I believe that the value I tend to provide is more often front-loaded — the early inception of ideas and helping make them reality, in shaping of features and helping set realistic outcomes for delivering them. This probably at times feels a little like Product Management but the ability to do this in a way that ensures engineering quality can be maintained is I think very valuable and something I initially underestimated.</p><blockquote>My day is less about pulling cards across the board and more about creating the cards in the first place, or making sure that they are in a pullable state. My to-do list is mostly littered with “try out X to see if it does Y” or “investigate whether Z is worth actually building around”. This is exciting and fulfilling work.</blockquote><p>There is a trap to avoid here also — the risk of losing sight of work in progress. It is helpful if you’re still there and available to help if the work gets caught up in knots or to spot if it’s going in a direction you didn’t expect. I can think of examples through the year of both where I managed this successfully and where I didn’t do this so well.</p><blockquote>Whilst I went into the year thinking I’d write more code — which didn’t happen as often as I thought it would — I did <strong>read</strong> an awful lot more code (and learnt a lot by doing so!).</blockquote><p><strong>5. You won’t get everything right and that’s ok</strong>. Try to put in place practices that help you avoid making bad mistakes. This is going to vary a fair bit, but in general I mean things like being directly involved in early exploratory activity to experiment with a new idea to really kick the tyres on it, and think about how its success (or not) can be measured as you go. If you work in an agile world then this ought to be bread ’n’ butter to how your team works anyway.</p><h3>Some Changes For This Year</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dQ5mJta9oJTp3f7l" /><figcaption>Photo by <a href="https://unsplash.com/@mjcoury?utm_source=medium&amp;utm_medium=referral">Michael Coury</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>So, after reflecting on the year that’s been, am I going to change my approach in any way? For sure — I have a few experiments in mind to see if they help me be more effective in my role in 2021. Some of my I-am-not-going-to-call-them-resolutions for the year are …</p><p><strong>1. Set aside some dedicated time to do technical work</strong>. Block out time to experiment with new things, or just fix some defects, do some refactoring … whatever really. I’m planning to do this every Friday, and will fight to keep those days as free from meetings and chat as possible, so that I can properly get into things.</p><blockquote>Without this, you will run the risk of losing track of what the problems in the codebase actually are that need fixing, and not staying on top of the new things in your field (techniques, tools, etc.).</blockquote><p>This ties back to both being informed in your opinions and also to aid decision-making — without it you’re fully reliant on others to help make sure you know what you need to know. Possible, but hard to cover all the bases and speak with the same authority.</p><p><strong>2. Make time every day to take in articles, videos or tutorials</strong>. This of course ties back to having informed opinions on things and continuously learning. I’m going to start with what I think will be a fairly modest two articles/tutorials/videos per day, and see how that goes. This was something I lost from my day when suddenly I wasn’t commuting into the office any more.</p><p><strong>3. Blog more</strong>. This article is the first of these! I’m going to try to write about something interesting — ideally technical — once a month. Why am I prioritising this? Well, writing about things is a really good reinforcement of learning and chance to reflect on choices made — even if nobody ever reads them! It really makes you think about what you’ve done more carefully. Plus, I find it pretty fun to do 😎</p><p><strong>4. Devote some time every day for code reviews</strong>. My team is extremely good at having a slick Pull Request procedure that very rarely leaves code unmerged for long — this has been a real boon during Covid times. I feel the biggest benefit this actually gives is visibility of changes — knowing what’s going on (for those informed decisions again …) and learning from others. I’m going to try blocking out the “dead time” of the first hour of the day to go through the previous’ days reviews. Previously, I’ve been dipping in/out — this will be more of a concerted effort.</p><p><strong>5. Be stronger in pushing back on meetings</strong>. Instead of almost always instantly accepting the invite, I’m going to stop and think about whether it really has to be me. Is a meeting needed at all — can it be covered in group/direct message conversations instead for example? Am I invited “just in case” or do I have something genuine to contribute? Is it a good learning opportunity for someone else to take the lead on, and I can instead be the sounding board if needed?</p><h3>Summing Up</h3><p>So, those are my thoughts as 2021 rolls into view. I acknowledge that this is a rather self-indulgent blog post to be brutally honest — if you’ve made it this far and took something of value out of my ramblings then I am very glad! I’ve personally found it useful to put pen to paper on some of these thoughts, and it’ll be interesting to see whether I’m back blogging about this in a year’s time reflecting on how phenomenally well / incredibly badly this plan went 😏.</p><p>At least those objectives I’ve set for myself are measurable, right?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/0*7AHYOIQNNiw0nsII" /></figure><p><em>This blog post is cross-posted on </em><a href="https://alexos.dev/2021/01/03/things-i-have-learnt-after-a-year-as-a-lead-engineer/"><em>my personal blog site</em></a><em>, where I tend to write about things in more depth. Please do take a look around — especially as I intend to write a lot more this year! </em>😉</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ae59a9a02fb3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Team Nimbus and the Agents of Chaos]]></title>
            <link>https://medium.com/john-lewis-software-engineering/team-nimbus-and-the-agents-of-chaos-ab257e41fe36?source=rss-75d735bef60b------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab257e41fe36</guid>
            <category><![CDATA[operability]]></category>
            <category><![CDATA[chaos-engineering]]></category>
            <category><![CDATA[digital-platforms]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Alex Moss]]></dc:creator>
            <pubDate>Thu, 12 Mar 2020 14:07:33 GMT</pubDate>
            <atom:updated>2020-03-12T14:07:33.777Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Hi — we are </em><a href="https://medium.com/@alex.moss"><em>Alex Moss</em></a><em> (Engineering Lead) and </em><a href="https://medium.com/@rob.hornby"><em>Rob Hornby</em></a><em> (Product Owner) for the John Lewis &amp; Partners Digital Platform. Today we’re talking about our team’s first venture into running a Chaos Day — a series of chaos experiments against our platform.</em></p><p>It was January 2020, and we had just gone through another Peak trading period for JL &amp; Partners. The Digital Platform had performed extremely well. There were no incidents, no last-minute panic scaling, and no fall-backs enabled — even though the number of services and overall complexity of the platform was significantly higher than <a href="https://medium.com/john-lewis-software-engineering/a-year-in-google-cloud-4586a117f352">this time last year</a>. Not perhaps the backdrop to make a compelling case for running a series of complex operational test scenarios then? Well, we are not the sort to be resting on our laurels…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aYhDQ-g2HfpTijIh" /><figcaption>Photo by <a href="https://unsplash.com/@federize?utm_source=medium&amp;utm_medium=referral">Federico Beccari</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Bring the Chaos</h3><p>The term <a href="http://principlesofchaos.org/?lang=ENcontent">Chaos Engineering</a> is not a particularly new one, but the benefits it can bring to an increasingly distributed and complex set of systems is significant. If you’re less familiar with the term, <a href="https://medium.com/@adhorn/chaos-engineering-ab0cc9fbd12a">this blog post</a> is an excellent primer, talking through its origins and some of the amazing work that the likes of Netflix Engineering do to pioneer these techniques, as well as what the numerous benefits can be when you do it well.</p><blockquote>As a team we are striving to help uplift our teams to achieve better operability — it seems only right that we should hold ourselves to the same standard and test our own proficiency too!</blockquote><p>Our platform code has a battery of automated tests, of course, but it is difficult to replicate every possible scenario or combination of events — we make implicit assumptions about the behaviour of our systems and in particular the behaviour of <strong>people </strong>involved. In other words, a bit of deliberately subversive thinking can potentially be very beneficial.</p><p>With that in mind, we started planning for our first Chaos Day before Christmas. Some of our colleagues had experience of doing this at other organisations, and this was helpful in giving us a framework for capturing ideas and structuring the day.</p><p>Our goals boiled down to four areas of learning:</p><ul><li>to learn how our platform team responded in incident situations.</li><li>to test our own &amp; other teams’ alerts and their responses.</li><li>to learn how other teams experience failure and how they report it to us as a platform team for assistance.</li><li>to learn how to run a Chaos Day with a small group of responders, before extending or scaling it out more widely as a practice.</li></ul><h3><strong>Running the Day</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hRtZkSrazBFG8fryJHTcLQ.jpeg" /></figure><p>Running the Chaos Day itself compromised six steps:</p><p>1. <strong>Visualise your system</strong>. In order to come up with effective scenarios, you need to understand how the system you are targeting behaves. We gathered all of our Platform Engineers in front of a whiteboard, drew a picture of the main components, and out came the Post-It notes! Tapping into the collective wisdom of the team generated a large backlog of ideas for us to explore further.</p><p>2. Next, you need to <strong>assemble a team of chaos agents</strong>. The prevailing wisdom here is to — especially for your first time doing this — go with experience. In other words, folks who’ve seen things go wrong, and know how they’d fix things if needed. Knowing how to reverse the effects of your test is very important! We opted for a team of three chaos engineers, supported by others on the day.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/0*s_2aNzWeQYmsAiVM" /><figcaption>Chaos Agents Assemble!</figcaption></figure><p>3. With your team assembled, it is time to <strong>capture the experiments</strong>. We followed guidance on how to structure these, which can be summarised as:</p><p><em>a. What is your test actually doing?</em></p><p><em>b. What are you expecting to see in the system as a result?</em></p><p><em>c. What are you expecting in terms of a response?</em></p><p><em>d. How will you roll back if needed?</em></p><p>We threw these on a Trello board to make orchestration easier on the day itself. Each agent ended up with two experiments to run, plus a backup experiment to use if needed.</p><blockquote>One of the hardest things when designing the experiments is not the initial idea, but coming up with a way to reproduce the conditions you are intending. In modern systems, you can rarely just “pull the plug” or simulate strange behaviour downstream in a realistic way. Next time, hopefully we’ll be armed with <a href="https://servicemesh.es/">some additional tools</a> to help with this too.</blockquote><p>4. <strong>Agree a date</strong>, and <strong>send comms</strong> to teams. We didn’t want to just spring it on people, but we did choose a “normal” day, rather than, say, running it on a Friday or early in the morning/evening. After all, we are testing the behaviour of people as much as the system, so wanted things to feel realistic.</p><p>5. <strong>Co-locate</strong> your chaos agents, and start b̵r̵e̵a̵k̵i̵n̵g̵ ̵s̵t̵u̵f̵f̵ experimenting. Our timeline of the day, at a high level:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LnjAa1WtpzeKKVM5" /><figcaption>Some tests were allowed to overlap, as they affected different areas of the platform</figcaption></figure><p>Having a timeline for the day is important — you want to think about the potential effect of overlapping tests in particular. Seeing what happens when folks are at lunch is fine, but tearing down servers while you’ve also pulled the plug on the monitoring tools is a bit harsh!</p><p>Being in the same room is important not just because of the snacks 😉 but because we could each help each other out with our tests — either through mutating the tests to be more effective on the fly, or helping watch the behaviour of responders while a test is going on. We were lucky enough to have a few other interested folks in the room, helping with this aspect (and sworn to secrecy of course!), as well as making sure things were running smoothly in general.</p><blockquote>We ran eight experiments on the day, six of which were planned. The extra two “on the day” tests were based on the behaviours we were seeing at the time. We deliberately tested only in Non-Production, as this was our first time doing this and we needed to practice the process (see <a href="https://medium.com/@njones_18523/chaos-engineering-traps-e3486c526059">Trap 5</a>). We supplied a steady low load through the environment to make it appear more live-like for teams.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*uRaPJrKiupfX6zvOym8Nlw.jpeg" /></figure><p>6. <strong>Learn from it</strong>. This is the most important aspect, otherwise there is no point in doing it! We saved transcripts and screenshots during the day (Slack was amazing for this), chatted to responders afterwards (either individually or in a retro), and wrote up the effects of our experiments in a style similar to a post-incident review (but lighter weight). We’ve also run showcases for our wider tech community, as well as this blog post!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r_cK_j3OVdDftITHJ3vJaQ.png" /></figure><blockquote>We’ve seen teams raise &amp; complete operability tasks on their backlog as a result of our testing, as well as surfaced some gaps in knowledge that have prompted a shift in priorities. We also feel more confident in running one of these again soon.</blockquote><h3><strong>The Tests Themselves</strong></h3><p>Below are examples of a couple of the tests we ran, and our learning from them, to bring this to life.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*TwkMvs9PfaHhk2K1" /></figure><h4>Data is hard</h4><p>For one of our tests, we stopped and later on deleted the data in a relational database that powers one of our customer-facing features on the website. Whilst a relatively unlikely scenario in normal running, human beings make mistakes, and this process was designed to test the response mechanisms to deal with this scenario.</p><p><em>What did we learn?</em></p><ul><li>We had gaps in our alerting to identify the underlying causes, although alerts did fire for an availability breach in the overall service itself.</li><li>We provide instructions to our teams on how to recover their data, but didn’t actually give them permission to do the restore — this resulted in a need to call the platform team. Oops!</li><li>The Tech Lead for the team affected was instrumental in resolving the issues. It would be interesting to repeat this test when he was on holiday — especially when you don’t have a runbook … 😉</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/1*H78mtVUPEgUP6V1xpKfn4Q.jpeg" /></figure><h4>Don’t forget your legacy</h4><p>We have two versions of our digital platform (imaginatively, v1 &amp; v2) — both run critical components for <a href="https://www.johnlewis.com/">www.johnlewis.com</a> — but v1 is older and changes more infrequently. One of our tests broke a component that runs in both. Our engineers spotted and fixed the issue in v2 very quickly ⭐️. But what about the legacy platform?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/918/1*6Opazw86QH6rNxbwv5MXNA.jpeg" /></figure><p><em>What did we learn?</em></p><ul><li>One of our platform engineers is too good at fixing problems — guess who is going to be a chaos agent next time… 😁</li><li>If you are relying on confidence in your alerting to pick up on problems, you need to make sure they have coverage of your older systems too, not just the shiny new stuff you work on day-to-day.</li><li>Older components that are heavily used, but not necessarily actively worked on or owned by a single team, creates a lot of confusion when they break.</li><li>When a component has a reputation for being generally unreliable, people make assumptions that the situation can be ignored (i.e. alert fatigue).</li><li>When an incident earlier in the day was attributed to one service/team, people make assumptions that subsequent issues later in the day are related to the same thing.</li><li>Maintaining engineers’ knowledge of code they don’t work on regularly is very hard — better to eliminate your legacy than carry this risk, perhaps? 😏</li></ul><h4>Asking for help and trusting the source</h4><p>Later in the day we ran an experiment <a href="https://grafana.com/blog/2019/07/24/how-a-production-outage-was-caused-using-kubernetes-pod-priorities/">inspired by an outage in Grafana Labs</a> to consume compute resources through a combination of high requests and Pod priorities. The configuration change was subtle, but the issue was reverted by identifying the misbehaving component and re-executing its deployment pipeline — this effectively restored the component in question to its default (intended) settings.</p><p>The nature of the test (consuming lots of resource on the Kubernetes nodes) meant that, until identified and resolved, the larger workloads on the platform were taken out of service — this happened to be our telemetry tooling (Prometheus) and part of our CI/CD tooling (the GitLab runners). The blast radius for these, especially in Non-Production and when other tests are going on, is high.</p><blockquote>In addition, we had another experiment running at the time: we’d revoked a couple of our engineers’ access to the platform. This was effectively an untested scenario — there’s no runbook for this one — and assumptions are made when onboarding new team members that certain people are around. What does your escalation path look like if they aren’t available and it’s an emergency?</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/299/0*cpW5EaolhQie3a_5" /></figure><p><em>What did we learn?</em></p><ul><li>Infrastructure as code, with automated pipelines that you trust as the source of truth, is a powerful mechanism to restore service.</li><li>When looking after critical systems, it’s important to know when to ask for help from your teammates, and what your escalation routes are if you need them.</li><li>Taking out shared components, such as CI/CD infrastructure or telemetry tools, whilst other experiments are going on is particularly evil 😈. Should we segregate the components required to diagnose and restore service from the workloads in question?</li><li>Don’t taunt the chaos agents!</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/441/0*H1ep9ZKMfV1PCS16" /></figure><h3>Overall Reflections</h3><p>We felt that this first foray into running a Chaos Day went really well — it was a lot of fun and we learnt a lot! Teams involved in responding to the experiments dealt with the scenarios very effectively on the whole, and any experiments that took a long time to be picked up, tended to be where responsibilities were unclear. From a systems perspective, our observability tools, alerting and communication mechanisms for teams worked well.</p><blockquote>It was clear that being a responder to these tests — especially on multiple experiments in a day — is exhausting, much like responding to a real incident.</blockquote><p>The experiments we ran against our older “legacy” components of the platform were the ones that took longer to spot and resolve. As we’ve evolved our platform to meet the needs of newer products, the components we’ve left behind on the older version of our platform have suffered — they work, but they’re less well-understood because they’re not being as actively developed. This has helped us justify making the time to draw down on the larger pieces tech debt.</p><p>There were also a number of learnings at a practical level, such as gaps in alerting, missing runbooks, and incorrect permissions — these should be straightforward to address. It will be interesting to see whether the more of these sessions we run, the fewer examples like this we find.</p><h3>What Next?</h3><p>We do it again! We felt this was valuable enough to be worth running again, extending the number of things we break, and covering a wider range of services and teams.</p><p>Our broader goals include making this a facility that teams feel confident doing themselves, perhaps with assistance in getting started, and eventually moving the testing into Production too.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/0*eSvUY76-7XoAPoaO" /><figcaption>We need to prove that this 👆 is not how we think about things any more!</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab257e41fe36" width="1" height="1" alt=""><hr><p><a href="https://medium.com/john-lewis-software-engineering/team-nimbus-and-the-agents-of-chaos-ab257e41fe36">Team Nimbus and the Agents of Chaos</a> was originally published in <a href="https://medium.com/john-lewis-software-engineering">John Lewis Partnership Software Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Down with the Krew — my favourite kubectl plugins]]></title>
            <link>https://alex-moss.medium.com/down-with-the-krew-my-favourite-kubectl-plugins-279d8d2d5640?source=rss-75d735bef60b------2</link>
            <guid isPermaLink="false">https://medium.com/p/279d8d2d5640</guid>
            <category><![CDATA[krew]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[plugins]]></category>
            <dc:creator><![CDATA[Alex Moss]]></dc:creator>
            <pubDate>Sun, 01 Mar 2020 16:15:30 GMT</pubDate>
            <atom:updated>2022-06-19T22:44:30.389Z</atom:updated>
            <content:encoded><![CDATA[<h3>Down with the Krew — my favourite kubectl plugins</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HJQpa-FC6Sf8EpJO" /><figcaption>Photo by <a href="https://unsplash.com/@quinoal?utm_source=medium&amp;utm_medium=referral">Quino Al</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In this post, I’ll briefly introduce <a href="https://github.com/kubernetes-sigs/krew"><strong>krew</strong></a>, a plugin manager for kubectl, and then run through some of the plugins I’ve discovered with it that I’ve found particularly useful.</p><blockquote>Yes! Made it through the intro without a clickbait title … “<em>top 10 krew plugins to help you devop harder”</em></blockquote><p>If you’d like to see some of the commands and their output in more detail — as well as a little bit more of me attempting to be funny … 😏 — then check out <a href="https://alexos.dev/2020/02/29/getting-down-with-the-kubernetes-krew/">my more detailed post on this here</a>.</p><h3>Introducing krew</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*9_ojFLV081pmjTqO4KP_0g.png" /><figcaption>The official krew logo. It <strong>always</strong> important to have a good logo</figcaption></figure><p>Krew has actually been around since the end of 2018 — mostly likely around the time that kubectl plugins became a thing (v1.12). Krew’s job is to make installing these plugins easier, and was created by the <a href="https://twitter.com/ahmetb">same awesome person </a>responsible for the indispensable <a href="https://github.com/ahmetb/kubectx/">kubectx/kubens</a>.</p><p><a href="https://github.com/kubernetes-sigs/krew">Installation</a> is a breeze, and the basic usage is intuitive:</p><ul><li>kubectl krew list — list your plugins installed through krew. If you only use krew to do this, this should be the same as kubectl plugin list.</li><li>kubectl krew search — find plugins. Without &lt;string&gt;, all are listed.</li><li>kubectl krew info &lt;plugin&gt;— more info about what a plugin does. I always check this to make sure a plugin isn’t going to do something dubious — it invariably contains a link to the source code. And let’s be honest, you’re basically running someone else’s bash on your PC with these plugins.</li><li>kubectl krew install &lt;plugin&gt; — installs it. It can then be used through kubectl &lt;plugin&gt;.</li><li>kubectl krew upgrade — upgrade your plugins.</li></ul><blockquote>At time of writing, there are now ~70 plugins available through it. I feel like when I first started looking at krew in mid-2019, there were only a couple of dozen, so usage is clearly accelerating! 👍</blockquote><p>I’ve grouped a few useful plugins I’ve found into five areas:</p><ol><li>Viewing resources</li><li>Resource usage</li><li>Housekeeping</li><li>Security</li><li>Debugging</li></ol><p>As mentioned earlier, if you want to see these in action with a bit more detail check out <a href="https://mosstech.io/2020/02/29/getting-down-with-the-kubernetes-krew/">my more detailed blog post on the subject</a>.</p><h3><strong>Viewing resources</strong></h3><p>First up in this section we have kubectl tail (usually kail). This is a handy plugin for tailing logs — it’s just nice and simple. There are a number of alternatives out there (I know some folks like <a href="https://github.com/wercker/stern">Stern</a>, which I’ve never got round to trying). The main feature I like is being able to target a deployment/service/ingress (e.g. kubectl tail --ing=mosstech) and be able to watch all the logs from the pods behind it, without necessarily needing to know how the resource has been labelled.</p><p>kubectl get-all <a href="https://en.wikipedia.org/wiki/Does_exactly_what_it_says_on_the_tin">does exactly what it says on the tin</a>. They are not kidding — this <em>really</em> gets everything. I’ve found this to be increasingly useful with our uptake of CustomResourceDefinition — i.e. where it’s trickier to remember every type of object in a namespace.</p><p>Almost the opposite is kubectl neat which neatens up verbose output. If you find your mind glazing over the system-injected annotations and such when get you get a pod’s details, this one might be for you — it strips the gumpf from kubectl get — for example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*np30Ec4Tj8zTeTHHhaRvNw.png" /></figure><p>Lastly for this section are a couple of plugins that help visualise the relationship between a resource and other things in the cluster. kubectl pod-dive is good if you want to know what surrounds a particular pod, e.g.</p><pre>[~ (⎈ |sandpit-prod:prometheus)]$ k pod-dive prometheus-sandpit-0<br>[node]      gke-sandpit-prod-np-0-947e5a45-84zk [ready]<br>[namespace]  ├─┬ prometheus<br>[type]       │ └─┬ statefulset<br>[workload]   │   └─┬ prometheus-sandpit [2 replicas]<br>[pod]        │     └─┬ prometheus-sandpit-0 [running]<br>[containers] │       ├── prometheus [228 restarts]<br>             │       ├── prometheus-config-reloader [0 restarts]<br>             │       └── rules-configmap-reloader [0 restarts]<br>            ...<br>[siblings]   ├── grafana-0<br>             ├── nginx-ingress-controller-85744dcf89-nswr9<br>             ├── fluentd-gcp-v3.1.1-bzh54<br>             ├── heapster-v1.6.1-5b6bf6cc74-cnnlh<br>             ├── kube-proxy-gke-sandpit-prod-np-0-947e5a45-84z<br>             ├── prometheus-to-sd-cjkwx<br>             ├── node-exporter-7ml5r</pre><pre>WAITING:<br>prometheus crashloopbackoff (Back-off 5m0s restarting failed container=prometheus pod=prometheus-sandpit-0_prometheus)<br>TERMINATION:<br>prometheus error (code 1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*lQxB7498xZacHyYenVwZLQ.png" /></figure><p>Somewhat similarly, kubectl tree shows the hierarchy for a particular resource — especially useful when working with CustomResourceDefinition.</p><h3><strong>Resource usage</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*PDFy6NcqPQHpVuueCST41w.png" /></figure><p>There are a whole bunch of plugins in this area — I’ve picked out a couple that I think do a bit more than what is easily achieved with basic kubectl usage. Some of these were particularly helpful when trying to <a href="https://mosstech.io/2019/09/28/squeezing-gke-system-resources-in-small-clusters/">squeeze more out of my tiny ‘home’ GKE cluster</a>.</p><ul><li>First, kubectl resource-capacity offers a nice and simple view for an overall cluster usage.</li><li>Occasionally, info about disk utilisation is useful, and not super-obvious from other sources … enter kubectl df-pv.</li><li>Finally, kubectl view-allocations is handy for spotting pods without resource requests/limits:</li></ul><p><a href="https://asciinema.org/a/NKAfiQL6027Rn7TixYCX1d5tk">untitled</a></p><h3><strong>Housekeeping</strong></h3><p>If you’re running a cluster that’s been around for a while, there are a few plugins that help you track down old or unused object:</p><ul><li>kubectl deprecations — as the name suggests, warns you of objects in your cluster using older versions of objects. On my home setup, it was pretty good at telling me about all the APIs I was using that I’d specified using v1beta1 for example 😃</li><li>kubectl prune-unused configmaps is a handy one — it wipes out (… in other words you should definitely run with --dry-run first …) unused Secrets or ConfigMaps. Great for clusters I use for experimentation — which often leads to trashing things that don’t work out</li><li>kubectl outdated is a very neat idea — it finds images that running older versions than what’s out there in public — you can see it in action below:</li></ul><p><a href="https://asciinema.org/a/M6C23lv6IJc1xp0rVpvBbfPV1">untitled</a></p><p>Note that as it needs to be able to connect to the registry anonymously, it can’t check things that are in my private GCR — but then again, they should have healthy CI/CD pipelines pushing out the latest image automatically. If you pull and re-tag images privately though (e.g. to run them through a vulnerability scanner) then this is a bit less useful to you (or at least, it is without some fiddling).</p><h3><strong>Security</strong></h3><p>I’ll be the first to admit I didn’t “get” the RBAC setup in Kubernetes straight away. Even though I feel like I now do, having a couple of plugins to visualise permissions I find really useful, especially at work where our setup is needfully more complex.</p><p>The first of these is kubectl access-matrix, which in an RBAC-enabled cluster has two modes:</p><ul><li>kubectl access-matrix [-n=namespace] displays a table showing what the current user can do against each resource type. With no namespace, it looks at the cluster scope</li><li>kubectl access-matrix for &lt;resource&gt; displays, for a given resource (e.g. pod), which users/groups/serviceaccounts can perform which roles in which namespaces if applicable.</li></ul><p>It is the latter I find particularly helpful in our PodSecurityPolicy-enabled, multi-tenant (i.e. permissions per namespace) cluster at work.</p><p>Related is kubectl who-can, which as the name suggests shows which subjects can perform what actions on which objects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/1*1BQd-HXQrTArDKFKclmWKA.jpeg" /></figure><p>If you’re worried about the hardening of your pods and willing to trust the folks at <a href="https://kubesec.io/">kubesec.io</a>, then kubectl kubesec-scan is quite interesting. Here’s some output from an old deployment I know still has some dodgy stuff in it 😜</p><pre>[~ $ kubectl kubesec-scan deployment dodgy-app -n=dodgy<br>scanning deployment dodgy-app in namespace dodgy<br>kubesec.io score: -56<br>-----------------<br>Critical<br>1. containers[] .securityContext .capabilities .add | index(&quot;SYS_ADMIN&quot;)<br>CAP_SYS_ADMIN is the most privileged capability and should always be avoided<br>2. containers[] .securityContext .privileged == true<br>Privileged containers can allow almost completely unrestricted host access<br>-----------------<br>Advise1. containers[] .securityContext .runAsNonRoot == true<br>Force the running image to run as a non-root user to ensure least privilege<br>2. containers[] .securityContext .capabilities .drop<br>Reducing kernel capabilities available to a container limits its attack surface<br>3. containers[] .securityContext .readOnlyRootFilesystem == true<br>An immutable root filesystem can prevent malicious binaries being added to PATH and increase attack cost<br>4. containers[] .securityContext .runAsUser &gt; 10000<br>Run as a high-UID user to avoid conflicts with the host&#39;s user table<br>5. containers[] .securityContext .capabilities .drop | index(&quot;ALL&quot;)<br>Drop all capabilities and add only those required to reduce syscall attack surface</pre><p>And finally in this section, a quick one for the lazy / keyboard efficient — kubectl view-secret &lt;secret&gt; &lt;key|-all&gt; — no need to base64 decode things — tedious!</p><h3><strong>Debugging</strong></h3><p>In the last section, I cover a few plugins that can help with debugging issues.</p><p>First up, another one for those who don’t like typing — kubectl iexec — simplifies the kubectl exec -it &lt;pod&gt; /bin/sh wrapper by offering an interactive menu to pick the pod (+ container) you want to exec onto. kubectl pod-shell does basically the same thing too.</p><p>Sometimes — hopefully rarely — you need to elevate privilege to get to the bottom of what’s going on. There are a few krew plugins that help with this process, assuming you have enough permissions to do so:</p><ul><li>kubectl node-admin is scary — it spins up a privileged Pod with the host node mounted, effectively acting as a “remote onto nodes” container. kubectl node-shell does largely the same thing too (except with Alpine, and no fancy node-picker).</li><li>kubectl spy (kubespy) is a handy little utility for spinning up a (privileged) busybox pod that can attach to another pod for debugging purposes. There is also kubectl debug, but this needs the 1.16 alpha feature EphemeralContainers switched on, which I don’t have.</li></ul><p>And finally, if you’ve made it this far, something to bring perfect balance — kubectl snap. For Avengers fans hopefully what this is going to do is obvious 😁 (It deletes half of … everything).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aeaR8lSe3sG-RiOhbCCPnQ.jpeg" /></figure><p>In reality, it is somewhat gentle — see recording below:</p><p><a href="https://asciinema.org/a/FKMYgNQtmvqAxlV4jvQRUi3wE">untitled</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=279d8d2d5640" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>