<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Joseph Irving on Medium]]></title>
        <description><![CDATA[Stories by Joseph Irving on Medium]]></description>
        <link>https://medium.com/@joseph.irving?source=rss-9d42c2a4fd6e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*qR1PSx8NG5Ug-ifzR4SjXw.jpeg</url>
            <title>Stories by Joseph Irving on Medium</title>
            <link>https://medium.com/@joseph.irving?source=rss-9d42c2a4fd6e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 17:59:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@joseph.irving/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Proposing an enhancement to Kubernetes: The story of a KEP]]></title>
            <link>https://labs.rvu.co.uk/proposing-an-enhancement-to-kubernetes-the-story-of-a-kep-f652656d0628?source=rss-9d42c2a4fd6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/f652656d0628</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[contributor]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Joseph Irving]]></dc:creator>
            <pubDate>Tue, 15 Jan 2019 15:25:15 GMT</pubDate>
            <atom:updated>2021-08-13T10:17:19.011Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://kubernetes.io/">Kubernetes</a> is a big project — it has a wide range of users and a large number of people working on it at any given time. For simple bug-fixes and small changes, issues/pull requests are great. However, when trying to convey a much larger change, where a lot of different people may need to be involved, you need something more. This is where <a href="https://github.com/kubernetes/enhancements/blob/master/keps/README.md">Kubernetes Enhancement Proposals (KEPs)</a> come in. In this blog post I’ll talk about my experience, as a relative community outsider, trying to contribute one.</p><h3>Background</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*alVQG8f8F1UBDK8M" /><figcaption>Sidecars are a relatively common design pattern when running applications in pods</figcaption></figure><p>We hit problems working on our <a href="https://github.com/uswitch/vault-creds">Vault sidecar container</a> when attempting to use it as part of a <a href="https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/">Job</a>: when the primary container process completed, the sidecar wouldn’t recognise it should exit and the Job would never complete. It turned out this was a fairly generic problem — imagine you have a Job where you have two containers; one is doing a task (e.g. database migration) and the other is doing something to assist (e.g. mysql proxy). When the container doing the task has completed, the other container doesn’t know to exit so the job will never finish.</p><p>There was a longstanding <a href="https://github.com/kubernetes/kubernetes/issues/25908">issue</a> for situations like this on Kubernetes but no progress had been made. I implemented a <a href="https://github.com/uswitch/vault-creds/commit/abee60e1960892ebcecebbd3e5c2a33df32686cc#diff-3002d695552ec7e7ebbaaf04a21d3439R130">workaround</a> into our Vault sidecar container but thought it would be much nicer to have a solution that would work for all sidecars. I attended KubeCon Copenhagen, went into the <a href="https://github.com/kubernetes/community/tree/master/sig-apps">SIG-Apps</a> session (they seemed like the people to talk to) and asked them if they had any plans to address it. Unfortunately they said no but they’d be happy for me to have a go at solving it. I asked them how I should go about starting this and they told me to raise a KEP, so I did.</p><h3>Making the KEP</h3><p>I decided to fill out the <a href="https://github.com/kubernetes/enhancements/blob/master/keps/0000-kep-template.md">KEP template</a> with a <a href="https://en.wikipedia.org/wiki/Straw_man_proposal">straw man</a> proposal, submit it, capture some feedback and hope that this would start a conversation around the idea.</p><p>The <a href="https://github.com/kubernetes/community/pull/2148">pull request</a> was opened May 14th 2018. I got a few comments but I wasn’t really sure how to progress it further. After a while I reached out to SIG-Apps and they suggested I present the proposal at one of their weekly meetings. I was a bit nervous about presenting my idea to a group of experts, but they were all very welcoming. I talked about my proposal, what I was trying to achieve and why. The general consensus was that it was worth doing, however doing something like this would require <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/">Kubelet</a> level changes so I would have to get another SIG, <a href="https://github.com/kubernetes/community/tree/master/sig-node">SIG-Node</a>, to agree to it as well.</p><p>I reached out to SIG-Node to talk at one of their weekly meetings. However, this didn’t go so well and I felt a little bit discouraged after talking to them. During the meeting, the discussion went on for far too long, there was a lot of confusion about what exactly was being proposed and whether the initial problem I was trying to solve was worth the effort. I think a lot of this was fair but I didn’t feel like we really got anywhere by having the discussion. I think the combination of me poorly communicating the idea (I was a bit nervous) coupled with them being less familiar with the problem to start with led to a somewhat unproductive session. SIG-Node also came at the problem from a very different perspective than SIG-Apps; the former focus on how to run workloads on the nodes themselves while the latter are more interested in how a user can express their workloads. In the end though this different take on the problem was useful and allowed me to have a better idea of how I was going to progress the idea, so I went back to the drawing board and suggested a number of ways forward.</p><h3>Evolving the KEP</h3><p>Activity on my KEP started to languish and it didn’t look like the proposal was going anywhere. It was at this point that I realised that increasing the scope of the proposal was actually the way to get it moving again. This realisation was prompted by someone from <a href="https://istio.io/">Istio</a> commenting. They were having issues with the istio-proxy sidecar: they wanted it to start up before other containers and exit after them. Now this wasn’t exactly the same as what I had been trying to solve but it had definite overlap. I was quite worried about feature creep but as someone pointed out, sometimes feature creep is a good thing if it allows you to solve multiple problems with one solution. I think this was a pretty valuable lesson and something I tried to keep in mind going forward. There were occasions where people would suggest expanding the scope in other ways but I felt they didn’t actually complement the current proposal and were instead just solving a different problem with another solution. Learning when to push back against ideas and when to incorporate them to strengthen the proposal was a skill I had to develop.</p><p>After expanding the scope, the discussion gathered momentum, a few people got quite heavily involved and we got to a point where the proposal was relatively solid. The main outstanding issue was how we implement it in the API: there are countless ways you could do this, all of which would work, so how do you know what the right way is? We had defined the problem, we had defined the solution, we just couldn’t decide the mechanism for actually using the solution. At the time of writing this is where we’re at: the API implementation is still the thing we’re struggling to decide on, but hopefully we’ll make some progress on this soon.</p><h3>Conclusion</h3><p>Proposing a KEP has been an interesting learning experience. Some things I took away from it are:</p><ul><li>Knowing when to incorporate someone’s suggestion and when to politely reject it is not easy.</li><li>Sometimes you just have to pester people on Slack (politely) if you want to get something done.</li><li>Defining the problem and coming up with solution isn’t always the hard part. The hard part is working out how to implement the solution.</li><li>It often just takes a few people who are also really interested in seeing it through to get a proposal moving, instead of a lot of people who just have a passing interest.</li></ul><p>Some feedback for the KEP process:</p><ul><li>Multi-SIG KEPs are hard to coordinate. Talking to an individual SIG in one meeting and then going to another SIG’s meeting to talk about it was not as productive as it could’ve been. You end up playing messenger between them if they disagree. I feel it would’ve been easier if there were some kind of KEP related call where we could’ve all discussed it together.</li><li>It can be quite unclear who has the final say on things — it often felt like it was down to me as the author to make the final decision.</li></ul><p>I shall continue to try to progress this proposal. Feel free to follow along or help out over at <a href="https://github.com/kubernetes/enhancements/issues/753">Kubernetes/enhancements</a>. I’d also like to give a big thanks to all the people who commented on the PR and helped to get it this far. The Kubernetes community really is amazing and I could tell that everyone who got involved just wanted what’s best for all users.</p><p><em>We’re looking for more people to join RVU’s Infrastructure team, where we do stuff like this, create new tools and run Kubernetes clusters. Our </em><a href="https://www.rvu.co.uk/careers"><em>careers</em></a><em> page has more information on </em><a href="https://apply.workable.com/uswitch/j/D03EB8268B/"><em>becoming a Platform Engineer at R</em>VU</a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f652656d0628" width="1" height="1" alt=""><hr><p><a href="https://labs.rvu.co.uk/proposing-an-enhancement-to-kubernetes-the-story-of-a-kep-f652656d0628">Proposing an enhancement to Kubernetes: The story of a KEP</a> was originally published in <a href="https://labs.rvu.co.uk/">RVU Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Making Prometheus more awesome with Thanos]]></title>
            <link>https://labs.rvu.co.uk/making-prometheus-more-awesome-with-thanos-fbec8c6c28ad?source=rss-9d42c2a4fd6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/fbec8c6c28ad</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[metrics]]></category>
            <dc:creator><![CDATA[Joseph Irving]]></dc:creator>
            <pubDate>Wed, 21 Nov 2018 15:29:19 GMT</pubDate>
            <atom:updated>2021-08-13T10:18:17.552Z</atom:updated>
            <content:encoded><![CDATA[<p>No it doesn’t delete half your metrics.</p><p>Standalone <a href="https://prometheus.io/">Prometheus</a> is pretty great: it provides a great query language along with a simple unified way of collecting and exposing metrics. Making Prometheus be highly available and scalable, however, can often be a bit of a challenge.</p><p>The key features we needed were:</p><ul><li>Highly available Prometheus</li><li>Single place to query all of your metrics</li><li>Easily back up and archive data</li></ul><p>This is where <a href="https://github.com/improbable-eng/thanos">Improbable’s Thanos</a> comes in.</p><h3>Making Prometheus HA</h3><p>Thanos at its most basic allows you to query multiple Prometheus instances at once and has the ability to deduplicate the same metric from multiple instances. This allows you to run multiple replicas of the same Prometheus setup without worrying about metric duplication.</p><p>One of Thanos’ components is a <a href="https://github.com/improbable-eng/thanos/blob/master/docs/components/sidecar.md">Thanos sidecar</a> that runs alongside each Prometheus container that together form a cluster. Instead of querying directly to the Prometheis (this is the official plural <a href="https://prometheus.io/docs/introduction/faq/#what-is-the-plural-of-prometheus?">according to Prometheus</a>) you query the <a href="https://github.com/improbable-eng/thanos/blob/master/docs/components/query.md">Thanos Query</a> component. The picture below helps to understand the relationship between Prometheus and Thanos.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/0*bQPH387iOTDBo7q5" /></figure><p>This alone is great as it allows you to easily make a HA Prometheus setup but there’s even more you can do with these building blocks.</p><h3>A single place to view Metrics</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GbDcwTfHonrUFdFjtPE8BA.jpeg" /><figcaption>Having too many places to look for metrics is not very efficient</figcaption></figure><p>The next problem we looked at was how to get all your metrics into one place.</p><p>We run multiple Kubernetes clusters all with their own Prometheus. Historically we aggregated the metrics by having a special Prometheus that scraped each of the Prometheus’ federate endpoint. This did the job but was wasteful as we were just duplicating all of our metrics and this special Prometheus was a single point of failure.</p><p>A Thanos Query node can use another Query node as a source of data; if we expose the gRPC endpoint of our Thanos Query nodes in each cluster we can create a Thanos Query that aggregates them together by using them as stores. The picture below helps to illustrate this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/0*kysQFHSsHFhxuncw" /></figure><p>This allows us to go to one Thanos Query and get all our metrics across all our clusters. In the following screenshot I’m querying the number of replicas for our fluentd Daemonset across our three clusters (red, black, blue) with just one query.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*d9Cku_a4FEkoFF_f" /></figure><p>However there is still a problem with this setup as we have one special Thanos Query in a special cluster that will take out our single metrics view if it goes down. Instead, we’d like to run multiple Thanos Query nodes, one in each cluster, for users to query against via some kind of load-balancing. Using our AWS multi-cluster load-balancing tool <a href="https://github.com/uswitch/yggdrasil">Yggdrasil</a> (read more about that <a href="https://medium.com/uswitch-labs/multi-cluster-kubernetes-load-balancing-in-aws-with-yggdrasil-c1583ea7d78f">here</a>) we can spread this out across multiple Kubernetes Clusters. The result is users can perform queries against any cluster and receive all metrics.</p><p>When you put this all together it looks something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/0*iMsyvFHQ4qtuiHpo" /></figure><p>Note that each Thanos Query layer will be a ReplicaSet of Thanos Query nodes for added resilience.</p><p>This gives us an incredibly resilient Prometheus setup, spread out across multiple clusters with multiple replicas giving us a lot of layers of redundancy. You have one place to go to if you want to see your Prometheus metrics, no need to worry about what cluster/namespace etc that your application is running in.</p><h3>Storage</h3><p>Another common problem with Prometheus is backing up and retaining all your metrics; keeping data on your Prometheus instances is often expensive in terms of storage as well as impacting performance. Thanos solves this by having the sidecar continuously back up your data to a cloud storage provider such as S3 and then exposing the data via a <a href="https://github.com/improbable-eng/thanos/blob/master/docs/components/store.md">Store</a> node. The store node acts kind of like having another Prometheus instance in your Thanos cluster but with all the data coming from the S3 bucket.</p><p>The store node also gives you some nice resilience in the event of a cluster outage, if we lose an entire cluster I can no longer query the most recent data as those Prometheis are gone, but I can still query the store node in another cluster which has access to the historical data from S3.</p><h3>Further Work</h3><p>One of the things we’ve wanted to do for a while is split up our Prometheis per team/namespace so that our individual Prometheus instances don’t have to be too large and to give us redundancy in the event of one team generating a very large amount of metrics that takes out Prometheus. This was always considered too much effort as having a separate endpoint for each team/namespace would’ve been a lot of overhead, but with Thanos we can just add the team based Prometheis to our Thanos cluster and still use the same single source for your metrics. So we’d like to switch to having many small Prometheis and use the <a href="https://github.com/coreos/prometheus-operator">Prometheus Operator</a> to make it nice and simple to create them.</p><p>To conclude, Thanos has given us:</p><ul><li>Highly available Prometheus — Deduplication with Thanos sidecar + Thanos query</li><li>Single place to query all of your metrics — Thanos query aggregating across all clusters</li><li>Easily back up and archive data — Thanos sidecar and Thanos store for s3 data storage</li></ul><p>Overall Thanos has really taken our metrics setup to the next level, it’s allowed us to simplify how our users find their metrics and given us a load of resilience in the process. It’s now incredibly simple to add metrics from a new Kubernetes cluster and even things not running in Kubernetes, as it all just works via the building blocks that Thanos provide. However, one caveat would be that Thanos is still a relatively new product so it’s not without the odd bug, thankfully this is countered by their very active team and community which are continuously improving it. Please check out their <a href="https://github.com/improbable-eng/thanos">Github project</a> or their <a href="https://improbable-eng.slack.com/">slack channel</a> if you want to join in.</p><p>Update: I did a talk on this topic at Prometheus London, where I talk about the setup described here as well as some of the further work we did with the Prometheus Operator, VPA and more. You can watch <a href="https://youtu.be/NuWTNrGyWLQ">the talk here</a>.</p><p><em>We’re looking for more people to join RVU’s Infrastructure team that work on implementing tools like this, create new tools and run our Kubernetes clusters. Our </em><a href="https://www.rvu.co.uk/careers"><em>careers</em></a><em> page has more information on </em><a href="https://apply.workable.com/uswitch/j/D03EB8268B/"><em>becoming a Platform Engineer at R</em>VU</a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fbec8c6c28ad" width="1" height="1" alt=""><hr><p><a href="https://labs.rvu.co.uk/making-prometheus-more-awesome-with-thanos-fbec8c6c28ad">Making Prometheus more awesome with Thanos</a> was originally published in <a href="https://labs.rvu.co.uk/">RVU Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dynamic database credentials with Vault in Kubernetes]]></title>
            <link>https://labs.rvu.co.uk/dynamic-database-credentials-with-vault-in-kubernetes-548d0bcc93d3?source=rss-9d42c2a4fd6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/548d0bcc93d3</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[vault]]></category>
            <category><![CDATA[gdpr]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[go]]></category>
            <dc:creator><![CDATA[Joseph Irving]]></dc:creator>
            <pubDate>Tue, 04 Sep 2018 15:43:28 GMT</pubDate>
            <atom:updated>2021-08-13T10:19:04.815Z</atom:updated>
            <content:encoded><![CDATA[<p>How we at Uswitch managed to get all our applications to use short-lived database credentials without changing any of their code (almost).</p><p>Static database credentials tend to slowly accumulate and get spread around in most organisations and, over time, they become a security liability and need rotating. Uswitch is no different.</p><p>This often causes untold pain when you have 50 different services all using the same password with each having to get it from a different place — meaning you have to change each one individually. Not only is this a time sink, you also have no way of knowing what service is interacting with your database at a given time. This can be quite problematic in the event of a malicious attack.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QQLRfTpt5ycyBM9ocDsC9w.jpeg" /></figure><p>The obvious solution to this was to switch to using short-lived database credentials. <a href="https://www.vaultproject.io/">Hashicorp’s Vaul</a>t has the ability to generate <a href="https://www.vaultproject.io/docs/secrets/databases/index.html">short-lived credentials for a database</a>, so it seemed like a good choice. We also needed way to tie the identity of the pod to the credentials that were generated, and Vault has the ability to use service accounts as a <a href="https://www.vaultproject.io/docs/auth/kubernetes.html">method of authentication</a> which would allow us to do this.</p><p>The basic authentication flow would be: you start your pod; you authenticate with Vault using your service account; then you request credentials for your database. This means you no longer have static credentials, because each set of credentials is tied to an individual pod.</p><h3>The Vault-Creds sidecar</h3><p>This is all great but, when you have hundreds of microservices (which we do at Uswitch) it takes a lot of time and effort to change every single one of them to authenticate with Vault to get their credentials.</p><p>This is why we built <a href="https://github.com/uswitch/vault-creds">vault-creds</a>, a sidecar container that will use your service account to authenticate with Vault, get your credentials and output them to a file. The sidecar will continually renew the credentials while the pod is running, sending a revocation request when the pod shuts down. By using this sidecar, you drastically reduce the amount of change required for the apps — as long as they can read their DB config from a file (which most of them will be doing anyway) it should be very easy to get them to use these short-lived credentials.</p><p>The config on the sidecar looks something like this:</p><pre>--vault-addr=https://vault.example.com</pre><pre>--ca-cert=/vault.ca</pre><pre>--secret-path=database/creds/mydb_readonly</pre><pre>--login-path=kubernetes/foo/login</pre><pre>--auth-role=mydb_mynamespace_myserviceaccount</pre><pre>--template=/creds/template/mytemplate</pre><pre>--out=/creds/output/mycreds</pre><pre>--renew-interval=1h</pre><pre>--lease-duration=12h</pre><p>As you can see, there are quite a few configuration options when using vault-creds. While this is good for flexibility, when you’re trying to apply this config to a large number of applications it can be quite error prone. To remedy this we made a go binary that will take your Kubernetes resource file, along with a few config options, and then output your resource with the vault-creds sidecar added to it. This was typically run as part of a <a href="https://drone.io/">Drone pipeline</a> (our CI tool) before applying your resource to the cluster.</p><p>The config in your .drone.yml would look something like this:</p><pre>kubernetes:<br> template: mydeployment_in.yaml<br> output: my_deployment_out.yaml<br>vault:<br> — database: my_database<br>   role: readwrite<br>   secret:<br>   templateVolume:<br>     name: database-template<br>     path: database.yml.in<br>   outputVolume:<br>     name: database-secret<br>     path: database.yml</pre><p>This definitely helped when switching to Vault, but it had some drawbacks.</p><p>First, it meant adding vault-creds to your resources was tied to using the CI system, which was awkward for local development. Second, it had way too many config options, a lot of which developers didn’t care about so they just copied and pasted the example around.</p><h3>Using a mutating admission webhook</h3><p>It would be a lot nicer if this all happened in Kubernetes itself, making it far more agnostic to how the applications are being deployed. This is why we made <a href="https://github.com/uswitch/vault-webhook">vault-webhook</a> which is a <a href="https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#mutatingadmissionwebhook-beta-in-1-9">MutatingAdmissionWebhook</a>.</p><p>A mutating admission webhook allows you to mutate a pod as it’s created, modifying its spec or adding new things. The vault-webhook adds the vault-creds sidecar and some volumes for mounting the credentials into. The configuration is all handled by a custom resource called a DatabaseCredentialBinding.</p><pre>---<br>apiVersion: vaultwebhook.uswitch.com/v1alpha1<br>kind: DatabaseCredentialBinding<br>metadata:<br> name: mybinding<br> namespace: mynamespace<br>spec:<br> serviceAccount: my_service_account<br> database: mydb<br> role: readonly<br> outputPath: /config #Optional: defaults to /etc/database<br> outputFile: mycreds #Optional: defaults to database-role</pre><p>The resource matches service accounts to databases as this is what you’re using to authenticate with Vault anyway. So, whenever you create a pod which has a service account associated with one of these bindings, it will automatically have the vault-creds sidecar added to it with the config all set up for you. It’s as simple as just deploying a binding and your application, using whatever method of deployment suits you.</p><p>The combination of Vault, vault-creds and the vault-webhook has allowed us to get to a point where it’s very easy to deploy applications with short-lived credentials without the developers needing to make massive changes to their code.</p><p>However, there are still some issues we need to address such as applications not coping well when the credentials get rotated by the vault-creds sidecar; something we might be able to solve using something like <a href="https://github.com/mysql/mysql-proxy">mysql-proxy</a>.</p><p>If you want to follow our progress please feel free to check out <a href="https://github.com/uswitch/vault-creds">https://github.com/uswitch/vault-creds</a> and <a href="https://github.com/uswitch/vault-webhook">https://github.com/uswitch/vault-webhook</a>. These are both open source, PRs, issues etc are welcome.</p><p><em>We’re looking for more people to join RVU’s Cloud Infrastructure team that work on on tools like this and run our Kubernetes clusters. Our </em><a href="https://www.rvu.co.uk/careers"><em>careers</em></a><em> page has more information on </em><a href="https://apply.workable.com/uswitch/j/D03EB8268B/"><em>becoming a Platform Engineer at R</em>VU</a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=548d0bcc93d3" width="1" height="1" alt=""><hr><p><a href="https://labs.rvu.co.uk/dynamic-database-credentials-with-vault-in-kubernetes-548d0bcc93d3">Dynamic database credentials with Vault in Kubernetes</a> was originally published in <a href="https://labs.rvu.co.uk/">RVU Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multi-cluster Kubernetes load balancing in AWS with Yggdrasil]]></title>
            <link>https://labs.rvu.co.uk/multi-cluster-kubernetes-load-balancing-in-aws-with-yggdrasil-c1583ea7d78f?source=rss-9d42c2a4fd6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/c1583ea7d78f</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[load-balancing]]></category>
            <dc:creator><![CDATA[Joseph Irving]]></dc:creator>
            <pubDate>Tue, 07 Aug 2018 14:13:55 GMT</pubDate>
            <atom:updated>2021-08-13T10:19:49.685Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HItcyUcWWnuqDzKgNCoprw.jpeg" /><figcaption>Yggdrasil, from Norse mythology, is the world tree that links the nine realms.</figcaption></figure><p><a href="https://github.com/uswitch/yggdrasil">Yggdrasil</a> is a tool we wrote to allow our services to be load balanced across multiple Kubernetes clusters running in AWS. It behaves as an <a href="https://github.com/envoyproxy/envoy">Envoy</a> control plane, generating configuration from Kubernetes Ingress resources. Yggdrasil is agnostic to the Ingress controller allowing it to work with existing resources.</p><p>At Us<a href="https://uswitch.com/">witch</a> we’re running almost everything on <a href="https://kubernetes.io/">Kubernetes</a> (you can read more about that <a href="https://medium.com/@pingles/convergence-to-kubernetes-137ffa7ea2bc">here</a>). It’s brought us a lot of benefits, but people were concerned it would introduce a single point of failure.</p><p>In an ideal world nothing would ever break, but we all know that occasionally an upgrade can go awry or some unanticipated scenario can take out your cluster. For some of our most important applications this was a deal breaker because going down even for a few minutes could lose us a lot of money.</p><p>To mitigate against a cluster-wide outage, we wanted to be able to deploy the same application to multiple Kubernetes clusters and have intelligent load balancing between them. Unfortunately there wasn’t an easy way to achieve this when running Kubernetes in AWS.</p><p><a href="https://kubernetes.io/docs/concepts/cluster-administration/federation/">Federation</a> has come a long way but federated Ingress only works on GCP. <a href="https://github.com/heptio/contour">Heptio’s Contour</a> was potentially an option but we wanted something that wouldn’t require drastically changing our existing setup; we just wanted to put something on top of it.</p><h3>The proposal</h3><p>We decided early on that we wanted the solution to build on top of things that were already familiar to developers. We didn’t want deploying an application to multiple clusters to be vastly different from deploying to a single one.</p><p>As almost all our applications were being exposed via <a href="https://kubernetes.io/docs/concepts/services-networking/ingress/">Ingress</a>, it made sense to build on top of that. However, Ingresses can be created, updated and deleted very rapidly, so we wanted something that could cope with this kind of dynamic environment and <a href="https://www.envoyproxy.io/">Envoy</a> seemed like an ideal choice due to its dynamic configuration capabilities, circuit breaking, health checking, etc.</p><p>We started off with a small Envoy cluster with static config for each Ingress that we wanted to be load balanced across multiple clusters. Envoy would listen on the same host as the host defined in the Ingress, with each Ingress load balancer being configured as an address for a given host.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/0*MA6zKtF3icTTPrBa" /></figure><p>For example, example.com would have ingress.a.com, ingress.b.com and ingress.c.com as addresses causing it to load balance across them. The diagram below helps describe this topology.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/0*8kiFvVtIZtpbEp3R" /></figure><p>We tested this out with one of our non-critical production services. We configured it to be served via Envoy, deployed it to each cluster and then scaled the number of replicas to zero in one of the clusters. Our monitoring showed that no users saw errors while Envoy continued to health check the broken cluster.</p><p>Below you can see a graph of requests going to an application running in multiple clusters behind Envoy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KBMSx2CLpNgdTt7r" /></figure><p>You can see where we turned off the application in Cluster B: requests drop to close to zero for that cluster and requests increase in the other clusters as load is redistributed.</p><p>A few requests remain in cluster B as Envoy is checking whether it is healthy again. Once it’s turned back on, Envoy starts redistributing the load back to that cluster.</p><p>This was a good start and developers soon started using it. Despite this, it was static, and adding multi-cluster support for a new Ingress would require some understanding of Envoy configuration language. And, as more people used it, the config grew and became unwieldy. We needed something that could automate the Envoy configuration for us.</p><h3>Enter Yggdrasil</h3><p>Envoy was built with the idea of it being able to <a href="https://github.com/envoyproxy/data-plane-api/blob/master/XDS_PROTOCOL.md">dynamically discover configuration via GRPC</a>. This is where <a href="https://github.com/uswitch/yggdrasil">Yggdrasil</a> comes in. Yggdrasil is an Envoy control plane, generating configuration for Envoy dynamically from Ingress objects in a Kubernetes cluster. It regularly searches for Ingresses of a specific class and will use the host and load balancer address from the Ingress object to create Envoy config, it will then combine Ingresses from different clusters with the same host together. The Envoy nodes are then deployed with very minimal config; they’re just pointed at Yggdrasil as a source of dynamic config.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/0*vNh1YmM66Gnft5Lg" /></figure><p>Once this was deployed it became relatively trivial to set up an Ingress that was multi-cluster: you just create an Ingress with the right Ingress class, deploy it to as many clusters as you like (our <a href="https://drone.io/">Continuous Integration tool</a> can target multiple clusters at once) and watch as traffic is automatically load balanced across them.</p><h3>What’s next</h3><p>Yggdrasil allowed us to get load balancing across our clusters in a fairly simple way that did not require us to change the way our clusters were set up, or create more methods of communication between them.</p><p>However, it does have some limitations: for example, you can’t have it communicate with a different set of certificates on a per-ingress basis, which can be a problem for apps that require mutual TLS. The setup is fairly opinionated and based around what we were doing, but I think the fundamental building blocks are now in place and we can definitely look at doing more complex things with it in the future. The project is totally open source, so if anyone wants to try it out, give feedback, raise issues, etc., please do.</p><p><a href="https://github.com/uswitch/yggdrasil">https://github.com/uswitch/yggdrasil</a></p><p><em>We’re looking for more people to join RVU’s Cloud Infrastructure team that work on on tools like Yggdrasil and run our Kubernetes clusters. Our </em><a href="https://www.rvu.co.uk/careers"><em>careers</em></a><em> page has more information on </em><a href="https://apply.workable.com/uswitch/j/D03EB8268B/"><em>becoming a Platform Engineer at R</em>VU</a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1583ea7d78f" width="1" height="1" alt=""><hr><p><a href="https://labs.rvu.co.uk/multi-cluster-kubernetes-load-balancing-in-aws-with-yggdrasil-c1583ea7d78f">Multi-cluster Kubernetes load balancing in AWS with Yggdrasil</a> was originally published in <a href="https://labs.rvu.co.uk/">RVU Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>