<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Neil White on Medium]]></title>
        <description><![CDATA[Stories by Neil White on Medium]]></description>
        <link>https://medium.com/@neilcameronwhite?source=rss-807665507843------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*LtzAVN8QvnUAZh5sKUfpRg.jpeg</url>
            <title>Stories by Neil White on Medium</title>
            <link>https://medium.com/@neilcameronwhite?source=rss-807665507843------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 19:50:59 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@neilcameronwhite/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Under Disk Pressure]]></title>
            <link>https://neilcameronwhite.medium.com/under-disk-pressure-34b5ba4284b6?source=rss-807665507843------2</link>
            <guid isPermaLink="false">https://medium.com/p/34b5ba4284b6</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[lsdcontainer]]></category>
            <category><![CDATA[lsdobserve]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[kibana]]></category>
            <dc:creator><![CDATA[Neil White]]></dc:creator>
            <pubDate>Tue, 24 Aug 2021 16:46:11 GMT</pubDate>
            <atom:updated>2021-08-24T16:46:11.190Z</atom:updated>
            <content:encoded><![CDATA[<p>This is a story of how a single pod can bring a node and cluster to it’s knees!</p><p>I’ve been working with Kubernetes for a while and one of the big issues I have to deal with is storage. Now I hear your cries, storage on Kubernetes is a problem, but I am not talking about <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/">Persistent Volumes</a>, I am talking about something else that might be a bigger pain in the butt, pod log storage!</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F6TLo4Z_LWu4%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D6TLo4Z_LWu4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F6TLo4Z_LWu4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/64d6c8a4359da9ba0506987a8ef27195/href">https://medium.com/media/64d6c8a4359da9ba0506987a8ef27195/href</a></iframe><p>What exactly am I talking about? I am talking about <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/#node-conditions">Node-pressure Eviction</a>, and specifically <strong><em>DiskPressure</em></strong>.</p><blockquote>Top Tip. DiskPressure != OutOfDisk</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KwmAuMpbstM7Z1KCr6eoKA.png" /><figcaption><a href="https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/#node-conditions">https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/#node-conditions</a></figcaption></figure><p>Yeah, so who cares! Disk Pressure Disk Schmessure! Well sir (or lady however you identify) because disk is one of the things you cannot really control, you should be caring. Yes I have <a href="https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/">resource requests and constraints</a> that I can set so that a single pod cannot use all the CPU or Memory, but AFAIK there is nothing stopping you from throttling disk IO or disk storage.</p><p>And what I mean by that is, what is stopping a tiny little pod from generating thousand upon thousands of log entries to stdout?</p><p>Nothing really. And this is a big big problem because those logs are written somewhere and are taking up storage somewhere? Yeah you never thought of that did you! Well thank Odin for me and what I am about to show you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*jhqFmUQjoJdUlkfgOQZFfw.gif" /><figcaption>Pod logs have to go somewhere, right!</figcaption></figure><p>This is important because when the root filesystem or image filesystem hits it’s eviction threshold (standard is 85%) pods get evicted! Yes Kubernetes tries to mitigate this by <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/#reclaim-node-resources">trying to clear up some storage</a> but if it cannot bring down the filesystem utilization it will evict all the pods on that node.</p><p>Where is the image filesystem? Good question. Unless you have asked for it to be in a different location, it is under /var/lib/docker (since most of us use docker)</p><p>Here is how you can find our your current utilization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/1*dmd1nvPgOfWbkF0LDvijow.png" /></figure><p>Remember if “<strong>Use%</strong>” hit’s <strong>85%</strong> we are in for a bad time.</p><h3>Let’s start at the beginning</h3><p>Most of us use Docker, so I am going to work with that container runtime, however all container runtimes work on the same principal. Docker saves all logs from stdout to a flat file somewhere on a node. Want to find out where?</p><p>I am going to pick on a random pod to demonstrate how you find out exactly where logs are located “vitality-orchestration-service-13-hl4hx”</p><p>If I describe that pod it gives me some clues:</p><pre>$ kubectl describe po vitality-orchestration-service-13-hl4hx | egrep &quot;Node:|Container ID:&quot;</pre><pre>Node:<br>vgqengapps01/172.26.X.X</pre><pre>Container ID:   docker://d28698dcd8830663d2980c632d3fc36786b6b57d0841092da3f2e6f4d2552177</pre><p>That tells us that this pod is running on a node called “vgqengapps01” and on that node the pod has the docker container ID of “d28698dcd8830663d2980c632d3fc36786b6b57d0841092da3f2e6f4d2552177”</p><p>Let’s jump onto that node and get some more detail (you need to become root, it’s simply easier)</p><pre>$ ssh vgqengapps01<br>$ sudo -s<br>$ docker inspect d28698dcd8830663d2980c632d3fc36786b6b57d0841092da3f2e6f4d2552177 | grep LogPath</pre><pre>&quot;LogPath&quot;: &quot;/var/lib/docker/containers/d28698dcd8830663d2980c632d3fc36786b6b57d0841092da3f2e6f4d2552177/d28698dcd8830663d2980c632d3fc36786b6b57d0841092da3f2e6f4d2552177-json.log&quot;,</pre><p>That shows us the local location of that log file. That means that if I were to run:</p><pre>$ cat /var/lib/docker/containers/d28698dcd8830663d2980c632d3fc36786b6b57d0841092da3f2e6f4d2552177/d28698dcd8830663d2980c632d3fc36786b6b57d0841092da3f2e6f4d2552177-json.log</pre><p>or</p><pre>$ kubectl logs vitality-programme-business-service-16-psm84</pre><p>I will get the same result! Amazing!</p><p>Oh, and what is the size of all those logs?</p><pre>$ ls -1sh /var/lib/docker/containers/d28698dcd8830663d2980c632d3fc36786b6b57d0841092da3f2e6f4d2552177/d28698dcd8830663d2980c632d3fc36786b6b57d0841092da3f2e6f4d2552177-json.log</pre><pre>15M </pre><p>15 Megabytes, not bad not bad. But is anything stopping this guy (or gal, you know my feeling on this) from using 15 Gigabytes of logs? I will not regale you of a story where someone put an entire Kafka cluster running on Kubernetes in <strong>DEBUG</strong> mode and generated about 1 Terabytes of logs in a day. But the short answer is nothing stops this problem out the box.</p><h3>Controlling the logs, somewhat…</h3><p>Luckily for us Docker has a couple of setting we can adjust to keep excessive logging under control. Here is an example of some Docker settings we use in our clusters</p><pre>$ cat /etc/docker/daemon.json</pre><pre>{<br>&quot;live-restore&quot;: true,<br>&quot;storage-driver&quot;: &quot;overlay2&quot;,<br>&quot;exec-opts&quot;: [&quot;native.cgroupdriver=systemd&quot;],<br>&quot;storage-opt&quot;:[&quot;overlay2.override_kernel_check=true&quot;,&quot;overlay2.size=10G&quot;],<br>&quot;log-driver&quot;: &quot;json-file&quot;,<br>&quot;log-level&quot;: &quot;info&quot;,<br>&quot;log-opts&quot;: {<br>  &quot;cache-disabled&quot;: &quot;false&quot;,<br>  &quot;cache-max-file&quot;: &quot;5&quot;,<br>  &quot;cache-max-size&quot;: &quot;20m&quot;,<br>  &quot;cache-compress&quot;: &quot;true&quot;,<br>  &quot;env&quot;: &quot;os,customer&quot;,<br><strong>  &quot;max-file&quot;: &quot;5&quot;,<br>  &quot;max-size&quot;: &quot;100m&quot;</strong><br>},<br>&quot;fixed-cidr&quot;: &quot;192.168.240.1/25&quot;,<br>&quot;bip&quot;: &quot;192.168.240.1/24&quot;<br>}</pre><p>The section highlighted in bold shows that Docker will make sure a log file does not use more than 100M and if it does, it will rotate it to a maximum of 5 log files. Meaning at no point can a single pod use more that 500M of logs.</p><p>Remember our pod we looked at earlier? This guy: vitality-programme-business-service-16-psm84</p><p>Did he use more than 100M?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*9b4OKtSwFAFG9OrpobgvZg.jpeg" /><figcaption>When you find out it was using more than 15M!</figcaption></figure><p>Let’s actually change directory (this text wrapping is the worst)</p><pre>$ cd /var/lib/docker/containers/d28698dcd8830663d2980c632d3fc36786b6b57d0841092da3f2e6f4d2552177</pre><pre>$ ls -1sh<br>total 397M<br> 16M d28698dcd8830663d2980c632d3fc36786b6b57d0841092da3f2e6f4d2552177-json.log<br> 96M d28698dcd8830663d2980c632d3fc36786b6b57d0841092da3f2e6f4d2552177-json.log.1<br> 96M d28698dcd8830663d2980c632d3fc36786b6b57d0841092da3f2e6f4d2552177-json.log.2<br> 96M d28698dcd8830663d2980c632d3fc36786b6b57d0841092da3f2e6f4d2552177-json.log.3<br> 96M d28698dcd8830663d2980c632d3fc36786b6b57d0841092da3f2e6f4d2552177-json.log.4</pre><p>I knew it! That sneaky pod has been using way more than 15M!!! We can see that it’s logs have been rotated to the max! If we had not had that Docker setting those logs would keep on growing and growing, filling up /var/lib/docker and then hitting on 85% threshold and then we get Disk Pressures on the Node.</p><p>So problem solved? Time to get a beer? Time to relax because we have solved our issue? I mean, what’s the big pain about local storage…</p><h3>Little Did He Know</h3><p>Now our story really begins (feels like a <a href="https://www.imdb.com/title/tt0110912/">Quentin Tarantino</a> film) because this is what actually happened. Our support team kept getting these alerts at random times.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*FBHMEYdlY-4vYwkrSeT-yg.png" /><figcaption>Prometheus and Alertmanger FTW</figcaption></figure><p>The team would log in and move pods around to balance memory, but this was becoming more and more frequent, so we want to get a better understanding. Enter <a href="https://grafana.com/">Grafana</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ofR4zP0a3jQflprc6b8_kQ.png" /><figcaption>All is fine until it isn’t</figcaption></figure><p>To the keen observer we can see that all is well until 23:30, when suddenly vgqengapps03 goes from 81% total memory utilization to 9% total utilization. This is the wonderful concepts of self healing and node evictions coming to bite us in the ass.</p><p>After vgqengapps03 suddenly has all this memory available, we can see vgqengapps04 jumps up from 57% to above 90%, which lasts until 8:00am and then pattern repeats itself.</p><p>Here is the <a href="https://en.wikipedia.org/wiki/Catch-22">Catch-22</a> of Kubernetes and the nature of self healing:</p><ul><li>Node goes Above 90% of Memory Usage</li><li>Alert is generated</li><li>Kubernetes evicts pod</li><li>Error clears!</li><li>All is fine again</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*8jBZ7Putxz-1AL4jTPNEUw.jpeg" /><figcaption>Kubernetes! The choice for a Gru generation</figcaption></figure><p>Because of the large mount of time between alerts not much attention was paid to this problem and because it was actually self healing every 12 hours. After a few days we decide to check out why suddenly we had a these huge memory spikes.</p><h3>Enter the DiskPressure!</h3><p>This is where it become a bit weird, because we were only getting alerts on memory and nothing else. So of course we assumed it was a memory issue. Some pod was eating all the memory? But thanks to <a href="https://kubernetes.io/docs/concepts/policy/limit-range/">LimitRanges</a>, which we implemented on all namespaces, we knew there couldn’t be a single pod eating up all the memory.</p><p>We also knew Docker had been configured to prevent a single pod from eating up all the storage and we hadn’t received a storage alert, so why on earth would we even look at storage?</p><blockquote>Also, when it comes to storage you normally do not alert when a threshold is breached immediately as you get so many false positives. We normally only alert if a threshold has been breached for over an hour.</blockquote><p>Nevertheless we brought up our Filesystem Utilization graph and …. what the hell!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F9FLFPuRNOKsDwvY5Ijahg.png" /><figcaption>See our alert is set to 85%. Yeah…</figcaption></figure><p>Ok, we have some correlation here. Why is the disk increasing like crazy! Why would that cause a node eviction…oh yeah…Node-pressure Eviction! Remember that old chest nut from the start of this story.</p><h3>In Space, No one can hear you Clean</h3><p>This makes no sense. We can see from Grafana that it is /var/lib/docker, let’s go have a look at what’s happening here</p><pre>$ ssh vgqengapps01<br>$ sudo -s<br>$ df -h /var/lib/docker</pre><pre>Filesystem                       Size  Used Avail Use% Mounted on</pre><pre>/dev/mapper/docker-docker--pool  100G   71G   30G  71% /var/lib/docker</pre><p>Ok, we have 30G available. So what is using the storage?</p><pre>$ du -hs /var/lib/docker/* | sort -h</pre><pre>0 /var/lib/docker/plugins<br>0 /var/lib/docker/swarm<br>0 /var/lib/docker/tmp<br>0 /var/lib/docker/trust<br>180K /var/lib/docker/network<br>26M /var/lib/docker/image<br>203M /var/lib/docker/volumes<br>5.3G /var/lib/docker/containers<br>36G /var/lib/docker/overlay2</pre><p>Huh!?! Ok, I did not get an A+ for mathematics but 36G+5.3G does not equal 71G. Luckily I come from a Linux background and I have seen this gremlin before and I have a clue what it could be.</p><blockquote>Hey Linux! What files do you have that are Open? Get it… Open! This guy…</blockquote><pre>$ lsof /var/lib/docker/ &gt; /tmp/lsof-docker.txt<br>$ cat /tmp/lsof-docker.txt | grep deleted | head</pre><p>Check out what popped up</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RduR1CCS7-QwYUvJen7i6A.png" /><figcaption>Hello filebeat!</figcaption></figure><p>Ok! Now we are getting somewhere. Those files look veeeeery familiar. Remember our log files from earlier. Remember when I took all that time to explain stuff to you! It’s all coming together now, isn’t it.</p><blockquote>For an explanation from a smarter person as to why deleted files are taking up space, go <a href="https://unix.stackexchange.com/questions/182077/best-way-to-free-disk-space-from-deleted-files-that-are-held-open/182082">here</a></blockquote><p>Let’s use some other fancy commands quickly. I want to see all the processes that are still holding on files.</p><p>This command will group, by the first column, all the processes holding onto files</p><pre>$ cat /tmp/lsof-docker.txt | grep deleted | awk  &#39;NR&gt;1{arr[$1]++}END{for (a in arr) print a, arr[a]}&#39;</pre><pre>filebeat 575</pre><p>That tells us that the process “filebeat” is holding onto 575 files.</p><p>And how much storage are all these deleted files taking up? Run this command to add up all the numbers in column 7</p><pre>$ cat /tmp/lsof-docker.txt | grep deleted | awk &#39;{print $7}&#39; | awk &#39;{ sum += $1 } END { print sum }&#39;</pre><pre>57600322165</pre><p>57600322165 bits. Great, what is that in human speak?</p><pre>$ echo &quot;57600322165/(1024*1024*1024)&quot; | bc<br>53 </pre><p>Wow! Those files are taking up 53G. And it’s all by the same process… <strong>filebeat</strong></p><h3>Turn the Beat Around</h3><p>At <a href="https://lsdopen.io/">LSD</a>, we use the <a href="https://www.elastic.co/guide/en/cloud-on-k8s/master/index.html">Elastic Cloud on Kubernetes</a> (ECK) to centrally gather all logs because remember we rotate log files from pods. Did anyone of you think to ask, what happens to those old logs once we use 501Mb of logging? Ironically they don’t get deleted (thanks Filebeat), jokes aside they are in fact deleted but they are also collected by the ECK and ingested into a central Elasticsearch instance that we use for log retention and analytics.</p><p>ECK is made of up 3 major components:</p><ul><li>Elasticsearch — Time series data store</li><li>Kibana — Visualization tool for Elasticsearch</li><li>Filebeat — Log collection</li></ul><p>Aaaah this is making sense now.</p><p>Remember these files (lots of remembering in this story)</p><pre>$ cd /var/lib/docker/containers/d28698dcd8830663d2980c632d3fc36786b6b57d0841092da3f2e6f4d2552177</pre><pre>$ ls -1sh<br>total 397M<br> 16M d28698dcd8830663d2980c632d3fc36786b6b57d0841092da3f2e6f4d2552177-json.log<br> 96M d28698dcd8830663d2980c632d3fc36786b6b57d0841092da3f2e6f4d2552177-json.log.1<br> 96M d28698dcd8830663d2980c632d3fc36786b6b57d0841092da3f2e6f4d2552177-json.log.2<br> 96M d28698dcd8830663d2980c632d3fc36786b6b57d0841092da3f2e6f4d2552177-json.log.3<br> 96M d28698dcd8830663d2980c632d3fc36786b6b57d0841092da3f2e6f4d2552177-json.log.4</pre><p>When those files are created by Docker, Filebeat picks them up, locks onto them and ingests them into Elasticsearch. But when Docker rotates the file, deleting it, Filebeat holds onto it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/1*VH87m-XAw2Clv72q0S0Ddg.jpeg" /></figure><p>A quick solution to this exact problem was simply to delete Filebeat on that affected node: vgqengapps01</p><pre>$ kubectl get po --all-namespaces -owide | grep filebeat | grep vgqengapps01</pre><pre>lsdobserve                          lsdobserve-filebeat-beat-filebeat-6gjtd                   1/1       Running     0          3d        172.26.23.203   vgqengapps01   &lt;none&gt;</pre><pre>$ kubectl delete po lsdobserve-filebeat-beat-filebeat-6gjtd -n lsdobserve</pre><p>Before I deleted the pod</p><pre>$ df -h /var/lib/docker</pre><pre>Filesystem                       Size  Used Avail Use% Mounted on<br>/dev/mapper/docker-docker--pool  100G   71G   30G  71% /var/lib/docker</pre><p>After I deleted the pod</p><pre>$ df -h /var/lib/docker</pre><pre>Filesystem                       Size  Used Avail Use% Mounted on<br>/dev/mapper/docker-docker--pool  100G   17G   84G  17% /var/lib/docker</pre><p>That is a difference of… you guessed it! 53G or 57600322165 bits</p><h3>But wait, all is well now?</h3><p>The point of this article wasn’t to pick on Filebeat. It was to highlight that many times we have open files that are not cleared up and how to find the process that is locking onto it. And that when you run out of storage on key mount points ( / &amp; /var/lib/docker) you get DiskPressure which can have adverse affects on your cluster.</p><p><a href="https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html#filebeat-input-log-close-inactive">ECK does have a fix for this by the way</a></p><p>But yes, all is not well. It’s OK, but not super well. Remember this graph?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F9FLFPuRNOKsDwvY5Ijahg.png" /><figcaption>The crazy disk usage.</figcaption></figure><p>Turns out someone enabled <strong>DEBUG</strong> on their applications which was generating 16G of logs an hour. We will not mention names, but if you’re reading this, you know who you are!</p><p>How did we find that?</p><p>Thank you to <a href="https://www.elastic.co/kibana/">Kibana</a> and thanks to me for thinking about using ECK (I take all credit for this), I set the time and date to those spikes we see above and said show me the top 10 pods with the most logs, which revealed…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LsUZW-QOowDtho4kbXKgqg.png" /><figcaption>You dirty rat!</figcaption></figure><p>A pod called “vitality-error-management-service” generated 13 million (yes that’s million with a “mill”) logs in 8 hours, using 90% of all logging during that time.</p><p>Remember I said earlier we had resources requests set, here are the resources from that pod</p><pre>Limits:<br>  cpu:     600m<br>  memory:  512Mi<br>Requests:<br>  cpu:     50m<br>  memory:  512Mi</pre><p>Imagine if I had given it more CPU.</p><p>Armed with that information we can now go to application team and ask them to please switch off Debugging as it is generating 16G of logs an hour, or 384G of logs a day or 11Tb of logs a month :)</p><h3>Are we living in a simulation?</h3><p>Whenever something like this happens above, I personally would like to know the details of this situation. For example when my disk fill ups, how much time do I have to recover? How long does it take to evict pods? Does it try to clean up? How many times. So just for you guys that stuck around to the end I wanted to run through a simulation of the above scenario.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vOElGZ5C_x-_qME0l6ZbOA.jpeg" /></figure><p>Let’s fill up a filesystem</p><pre>$ df -h /var/lib/docker</pre><pre>Filesystem                       Size  Used Avail Use% Mounted on<br>/dev/mapper/docker-docker--pool  100G   20G   81G  20% /var/lib/docker</pre><p>Commence with the filling…</p><pre>$ dd if=/dev/zero of=/var/lib/docker/70G-tmp bs=1G count=70</pre><pre>70+0 records in<br>70+0 records out<br>75161927680 bytes (75 GB) copied, 196.061 s, 383 MB/s</pre><p>While I was doing that, in another sessions I was following the disk usage</p><pre>$ df -h /var/lib/docker</pre><pre>Filesystem                       Size  Used Avail Use% Mounted on<br>/dev/mapper/docker-docker--pool  100G   64G   37G  64% /var/lib/docker</pre><pre>$ df -h /var/lib/docker</pre><pre>Filesystem                       Size  Used Avail Use% Mounted on<br>/dev/mapper/docker-docker--pool  100G   94G  6.6G  94% /var/lib/docker</pre><pre>$ df -h /var/lib/docker</pre><pre>Filesystem                       Size  Used Avail Use% Mounted on<br>/dev/mapper/docker-docker--pool  100G   88G   13G  88% /var/lib/docker</pre><pre>$ df -h /var/lib/docker</pre><pre>Filesystem                       Size  Used Avail Use% Mounted on<br>/dev/mapper/docker-docker--pool  100G   84G   16G  85% /var/lib/docker</pre><p>Check it out! We filled it up to 94% but then it suddenly dropped to 84%. Remember (yeah I am not done with that word) earlier we said that Kubernetes tries to clear up some storage and heal from this situation. Here we can see it doing exactly that. And you can get further details by describing the node</p><pre>$ kubectl describe no vgqengapps01</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tAgTwv0hY2uNCDa7srbASw.png" /></figure><p>It did in fact clear up some storage, but the “dd” command reclaimed it so quickly that it thinks it didn’t.</p><p>You can also see that the the status of <strong>DiskPressure</strong> is now <strong>True</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fdVoY44OVf7x3jZa43wc_A.png" /></figure><p>In my case, after a several attempts at trying to Free Disk Space and failing we eventually get an “EvictionthresholdMet”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dsD4Okyrl6_BPUSnUt_M3w.png" /></figure><p>This will evict all pods from that node, which will look as follows</p><pre>$ kubectl get po  -owide | grep vitality-error-management-service</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bPD42cle5GRS78xz2e1z_w.png" /></figure><p>And you can see exactly why that pod was evicted</p><pre>$ kubectl describe po vitality-error-management-service-0-6-5mhgh</pre><p>The 2 sections that are important are the Status</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/1*xKp7xmEOnDIOWgGHDIB2YQ.png" /></figure><p>And the Events</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VZ02b7EjwrzJTu0YoMgDtA.png" /></figure><p>And folks there you have it. Never again will you think about pod logs again. Never again will sleep soundly at night, knowing that a developer can ruin your day by simply putting his (her…) application into <strong>DEBUG</strong> mode and then turning their <strong>phone off!</strong></p><h3>And in closing</h3><p>I wanted to highlight in this article how pod logs are something people do not care about and how it can have a very negative affect on your cluster. I also took the time an effort to explain the “open file” problem because finding the bad pod was half the problem, the easy half. Understanding why you have less and less filesystem storage available even when logs are being rotated and even when the command “du” is showing you everything is fine, can be tricky to wrap your head around.</p><p>I hope this was useful to someone, and maybe next time I will write another article on how local storage in a pod (not logging storage) can also make your life a painful. What is stopping someone from creating a 100G file with the “dd” command inside a container and then eating up all your ephemeral storage? I will leave you with that thought while you go to sleep tonight…</p><h3>What is LSD?</h3><p>If you saw the words “LSD” and were curious…well…</p><p><a href="https://lsdopen.io/">LSD</a> is an open source focused company that helps companies along their journey into the cloud, or what we like to call the <a href="https://trip.lsdopen.io/">LSDtrip</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=34b5ba4284b6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Partial etcd Recovery on Openshift / Kubernetes]]></title>
            <link>https://neilcameronwhite.medium.com/partial-etcd-recovery-on-openshift-kubernetes-7909da28867c?source=rss-807665507843------2</link>
            <guid isPermaLink="false">https://medium.com/p/7909da28867c</guid>
            <category><![CDATA[etcd]]></category>
            <category><![CDATA[backup]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[restore]]></category>
            <dc:creator><![CDATA[Neil White]]></dc:creator>
            <pubDate>Mon, 14 Jun 2021 07:44:24 GMT</pubDate>
            <atom:updated>2021-06-14T07:45:17.872Z</atom:updated>
            <content:encoded><![CDATA[<p>For those out there, playing in the land of Kubernetes / Openshift, we know about etcd and how it’s the backend that stores the data for Kubernetes. We also know it’s important and that is why most Kubernetes installation docs will say “install 3 etcd replicas” so that if one fails you are fine.</p><p>etcd is also super lightweight, meaning you can back up clusters in a minute with a small snapshot file.</p><p>Here is an example of an etcd snapshot size of a cluster of 16 nodes running 662 pods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*iDbx7ezV_sfHZxeld40Z-g.png" /><figcaption>hostnames have been removed to protect the innocent.</figcaption></figure><p>About 300Mb for a daily backup and 2.3Gb for 8 days worth of backups is nothing these days. The backups are also very quick. In some clusters we backup 4 times a day because the sizes are so small and the backup/etcd snapshotting is so quick.</p><p>So etcd is amazing and quick and light and highly available, what is not to love. Well, there is one small thing that is a bit of a problem for us. When you do an etcd restore it’s all or nothing. These snapshots are designed to restore a cluster that has had a total failure. But I live in the real world, where I do not have clusters with total failures. I work with users. Users who work on large clusters with their own namespaces/projects who end up deleting their own stuff and then asking me to do a restore. Here is a typical conversation.</p><p>User: I deleted all my stuff, can you put it back.<br>Me: Did you make a backup?<br>User: No, don’t you have backups?<br>Me: Yes, Thank God for Me.<br>User: Ok, just restore my stuff!<br>Me: It doesn’t work that way, I have to restore the entire cluster which will affect everybody else.<br>User: Ok, let me know when you have restored my stuff.<br>Me: No, it doesn’t work that way!<br>User: I’m off to get a coffee, bye.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FeIGPvPRzEaQ%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DeIGPvPRzEaQ&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FeIGPvPRzEaQ%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/6dcbf4da1a9310822e5ec7b7432cc89a/href">https://medium.com/media/6dcbf4da1a9310822e5ec7b7432cc89a/href</a></iframe><h3>The Truth Please! What actually happened?</h3><p>Ok, you got me. In this case I was the user. I deleted some pretty important stuff in a pretty important namespace/project which caused some pretty nasty things to happen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7xPfd3jBGd_yrg3x9YG8Uw.png" /><figcaption>Much mem, such CPU</figcaption></figure><p>The cluster in question is a 49 Node cluster with 205 projects running 1715 pods. We have lots of teams on this cluster, making changes all the time, so it’s very active and runs business critical applications. I know everybody thinks their app is important and business critical, but these are actually important because they make money (businesses loves themselves some money).</p><p>I am not going to go into the details of why I did what I did, or what I was thinking, this is not important and embarrassing. What I want to do is celebrate the important lesson I learnt, which is when you break something, find someone smarter than you and can get them to save your arse. At least that way you can also blame the other person.</p><p>Seriously though, I am happy this happened because it taught me something new about etcd &amp; Kubernetes / Openshift that I want to share with everybody because I am certain someone else out there is going to face a similar situation.</p><p>So what happened? Something like this…</p><pre>$ oc delete secrets --all -n openshift-infra</pre><p>There are secrets in the openshift-infra projects? Yeah…. and there are kinda of a lot…..</p><pre>$ oc get secrets -n openshift-infra | wc -l<br>168</pre><p>I deleted 168 secrets that are pretty important… But let’s not dwell on this for now, let’s move forward!</p><p>The situation we (remember I have a buddy now ) found ourselves in, is that we need to restore a bunch of secrets from a specific namespace without impacting any of the users in the cluster. And this is how we ended up saving ourselves from ourselves without anybody even noticing.</p><h3>Get On With it!</h3><p>The goal was to get only the information I wanted, out of etcd, into a usable YAML format, where I could simply add it back to the cluster.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FsXE8LdXzeHM%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DsXE8LdXzeHM&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FsXE8LdXzeHM%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/527d7da6c5ef7f0a0a1ec2b14b3c4f58/href">https://medium.com/media/527d7da6c5ef7f0a0a1ec2b14b3c4f58/href</a></iframe><p>I am now gonna show you how we recovered from this, which is a procedure you can use when you want to only restore a specific section of your Kubernetes cluster</p><p><strong>Step 1: Get the backups!</strong></p><p>I downloaded the last nightly backup to my local machine</p><pre>scp master01-from-important-client:/var/lib/etcd/snapshot.db /tmp/<br>snapshot.db                           100%  166MB   3.6MB/s   00:45</pre><p><strong>Step 2: Install etcd locally</strong></p><p>I run Fedora. I also didn’t care that the etcd version were not identical. This will make sense later.</p><pre>$ dnf install etcd</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x5UjUxmu_eTgV_JZLG8g1w.png" /><figcaption>dnf install etcd (i am doing screenshots because the code paste just looks ugly)</figcaption></figure><p><strong>Step 3: Restore the snapshot to my local PC</strong></p><pre>$ etcdctl snapshot restore /tmp/snapshot.db </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_JJzljgGanU6YedXiPrdCA.png" /><figcaption>Proof I am not lying</figcaption></figure><p>It should have restored the name as “default”.</p><p>You can verify this by looking at the default etcd location</p><pre>$ ls -la /var/lib/etcd/<br>total 4<br>drwxr-xr-x.  3 etcd etcd   26 Jun 11 13:41 .<br>drwxr-xr-x. 70 root root 4096 Jun 11 11:08 ..<br>drwx------.  3 etcd etcd   20 Jun 11 13:41 default.etcd</pre><p>You should see “default.etcd” If you see a different name, then when you start up etcd change “ — name default” to whatever you see.</p><p><strong>Step 4: Start up etcd</strong></p><p>This command will start up the etcd named “default”</p><pre>$ etcd --name default --listen-client-urls <a href="http://localhost:2379/">http://localhost:2379</a>  --advertise-client-urls <a href="http://localhost:2379/">http://localhost:2379</a> --listen-peer-urls <a href="http://localhost:2380/">http://localhost:2380</a></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Gvmf0WXJeEfhtqG-uw9Lg.png" /><figcaption>etcd up!</figcaption></figure><p>This will run etcd in the foreground. Leave it there and start up another terminal session</p><p><strong>Step 5: Confirm snapshot actually worked</strong></p><p>Now I have all the data from my client running on my laptop, thanks to the simple snapshot capability of etcd</p><p>Let’s see what’s inside.</p><pre>$ etcdctl get / --prefix --keys-only</pre><p>You should see lots and lots of keys</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/919/1*7ugh8azI-mV-UTOC4Om7KQ.png" /></figure><p>This is how many I see</p><pre>$ etcdctl get / --prefix --keys-only | wc -l<br>12932</pre><p>I am interested in is everything in the namespace/project “openshift-infra”</p><p>Let’s see if there is data for that namespace/project</p><pre>$ etcdctl get / --prefix --keys-only  | grep openshift-infra</pre><pre>/kubernetes.io/controllers/openshift-infra/hawkular-cassandra-1<br>/kubernetes.io/controllers/openshift-infra/hawkular-metrics<br>/kubernetes.io/controllers/openshift-infra/heapster<br>/kubernetes.io/jobs/openshift-infra/hawkular-metrics-schema<br>/kubernetes.io/namespaces/openshift-infra<br>/kubernetes.io/pods/openshift-infra/hawkular-cassandra-1-7ftn8<br>/kubernetes.io/pods/openshift-infra/hawkular-metrics-5sckh<br>/kubernetes.io/pods/openshift-infra/heapster-d9gwb<br>/kubernetes.io/rolebindings/openshift-infra/admin<br>/kubernetes.io/rolebindings/openshift-infra/edit<br>/kubernetes.io/rolebindings/openshift-infra/hawkular-view<br>/kubernetes.io/rolebindings/openshift-infra/system:deployer<br>/kubernetes.io/rolebindings/openshift-infra/system:deployers<br>/kubernetes.io/rolebindings/openshift-infra/system:image-builder<br>/kubernetes.io/rolebindings/openshift-infra/system:image-builders<br>/kubernetes.io/rolebindings/openshift-infra/system:image-puller<br>/kubernetes.io/rolebindings/openshift-infra/system:image-pullers<br>/kubernetes.io/secrets/openshift-infra/build-controller-token-hk9gz<br>/kubernetes.io/secrets/openshift-infra/build-controller-token-xv2gt<br>/kubernetes.io/secrets/openshift-infra/builder-dockercfg-qtt2c<br>/kubernetes.io/secrets/openshift-infra/builder-token-dx9rb<br>/kubernetes.io/secrets/openshift-infra/builder-token-pvss7<br>/kubernetes.io/secrets/openshift-infra/cassandra-dockercfg-hqcgd<br>/kubernetes.io/secrets/openshift-infra/cassandra-dockercfg-jcjm9<br>/kubernetes.io/secrets/openshift-infra/cassandra-dockercfg-wlhcf<br>/kubernetes.io/secrets/openshift-infra/cassandra-token-27m8v<br>/kubernetes.io/secrets/openshift-infra/cassandra-token-khzqx<br>/kubernetes.io/secrets/openshift-infra/cassandra-token-qq4lh<br>/kubernetes.io/secrets/openshift-infra/cassandra-token-v4zg7<br>/kubernetes.io/secrets/openshift-infra/cassandra-token-vt4kn<br>/kubernetes.io/secrets/openshift-infra/cassandra-token-wgczj<br>...</pre><p>There we go baby! Some sweet sweet data.</p><p><strong>Step 6: Get the exact data we want</strong></p><p>Now that we can see our data, I want to only extract exactly what we need. In my case it is the secrets in the namespace/project “openshift-infra”. You might need persistent volumes, or config maps, it doesn’t matter the resource as the procedure is the exact same.</p><p>In this example I am going to pick on one secret to demonstrate how we do this: the default secret</p><p>The default secret is part of every single namespace that gets created in Kubernetes.</p><pre>$ etcdctl get / --prefix --keys-only  | grep openshift-infra | grep secret | grep default</pre><pre>/kubernetes.io/secrets/openshift-infra/default-dockercfg-5w5dq<br>/kubernetes.io/secrets/openshift-infra/default-rolebindings-controller-dockercfg-xm6k8<br>/kubernetes.io/secrets/openshift-infra/default-rolebindings-controller-token-9lpq9<br>/kubernetes.io/secrets/openshift-infra/default-rolebindings-controller-token-pn2n4<br>/kubernetes.io/secrets/openshift-infra/default-token-5np58<br>/kubernetes.io/secrets/openshift-infra/default-token-7c7tz</pre><p>We are going to inspect: /kubernetes.io/secrets/openshift-infra/default-token-7c7tz</p><pre>$ etcdctl get /kubernetes.io/secrets/openshift-infra/default-token-7c7tz</pre><p>You should see something like this (this is from a test cluster, so these certificates you see are useless, in case you are concerned for me)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*1WMUZHb4XmdZNTGHR2MetQ.png" /></figure><p>Alright then! We can see data!! But… I wish it was that simple…</p><p>See those pesky Unicode symbolism…Yeah, they are a problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/94/1*wY-VXeV0OrxTWGnLilK-_A.png" /></figure><p>What we need is that data, without the stupid question marks, and in a YAML format.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/546/1*0G3KnillCyuDgj5RX28uPw.jpeg" /></figure><p><strong>Step 7: Why Hello </strong><a href="https://github.com/jpbetz/auger"><strong>Auger</strong></a></p><p>I say this all the time, I am so happy there are smart generous people in this world, that have created such magnificent tools like <a href="https://github.com/jpbetz/auger">Auger</a></p><p>Auger will allow us to decode the data in a format we can use, check it out!</p><pre>$ etcdctl get /kubernetes.io/secrets/openshift-infra/default-token-7c7tz  --print-value-only | auger decode</pre><p>And we get the beautiful, usable, YAML that we love so much.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*foDe_TrhmS1xgsKRa0XZFQ.png" /><figcaption>Mmmmm, YAML….</figcaption></figure><p><strong>Step 8: Rinse and Repeat</strong></p><p>Now I can use a simple for loop to get my lovely data</p><pre>$ mkdir -p /tmp/yaml ; cd /tmp/yaml<br>$ etcdctl get / --prefix --keys-only | grep openshift-infra | grep &quot;/secrets/&quot; &gt; secrets.list</pre><p>The contents of that file</p><pre>$ cat secrets.list</pre><pre>/kubernetes.io/secrets/openshift-infra/cassandra-token-27m8v<br>/kubernetes.io/secrets/openshift-infra/cassandra-token-khzqx<br>/kubernetes.io/secrets/openshift-infra/cassandra-token-qq4lh<br>/kubernetes.io/secrets/openshift-infra/cassandra-token-v4zg7<br>/kubernetes.io/secrets/openshift-infra/cassandra-token-vt4kn<br>/kubernetes.io/secrets/openshift-infra/cassandra-token-wgczj<br>/kubernetes.io/secrets/openshift-infra/default-dockercfg-5w5dq<br>/kubernetes.io/secrets/openshift-infra/default-token-5np58<br>/kubernetes.io/secrets/openshift-infra/default-token-7c7tz<br>/kubernetes.io/secrets/openshift-infra/deployer-dockercfg-tdjnz<br>/kubernetes.io/secrets/openshift-infra/deployer-token-6jc8c<br>/kubernetes.io/secrets/openshift-infra/deployer-token-8hp6c<br>/kubernetes.io/secrets/openshift-infra/deployment-trigger-<br>...</pre><p>Then we loop through that and get all our secrets</p><pre>$ for x in $(cat secrets.list) ; do secret_name=$(echo &quot;$x&quot; | awk -F\/ &#39;{print $5}&#39;) ; echo $secret_name ; etcdctl get $x  --print-value-only  | auger decode &gt; $secret_name.secret.yaml ; done</pre><p>And now we have all our secrets for “openshift-infra”</p><pre>$ ls -1<br>build-config-change-controller-dockercfg-9khqd.secret.yaml<br>build-config-change-controller-token-2csmb.secret.yaml<br>build-config-change-controller-token-h2wxd.secret.yaml<br>build-controller-dockercfg-9chcf.secret.yaml<br>build-controller-token-hk9gz.secret.yaml<br>build-controller-token-xv2gt.secret.yaml<br>builder-dockercfg-qtt2c.secret.yaml<br>builder-token-dx9rb.secret.yaml<br>builder-token-pvss7.secret.yaml<br>cassandra-dockercfg-hqcgd.secret.yaml<br>cassandra-dockercfg-jcjm9.secret.yaml<br>cassandra-dockercfg-wlhcf.secret.yaml<br>...</pre><p>Check it out! We have glorious data!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5vFj45cy_T-_27qF8c6PdA.png" /></figure><p><strong>Step 9: Import</strong></p><p>I suppose it obvious now that we have the data, we simply apply it</p><pre>$ for x in $(ls -1 *.yaml ) ; do oc apply -f $x ; done</pre><p><strong>Step 10: Openshift v3 Extra Steps</strong></p><p>You might be asking why were any of these secrets important in “openshift-infra”. How we discovered they were important was when no new builds were working or new deployment configs were rolling out.</p><p>For example these commands where executing, but nothing was actually happening</p><pre>$ oc rollout latest deploymentconfig/libretransate<br>$ oc start-build -n multisite-routing-prod multisite-routing</pre><p>After reading through 1000s of logs and determining it was the secrets we removed, we had to do an additional step to revive the cluster, and that was to restart the controllers and api on all the masters</p><p>So for you Openshift v3 people out there, log into all your masters and run the following command to wake up the cluster</p><pre>$ /usr/local/bin/master-restart api<br>$ /usr/local/bin/master-restart controllers</pre><h3>Closing Thoughts</h3><p>I hope this helps someone out there that might have a similar situation and that is why I take the time out to write these articles. I would rather be installing a better solution to these types of problems, but I don’t want people to panic and get grey hairs when there is always hope.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*ojPEb8KnEKv0rJu1lVg-Dg.jpeg" /></figure><p>The correct solution to this problem would have a better backup tool than relying solely on etcd. And a better backup tool I can recommend is <a href="https://velero.io/">Velero</a>. We will start implementing it at LSD on all our clients that use <a href="https://trip.lsdopen.io/lsdcontainer/">LSDcontainer</a>, to help us in times of crisis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*_cDUS1Bfczs0fuHEfZbwIQ.jpeg" /><figcaption>I am not a Velero employee, but I do think it’s neat.</figcaption></figure><p>And of course a huge thank you to everybody that helped build <a href="https://github.com/jpbetz/auger">Auger</a>. Check it out at: <a href="https://github.com/jpbetz/auger">https://github.com/jpbetz/auger</a> if you haven’t already clicked on any of the hyperlinks I posted about it. Auger really saved my butt and hopefully can save other butts out there.</p><h3>What is LSD?</h3><p>If you saw the words “LSD” and were curious….well….</p><p><a href="https://lsd.co.za/">LSD</a> is an open source focused company that helps companies along their journey into the cloud, or what we like to call the <a href="https://trip.lsdopen.io/">LSD Trip</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7909da28867c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ThunderStruck! When Clouds Scare the hell out of you.]]></title>
            <link>https://neilcameronwhite.medium.com/thunderstruck-when-clouds-scare-the-hell-out-of-you-6fd526f23be4?source=rss-807665507843------2</link>
            <guid isPermaLink="false">https://medium.com/p/6fd526f23be4</guid>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[argo]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Neil White]]></dc:creator>
            <pubDate>Thu, 25 Mar 2021 18:48:28 GMT</pubDate>
            <atom:updated>2021-03-26T11:20:45.179Z</atom:updated>
            <content:encoded><![CDATA[<h3>ThunderStruck! When Clouds scare the hell out of you.</h3><h3>Disclaimer</h3><p>Before I begin I want to state that this is not an article to throw anybody under the bus, or to blame any company. We are partners with all the major cloud providers, all the <a href="https://i.pinimg.com/originals/60/f2/e5/60f2e5619311c357f9149bd9236a1dd7.jpg">major</a> Open Source vendors and we value those partnerships. This is more a dark tale of horror and fear that could happen to each and every one of you, so beware and heed our warning…</p><h3>On with the show</h3><p>At <a href="https://lsdopen.io/">LSD</a>, where I work, one of the <a href="https://wiki.lsd.co.za/en/public/LSD/values">core values</a> is freedom (and openness, that one is pretty cool too). Because <a href="https://lsdopen.io/">LSD</a> is an open source technology company, with a core value of freedom, the idea of cryptocurrency to us is a match made in heaven. I am not going to delve into cryptocurrency, as there are so many amazing articles about it, but I simply want to say that the idea of <a href="https://time.com/5486673/bitcoin-venezuela-authoritarian/">economic freedom</a>, by using technology is something we strongly believe in. So when <a href="https://ethereum.org/en/eth2/staking/">Ethereum Staking</a> launched this was something we jumped into quickly and happily.</p><p>I am fortunate to work with people far smarter than me and a fun aspect of whenever we look at a new project is how to build it. We could build it quick and easy by deploying a virtual machine in the cloud somewhere and installing the <a href="https://docs.prylabs.network/docs/terminology#validator">Validator</a>. But why make it so simple when you can use <a href="https://kubernetes.io/">Kubernetes</a>, and containers, and <a href="https://argoproj.github.io/argo-cd/">ArgoCD</a>, and <a href="https://www.terraform.io/">Terraform</a>, and <a href="https://letsencrypt.org/">Let’s Encrypt</a> and… you get the picture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*SBIHeDBaYBU3NxmZ-px6gg.png" /></figure><p>But why make it way harder than it needs to be? At LSD we preach the gospel of Multi Cloud and Hybrid Cloud. We spread the good word of containers and portability, we shout from the mountain tops about GitOps and ArgoCD, of Terraform and Infrastructure as Code and of cabbages and kings. We believe that no-one should ever be locked into a technology because we believe in this freedom, which means you should also be portable, in the way you can move your tech whenever you feel like, regardless of the reason.</p><p>This freedom does come at a cost, a cost in energy, time and complexity. But as I said earlier, I work with a lot of smart people, so the complex stuff is their problem.</p><h3>Plans are worthless, but planning is everything</h3><p>Once we decided to start staking, we quickly agreed that where the validators run shouldn’t matter greatly because we should be able to move them if we wanted. So this was our plan:</p><ul><li>Terraform to provision the infrastructure</li><li>Kubernetes to manage our containers (we containerize everything)</li><li>Helm to manage our application deployment on Kubernetes</li><li>ArgoCD to manage our deployments within Kubernetes</li><li>Let’s Encrypt to make it all ingress secure and safe</li></ul><p>With these components we ticked the following boxes:</p><ul><li><strong>Infrastructure as Code<br></strong>No more asking people how stuff was deployed, we could look at code</li><li><strong>Portability</strong><br>Every cloud has a Kubernetes offering, so containers were not a problem</li><li><strong>GitOps</strong><br>All deployment information will be stored in Git as the source of truth</li><li><strong>Security</strong><br>All certificates will be managed by <a href="https://letsencrypt.org/">Let’s Encrypt</a>, you beautiful company you</li></ul><p>Happy with all those elements, we decided to start with Google Cloud because we have been using them from the very beginning and had a lot of infrastructure running there already.</p><p>Fast forward 2 months later and we have our own dedicated project on Google Cloud, running Kubernetes, built by Terraform, enforced by ArgoCD, secured by Let’s Encrypt, running our Ethereum Validators…when one morning…</p><p><em>while I nodded, nearly napping, suddenly there came a tapping,<br>As of someone gently rapping, rapping at my chamber door.</em></p><p>(it was an email, but there ain’t much poetry out there about email ok‽)</p><p>They say pictures are worth a 1000 words, so I will show you the email we received in a picture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*KevCXJ1BNsf0bOYuOXegRg.png" /><figcaption>You have been warned!</figcaption></figure><blockquote>Projects Names and Project IDs have been removed to protect the innocent.</blockquote><p>Now I want to make it clear, we were not using Google Cloud to do any crypto mining. I can only assume that would be an extremely expensive endeavor and something we really do not care for. We were not worried and clicked the magic appeal button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/1*h3eEYM_c5gula_srNp9aDA.png" /><figcaption>I’m innocent I tell you!</figcaption></figure><p>And like I said, I was not too worried. We got the warning, we clicked on appeal, we were not doing anything wrong and were happy to help get to the bottom of it. But then sh*t went sideways…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*qJbdJWU_-1fxavB_LXskOw.png" /><figcaption>Suspended!</figcaption></figure><p>Here is the timelines of events (all happening in less than 1 hour):</p><p>10:46am — Warning<br>11:02am — Appeal requested<br>11:39am — project suspended</p><p>It is at this point our story becomes a bit murky, where uncertainty creeps in and you start questioning your life decisions.</p><p>We have multiple projects on Google Cloud, so I want to make it clear to all two people reading this, that Google only suspended the project that was running our Ethereum Staking Validators. All our other projects were fine. But what does suspended mean?</p><p>We were learning about this in real time, but here is the TL;DR. <strong>You cannot do anything!</strong> You cannot access the Kubernetes Engine, you cannot get hold of your data, you cannot control any traffic. Simply put, anything in that project is dead to you. If you tried to go to the project you were redirected to a page that said the project was suspended.</p><blockquote>Come to think of it, we did not use this project for any DNS requests, so I have no idea how DNS would have been affected, i.e. would our DNS lookups work? But from what we experienced I would assume the suspension would have killed that ability as well, which would have been catastrophic for us. (Mental Note: Put DNS on it’s own project thereby making it safe from any overlord interference)</blockquote><p>And here is where the problems began. When you are Staking Ethereum you cannot have multiple Validators running, else you will be <a href="https://www.bloxstaking.com/blog/ethereum-2-0/understanding-eth2-slashing-preventative-measures/"><strong>slashed</strong></a>. So what could we do? We could wait for Google to unsuspend the project, or we could build it somewhere else.</p><p>But what if Google only unsuspends the project in a couple of days time, which means each passing hour we lose staking rewards for being down, which means we lose actual <a href="https://ethereum.org/en/eth/">Ether</a>.</p><p>Or what if we build a new project, get our Validators up and running again and then Google unsuspends the account and we get slashed for having double Validators? Which is even worse! Unless Google only unsuspends our accounts in a couple of weeks!! Which is even worse!!</p><blockquote>I tried calling Google to get some help on this, but didn’t have much success. Their on hold music is a Circus Music theme, which is not the best sound you want to hear when you are stressed out and looking for support. Just saying…</blockquote><p>And this was the heart of our problem. We were in a position whereby we could not make an accurate decision because we had never been in this situation before. I also understands that Google didn’t target us, but that they picked up an anomaly on our project which triggered an automatic response to suspend our project automatically while we sat there helpless.</p><p>And this is what we wanted to highlight — we were at the mercy of Google.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*mB9gAfqonGiCS_UXwei3CA.jpeg" /><figcaption>You’ve been……THUNDERSTRUCK!</figcaption></figure><h3><strong>Buy it, use it, break it, fix it, trash it, change it, mail — upgrade it</strong></h3><p>We put a clock on this issue, and decided that if the project was not unsuspended within 6 hours we would re-platform, which is what we ended up doing.</p><p>When the deadline arrived, we pulled the trigger!</p><pre>$ gcloud projects delete XXXXXXXX<a href="https://cloudresourcemanager.googleapis.com/v1/projects/openshift-258916">-258916</a><br>Your project will be deleted.</pre><pre>Do you want to continue (Y/n)?  y</pre><pre>Deleted [<a href="https://cloudresourcemanager.googleapis.com/v1/projects/openshift-258916">https://cloudresourcemanager.googleapis.com/v1/projects/</a>XXXXXXXX<a href="https://cloudresourcemanager.googleapis.com/v1/projects/openshift-258916">-258916</a>].</pre><pre>You can undo this operation for a limited period by running the command below.<br>    $ gcloud projects undelete XXXXXXXX<a href="https://cloudresourcemanager.googleapis.com/v1/projects/openshift-258916">-258916</a></pre><pre>See <a href="https://cloud.google.com/resource-manager/docs/creating-managing-projects">https://cloud.google.com/resource-manager/docs/creating-managing-projects</a> for information on shutting down projects.</pre><p>And this is now where our earlier decisions helped us out. We quickly decided to change clouds, going from Google to Amazon. We thought that if we simply rebuilt on Google we would be suspended again, so Amazon here we go!</p><h4>Terraform to the Rescue!</h4><p>When we built our original <a href="https://cloud.google.com/kubernetes-engine">Google GKE</a> cluster we only required 4 files for Terraform to work:</p><ul><li>backend.tf (we are storing our <a href="https://www.terraform.io/docs/language/state/index.html">state</a> in a google bucket)</li><li>variables.tf (defining our input variables)</li><li>terraform.tfvars (our actual variables)</li><li>main.tf (the actual file that will build our GKE cluster)</li></ul><p>And you want to see how simple they are?</p><p>backend.tf</p><pre>terraform {<br>  backend &quot;gcs&quot; {<br>    bucket = &quot;XXXXXXXX-terraform-state&quot;<br>    prefix = &quot;cluster-01&quot;<br>    credentials = &quot;XXXXXXXX-3d84f0b78bdc.json&quot;<br>  }<br>}</pre><p>variables.tf</p><pre>variable &quot;google_project_name&quot; {<br>  type = string<br>}</pre><pre>variable &quot;google_project_id&quot; {<br>  type = string<br>}</pre><pre>variable &quot;google_gke_cluster_name&quot; {<br>  type = string<br>}</pre><pre>variable &quot;google_gke_cluster_location&quot; {<br>  type = string<br>}</pre><pre>variable &quot;google_bucket&quot; {<br>  type = string<br>}</pre><pre># Sets the number of nodes in the pool. This appears to be multiplied by x3. Therefore if you set this to 1, you will get 3 nodes. Set it 2, you will get 6 nodes.<br>variable &quot;google_node_count&quot; {<br>  type = number<br>}</pre><p>terraform.tfvars</p><pre>google_project_name             = &quot;XXXXXXX&quot;<br>google_project_id               = &quot;XXXXXXX&quot;<br>google_gke_cluster_name         = &quot;cluster-01&quot;<br>google_gke_cluster_location     = &quot;europe-west4&quot;<br>google_bucket                   = &quot;XXXXXXX&quot;<br>google_node_count               = &quot;1&quot;</pre><p>main.tf (110 lines)</p><pre>provider &quot;google&quot; {<br>  credentials = file(&quot;XXXXXX-3d84f0b78bdc.json&quot;)<br>  project     = var.google_project_id<br>}</pre><pre>provider &quot;google-beta&quot; {<br>  credentials = file(&quot;XXXXXXX-3d84f0b78bdc.json&quot;)<br>  project     = var.google_project_id<br>}</pre><pre>resource &quot;google_container_cluster&quot; &quot;primary&quot; {<br>  provider                  = google-beta<br>  name                      = var.google_gke_cluster_name<br>  location                  = var.google_gke_cluster_location<br>  remove_default_node_pool  = true<br>  initial_node_count        = 1<br>  enable_shielded_nodes     = true<br>  enable_intranode_visibility = true<br>  release_channel {<br>    channel = &quot;RAPID&quot;<br>  }<br>  master_auth {<br>    username = &quot;kubeadmin&quot;<br>    password = &quot;XXXXXXXXXXX&quot;<br>    client_certificate_config {<br>      issue_client_certificate = false<br>    }<br>  }<br>  network_policy {<br>    enabled = true<br>    provider = &quot;CALICO&quot;<br>  }<br>  ip_allocation_policy {<br>    cluster_ipv4_cidr_block = &quot;192.168.0.0/18&quot;<br>    services_ipv4_cidr_block = &quot;192.168.64.0/18&quot;<br>  }<br>  addons_config {<br>    http_load_balancing {<br>      disabled = false<br>    }<br>    horizontal_pod_autoscaling {<br>      disabled = true<br>    }<br>    istio_config {<br>      disabled = false<br>      auth = &quot;AUTH_MUTUAL_TLS&quot;<br>    }<br>  }<br>  timeouts {<br>    create = &quot;30m&quot;<br>    update = &quot;40m&quot;<br>  }<br>}</pre><pre>resource &quot;google_container_node_pool&quot; &quot;np-01&quot; {<br>  name        = &quot;node-pool-01&quot;<br>  location    = var.google_gke_cluster_location<br>  cluster     = google_container_cluster.primary.name<br>  node_count  = var.google_node_count</pre><pre>management {<br>    auto_repair  = true<br>    auto_upgrade = true<br>  }</pre><pre>node_config {<br>    preemptible  = true<br>    machine_type = &quot;e2-medium&quot;</pre><pre>metadata = {<br>      disable-legacy-endpoints = &quot;true&quot;<br>    }</pre><pre>oauth_scopes = [<br>      &quot;<a href="https://www.googleapis.com/auth/logging.write">https://www.googleapis.com/auth/logging.write</a>&quot;,<br>      &quot;<a href="https://www.googleapis.com/auth/monitoring">https://www.googleapis.com/auth/monitoring</a>&quot;,<br>    ]<br>  }<br>}</pre><pre>resource &quot;google_container_node_pool&quot; &quot;np-02&quot; {<br>  name       = &quot;node-pool-02&quot;<br>  location   = var.google_gke_cluster_location<br>  cluster    = google_container_cluster.primary.name<br>  node_count = var.google_node_count</pre><pre>management {<br>    auto_repair  = true<br>    auto_upgrade = true<br>  }</pre><pre>node_config {<br>    preemptible  = true<br>    machine_type = &quot;e2-medium&quot;</pre><pre>metadata = {<br>      disable-legacy-endpoints = &quot;true&quot;<br>    }</pre><pre>oauth_scopes = [<br>      &quot;<a href="https://www.googleapis.com/auth/logging.write">https://www.googleapis.com/auth/logging.write</a>&quot;,<br>      &quot;<a href="https://www.googleapis.com/auth/monitoring">https://www.googleapis.com/auth/monitoring</a>&quot;,<br>    ]<br>  }<br>}</pre><p>With those 4 files we can spin up a GKE cluster in minutes and have the <a href="https://www.terraform.io/docs/language/state/index.html">Terraform state</a> running in a globally accessible bucket.</p><p>But now we need to modify this for an <a href="https://aws.amazon.com/eks/">Amazon EKS</a> deployment.</p><p>backend.tf</p><pre># List of Regions: <a href="https://docs.aws.amazon.com/general/latest/gr/rande.html">https://docs.aws.amazon.com/general/latest/gr/rande.html</a><br>terraform {<br>  backend &quot;s3&quot; {<br>    bucket = &quot;XXXXXXXXX-terraform&quot;<br>    key    = &quot;cluster-01/terraform/state&quot;<br>    region = &quot;eu-west-1&quot;<br>  }<br>}</pre><p>variables.tf</p><pre>variable &quot;aws_region&quot; {<br>  type = string<br>}</pre><pre>variable &quot;aws_vpc_name&quot; {<br>  type = string<br>}</pre><pre>variable &quot;aws_eks_cluster_name&quot; {<br>  type = string<br>}</pre><pre>variable &quot;aws_eks_k8s_version&quot; {<br>  type = string<br>}</pre><pre>variable &quot;aws_eks_k8s_instance&quot; {<br>  type = string<br>}</pre><p>terraform.tfvars</p><pre>aws_region             = &quot;eu-west-1&quot;<br>aws_vpc_name           = &quot;XXXXXXXXXX&quot;<br>aws_eks_cluster_name   = &quot;cluster-01&quot;<br>aws_eks_k8s_version    = &quot;1.19&quot;<br># <a href="https://aws.amazon.com/ec2/instance-types/">https://aws.amazon.com/ec2/instance-types/</a><br>aws_eks_k8s_instance   = &quot;t3.medium&quot;</pre><p>main.tf (70 lines)</p><pre>provider &quot;aws&quot; {<br>  region = var.aws_region<br>}</pre><pre>data &quot;aws_eks_cluster&quot; &quot;cluster&quot; {<br>  name = module.eks.cluster_id<br>}</pre><pre>data &quot;aws_eks_cluster_auth&quot; &quot;cluster&quot; {<br>  name = module.eks.cluster_id<br>}</pre><pre>provider &quot;kubernetes&quot; {<br>  host                   = data.aws_eks_cluster.cluster.endpoint<br>  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)<br>  token                  = data.aws_eks_cluster_auth.cluster.token<br>}</pre><pre>data &quot;aws_availability_zones&quot; &quot;available&quot; {<br>}</pre><pre>locals {<br>  cluster_name = var.aws_eks_cluster_name<br>}</pre><pre>module &quot;vpc&quot; {<br>  source  = &quot;terraform-aws-modules/vpc/aws&quot;</pre><pre>name                 = var.aws_vpc_name<br>  cidr                 = &quot;172.16.0.0/16&quot;<br>  azs                  = data.aws_availability_zones.available.names<br>  private_subnets      = [&quot;172.16.1.0/24&quot;, &quot;172.16.2.0/24&quot;, &quot;172.16.3.0/24&quot;]<br>  public_subnets       = [&quot;172.16.4.0/24&quot;, &quot;172.16.5.0/24&quot;, &quot;172.16.6.0/24&quot;]<br>  enable_nat_gateway   = true<br>  single_nat_gateway   = true<br>  enable_dns_hostnames = true</pre><pre>public_subnet_tags = {<br>    &quot;kubernetes.io/cluster/${local.cluster_name}&quot; = &quot;shared&quot;<br>    &quot;kubernetes.io/role/elb&quot;                      = &quot;1&quot;<br>  }</pre><pre>private_subnet_tags = {<br>    &quot;kubernetes.io/cluster/${local.cluster_name}&quot; = &quot;shared&quot;<br>    &quot;kubernetes.io/role/internal-elb&quot;             = &quot;1&quot;<br>  }<br>}</pre><pre>module &quot;eks&quot; {<br>  source  = &quot;terraform-aws-modules/eks/aws&quot;</pre><pre>cluster_name    = &quot;${local.cluster_name}&quot;<br>  cluster_version = var.aws_eks_k8s_version<br>  subnets         = module.vpc.private_subnets</pre><pre>vpc_id = module.vpc.vpc_id</pre><pre>node_groups = {<br>    first = {<br>      desired_capacity = 1<br>      max_capacity     = 10<br>      min_capacity     = 1<br>      <br>      instance_type = var.aws_eks_k8s_instance<br>    }<br>  }</pre><pre>write_kubeconfig   = true<br>  config_output_path = &quot;./&quot;<br>}</pre><p>To execute that Terraform, you run the following</p><pre>terraform init<br>terraform plan<br>terraform apply</pre><p>For the guys that are not familiar with the above, the “terraform apply” created us an EKS cluster, running in Western Europe in an autoscale configuration, starting with 1 worker node and being able to expand to 10 worker nodes automatically.</p><p><strong>70 lines of code</strong> got me a highly available <strong>Kubernetes cluster</strong> running in Europe, in about <strong>20 minutes</strong>!</p><p>And how long did it take me to make the new EKS configuration for Terraform? About 1 hour! Unfortunately Discord doesn’t save the exact time, but below is a shot from our conversation during this operation. Again, I am only demonstrating how quick this operation went.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rNlJu3f2lXmlxpeXRKWszQ.png" /><figcaption>Sensitive information has been removed to protect the innocent.</figcaption></figure><blockquote>I want to point out that I am not a smart guy; The reason this was quick and easy is because Terraform makes it quick and easy.</blockquote><h3>HELM 108!</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/455/1*hn_oMUybejURG2wUSINYlg.gif" /></figure><p>With EKS up and running we can now use Helm to deploy all our GitOps application stacks.</p><p>First things first, we set up our Ingress. We chose <a href="https://github.com/traefik/traefik">Traefik</a>, and ran the following 4 lines (yes 4 lines)</p><pre>helm repo add traefik <a href="https://helm.traefik.io/traefik">https://helm.traefik.io/traefik</a><br>helm repo update<br>helm show values traefik/traefik &gt; traefik.values.yaml<br>helm upgrade -i --namespace traefik traefik traefik/traefik --create-namespace --values traefik.values.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ui6D_carPR_fUrjAtC7bQw.png" /><figcaption>Boom, we got Ingress</figcaption></figure><p>Next we install Cert-Manager to we can utilize Let’s Encrypt (yes, 4 lines again)</p><pre>helm repo add jetstack <a href="https://charts.jetstack.io/">https://charts.jetstack.io</a><br>helm repo update<br>helm show values jetstack/cert-manager &gt; cert-manager.values.yaml<br>helm upgrade -i cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace  --values cert-manager.values.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*YY5tD4uUkjE8wqQbn4eKGw.png" /><figcaption>Boom! we got Cert-manager!</figcaption></figure><p>Now install ArgoCD. Guess what I am going to do… in 4 lines…</p><pre>helm repo add argo <a href="https://argoproj.github.io/argo-helm">https://argoproj.github.io/argo-helm</a><br>helm repo update<br>helm show values argo/argo-cd &gt; argocd.values.yaml<br>helm upgrade -i --namespace argocd argocd argo/argo-cd --create-namespace --values argocd.values.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iOZxiQp5sMFZa3WSWQzD1Q.png" /><figcaption>Boom!! ArgoCD!!</figcaption></figure><p>And just like that we have all our supporting applications ready to go. That took about 2 minutes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f5LC2lKKYr8NPfGYngCxfQ.png" /><figcaption>Big Baadaa Boom!</figcaption></figure><h3>Argo, Argo, it’s off to work we Argo</h3><p>Next we can deploy our Validator, thanks to the <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/">CRD</a>, by simply using the ArgoCD “application”</p><p>Here is an example of an ArogCD application that will deploy our applications, from Github, and keep it in sync so only changes that are made to the source code are applied to the deployed application. Or in a word “GitOps”.</p><pre>apiVersion: argoproj.io/v1alpha1<br>kind: Application<br>metadata:<br>  name: prysm-validator<br>  namespace: argocd<br>spec:<br>  destination:<br>    namespace: eth2<br>    server: <a href="https://kubernetes.default.svc/">https://kubernetes.default.svc</a><br>  project: staking-prod<br>  source:<br>    helm:<br>      valueFiles:<br>        - values.yaml<br>    path: argocd/charts/test/prysm-validator<br>    repoURL: <a href="/service/mailto:git@github.com">git@github.com</a>:FAKENAME-prod/api.cluster-01.FAKENAME-prod.git<br>    targetRevision: main<br>    automated:<br>      prune: true<br>      selfHeal: true<br>    syncOptions:<br>      - CreateNamespace=true</pre><p>And here is the structure of our Github repo</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*zC5frsnmg8xFEtDELJwa-g.png" /></figure><p>Which means we can apply all our CRD applications with a single command!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/1*WiF9vfZBhAgllSmMb7bSag.gif" /></figure><pre>$ kubectl apply -R -f argocd/</pre><p>And just like that ladies and gentleman, we have our application stack restored and making sweet, sweet Ether.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*USvwsDOAIHbdZSsTsZmikg.png" /></figure><h3>Didn’t I mention something about Let’s Encrypt?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*3pcaSmNaW_aRSgN2IvGBIA.jpeg" /></figure><p>Did I mention how much I love Let’s Encrypt? Check out how we automatically have all our Ingress traffic secured by Let’s Encrypt’s ACME bot.</p><p>Check out these Issuers!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*AxOOHr5qMarWUkfCKuz16A.png" /><figcaption>I got issues man</figcaption></figure><p>Check out these Request and Orders and Certificates!!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J2m7nSPgfPH8CdkrnOX42w.png" /></figure><h3>Recap Time!</h3><p>So how long did this all take to re-platform our entire Ethereum Staking Validator project from Google Cloud to Amazon AWS? I am going to round up these numbers to be generous</p><ul><li>Destroy project in Google Cloud (1 minute)</li><li>Rework Terraform Code to build Amazon EKS (2 hour)</li><li>Terraform to build EKS (30 minutes)</li><li>Deploy Ingress + Cert-manager + ArgoCD (5 minutes)</li><li>Apply ArgoCD application CRD (1 minute)</li><li>Wait for ArgoCD to finish doing it’s than (5 minutes)</li></ul><p>Therefore I declare that in under 3 hours, you too can move an entire application stack from one cloud to another cloud by following the above practices and not being scared of being… ThunderStruck!</p><h3>Closing Statement</h3><p>If you have reached this part, wow, I am surprised. Someone actually read this? Now it is important to state that we were able to be agile and mobile because of <strong>one very big constraint we did not have</strong>, and that is persistent data. The project we are running does not require persistence, which made everything a million times easier. If we did have data in the suspended Google project this article would never have been written. That said, this has highlighted to me that you want to make plans to get your data out of a Cloud regardless of the reason, because it’s the reason that you don’t think of that ends up biting you in the butt.</p><p>Thinking out loud here, I would probably backup my data to a separate Cloud as an exit plan.</p><p>The reason I took the time out to write this (I am actually a very busy guy), is to highlight to everyone that Cloud lock in/<a href="https://www.computerweekly.com/news/252494751/Cut-off-from-the-cloud-How-Parler-became-a-tech-industry-pariah">lock out</a> is real, but it doesn’t have to bring fear into your life. All the software we used here is 100% free and open source (yes yes, the Clouds are not free), and this can be achieved by anybody these days. This article also wasn’t designed to be a walk through with exact steps, it is a high level overview of what we did and how we did it.</p><p>Again to make it very clear, this is not an article to bad mouth Google and praise Amazon. I’m personally a huge fan of Google. To be honest, we will probably be re-platforming to Digital Ocean and Azure so we can test the waters (or raindrops) on the other Clouds, and then probably back to Amazon to take advantage of their <a href="https://aws.amazon.com/ec2/spot/">spot instances</a>.</p><p>This is the wonderfulness of this design/architecture as it allows us to move around, to explore, encouraging us to use new technology, all while feeling confident and safe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D0Ncx0n8AZ90cfJO_5CARw.png" /></figure><p>Oh yeah, Courage is another <a href="https://wiki.lsd.co.za/en/public/LSD/values">core value</a> of LSD :)</p><h3>What is LSD?</h3><p>If you saw the words “LSD” and were curious….well….</p><p><a href="https://lsd.co.za/">LSD</a> is an open source focused company that helps companies along their journey into the cloud, or what we like to call the LSDtrip.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6fd526f23be4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Recovering from PostgreSQL-HA Failure on Kubernetes]]></title>
            <link>https://neilcameronwhite.medium.com/recovering-from-postgresql-ha-failure-on-kubernetes-8c523c986f85?source=rss-807665507843------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c523c986f85</guid>
            <category><![CDATA[bitnami]]></category>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[postgresql]]></category>
            <dc:creator><![CDATA[Neil White]]></dc:creator>
            <pubDate>Wed, 16 Sep 2020 19:52:29 GMT</pubDate>
            <atom:updated>2020-09-17T06:26:25.079Z</atom:updated>
            <content:encoded><![CDATA[<p>I am writing this article so that others do not have to go through the pain I just experienced, after I rebooted a couple of Kubernetes nodes because the time had drifted substantially on all of them.</p><p>We have a project running on Kubernetes that is composed of many services all sharing a central, highly available PostgreSQL database. We first detected the time drift issue when <a href="https://grpc.io/">gRPC</a> on <a href="https://gitlab.com/">Gitlab</a> wasn’t working. Once we corrected the <a href="https://rancher.com/docs/os/v1.x/en/configuration/write-files/#writing-files-in-specific-system-services">NTP settings</a>, I issued a reboot on the nodes and that is where our story begins…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*3mF8u97P2bq2V7WrH5vL3g.gif" /></figure><blockquote>Disclaimer!<br>I am in no way a PostgreSQL expert. If there is a better way of solving this problem, please share, as I couldn’t find the answer.</blockquote><p>Because we have a central PostgreSQL that multiple services use, we opted to deploy the the <a href="https://bitnami.com/stack/postgresql-ha/helm">Bitnami PostgreSQL-HA helm chart</a>. It was quick and simple and I didn’t take to much time digging into it. Once deployed it provides the following pods</p><pre>postgresql-ha-pgpool-74c8574dbf-skdwx<br>postgresql-ha-postgresql-0<br>postgresql-ha-postgresql-1</pre><p>With the following kubernetes services</p><pre>postgresql-ha-pgpool<br>postgresql-ha-postgresql<br>postgresql-ha-postgresql-headless</pre><p>After I rebooted the nodes, all 3 PostgreSQL pods where going into CrashLoopBackOff. When I looked at the logs, I saw the following error</p><pre>postgresql-repmgr 18:20:05.54 INFO  ==&gt; Starting PostgreSQL in background...<br>waiting for server to start....2020-09-16 18:20:05.612 GMT [120] LOG:  listening on IPv4 address &quot;0.0.0.0&quot;, port 5432<br>2020-09-16 18:20:05.613 GMT [120] LOG:  listening on IPv6 address &quot;::&quot;, port 5432<br>2020-09-16 18:20:05.616 GMT [120] LOG:  listening on Unix socket &quot;/tmp/.s.PGSQL.5432&quot;<br>2020-09-16 18:20:05.640 GMT [120] LOG:  redirecting log output to logging collector process<br>2020-09-16 18:20:05.640 GMT [120] HINT:  Future log output will appear in directory &quot;/opt/bitnami/postgresql/logs&quot;.<br>2020-09-16 18:20:05.643 GMT [122] LOG:  database system was interrupted while in recovery at log time 2020-09-16 15:04:55 GMT<br>2020-09-16 18:20:05.643 GMT [122] HINT:  If this has occurred more than once some data might be corrupted and you might need to choose an earlier recovery target.<br>2020-09-16 18:20:05.769 GMT [122] LOG:  entering standby mode<br>2020-09-16 18:20:05.782 GMT [122] LOG:  redo starts at 0/24000028<br>2020-09-16 18:20:05.782 GMT [122] LOG:  consistent recovery state reached at 0/25000000<br>2020-09-16 18:20:05.783 GMT [120] LOG:  database system is ready to accept read only connections<br>2020-09-16 18:20:05.793 GMT [126] LOG:  started streaming WAL from primary at 0/25000000 on timeline 4<br> done<br>server started<br>postgresql-repmgr 18:20:05.87 INFO  ==&gt; ** Starting repmgrd **<br>[2020-09-16 18:20:05] [NOTICE] repmgrd (repmgrd 5.1.0) starting up<br>[2020-09-16 18:20:05] [ERROR] this node is marked as inactive and cannot be used as a failover target<br>[2020-09-16 18:20:05] [HINT] Check that &quot;repmgr (primary|standby) register&quot; was executed for this node</pre><p>So the Primary node has entered Standby Mode. Not a problem! (or so I thought), I will quickly repair a PostgreSQL database, inside of a pod that is crashing, and run whatever command I need to run to make it cool again. How difficult could that be?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/985/1*0X3jTHZ5f0WCKbr1yT8moQ.jpeg" /></figure><p>I will not bore you with what I tried, I will simply tell you what I ended up doing.</p><h3>1) Prevent the pod from CrashLoopBackOff</h3><p>Edit the statefuleset</p><pre>kubectl edit statefulsets.apps postgresql-ha-postgresql</pre><p>And put in a sleep command</p><pre>spec:<br>      containers:<br>      - command: [&#39;sh&#39;, &#39;-c&#39;, &#39;echo The app is running! &amp;&amp; sleep 3600&#39;]<br>        env:<br>        - name: BITNAMI_DEBUG<br>          value: &quot;true&quot;<br>        - name: POSTGRESQL_VOLUME_DIR<br>          value: /bitnami/postgresql<br>        - name: PGDATA<br>          value: /bitnami/postgresql/data<br>        - name: POSTGRES_USER<br>          value: postgres</pre><p>This will make the pod sleep for 3600 second and override the CMD and Entrypoint from the container image.</p><h3>2) Remove Health Checks</h3><p>Remaining in that statefulset, remove the livenessProbe</p><pre>livenessProbe:<br>          exec:<br>            command:<br>            - sh<br>            - -c<br>            - PGPASSWORD=$POSTGRES_PASSWORD psql -w -U &quot;postgres&quot; -d &quot;postgres&quot;  -h<br>              127.0.0.1 -c &quot;SELECT 1&quot;<br>          failureThreshold: 6<br>          initialDelaySeconds: 30<br>          periodSeconds: 10<br>          successThreshold: 1<br>          timeoutSeconds: 5</pre><p>And readinessProbe</p><pre>readinessProbe:<br>          exec:<br>            command:<br>            - sh<br>            - -c<br>            - PGPASSWORD=$POSTGRES_PASSWORD psql -w -U &quot;postgres&quot; -d &quot;postgres&quot;  -h<br>              127.0.0.1 -c &quot;SELECT 1&quot;<br>          failureThreshold: 6<br>          initialDelaySeconds: 5<br>          periodSeconds: 10<br>          successThreshold: 1<br>          timeoutSeconds: 5</pre><p>If you do not do this, even though the pod is sleeping, it will restart because of the livenessProbe and will not accept traffic because of the readinessProbe.</p><p>Save and quit</p><h3>3) Become the Pod</h3><p>With those changes made, you will need to manually delete the pod</p><pre>kubectl delete po postgresql-ha-postgresql-0</pre><p>And once it has started up, you can shell into it</p><pre>kubectl exec postgresql-ha-postgresql-0 -it -- /bin/bash</pre><h3>4) Fix what you broke</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*TZF5O2o1btJIK1MMuyIgDA.jpeg" /></figure><p>Once you have shelled into the pod, start PostgreSQL</p><pre>/opt/bitnami/scripts/postgresql-repmgr/run.sh</pre><p>With PostgreSQL running, we can actually see what is wrong. Check the status of repmgr</p><pre>/opt/bitnami/scripts/postgresql-repmgr/entrypoint.sh repmgr -f /opt/bitnami/repmgr/conf/repmgr.conf cluster show</pre><p>Here is the error I had</p><pre>postgresql-repmgr 18:37:20.91 <br>postgresql-repmgr 18:37:20.91 Welcome to the Bitnami postgresql-repmgr container<br>postgresql-repmgr 18:37:20.92 Subscribe to project updates by watching <a href="https://github.com/bitnami/bitnami-docker-postgresql-repmgr">https://github.com/bitnami/bitnami-docker-postgresql-repmgr</a><br>postgresql-repmgr 18:37:20.92 Submit issues and feature requests at <a href="https://github.com/bitnami/bitnami-docker-postgresql-repmgr/issues">https://github.com/bitnami/bitnami-docker-postgresql-repmgr/issues</a><br>postgresql-repmgr 18:37:20.92 <br>postgresql-repmgr 18:37:20.93 DEBUG ==&gt; Configuring libnss_wrapper...</pre><pre>ID   | Name                       | Role    | Status               | Upstream | Location | Priority | Timeline | Connection string                                                                                                                                                         <br>------+----------------------------+---------+----------------------+----------+----------+----------+----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br> 1000 | postgresql-ha-postgresql-0 | primary | ! running as standby |          | default  | 100      | 4        | user=repmgr password=sMOIs9AZow host=postgresql-ha-postgresql-0.postgresql-ha-postgresql-headless.lsd-automate.svc.cluster.local dbname=repmgr port=5432 connect_timeout=5<br> 1001 | postgresql-ha-postgresql-1 | primary | ? unreachable        | ?        | default  | 100      |          | user=repmgr password=sMOIs9AZow host=postgresql-ha-postgresql-1.postgresql-ha-postgresql-headless.lsd-automate.svc.cluster.local dbname=repmgr port=5432 connect_timeout=5</pre><pre>WARNING: following issues were detected<br>  - node &quot;postgresql-ha-postgresql-0&quot; (ID: 1000) is registered as an inactive primary but running as standby<br>  - unable to connect to node &quot;postgresql-ha-postgresql-1&quot; (ID: 1001)<br>  - node &quot;postgresql-ha-postgresql-1&quot; (ID: 1001) is registered as an active primary but is unreachable</pre><pre>HINT: execute with --verbose option to see connection error messages</pre><p>That wrapping above is terrible, so here is a screenshot</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vS51NNvFwDGOI32ZCnDM2Q.png" /></figure><p>And here is the chicken and egg / dumb and dumber part. The first pod will not start, because it is running as Standby and because it is a statefulset the second pod never starts up because it is waiting for the first pod to be ready.</p><blockquote>“Insert Nordic curse words here”</blockquote><p>We can keep digging while we are here and see how the node is doing</p><pre>/opt/bitnami/scripts/postgresql-repmgr/entrypoint.sh repmgr -f /opt/bitnami/repmgr/conf/repmgr.conf node check</pre><p>And that returned the following</p><pre>postgresql-repmgr 18:41:05.32 <br>postgresql-repmgr 18:41:05.32 Welcome to the Bitnami postgresql-repmgr container<br>postgresql-repmgr 18:41:05.32 Subscribe to project updates by watching <a href="https://github.com/bitnami/bitnami-docker-postgresql-repmgr">https://github.com/bitnami/bitnami-docker-postgresql-repmgr</a><br>postgresql-repmgr 18:41:05.32 Submit issues and feature requests at <a href="https://github.com/bitnami/bitnami-docker-postgresql-repmgr/issues">https://github.com/bitnami/bitnami-docker-postgresql-repmgr/issues</a><br>postgresql-repmgr 18:41:05.33 <br>postgresql-repmgr 18:41:05.33 DEBUG ==&gt; Configuring libnss_wrapper...</pre><pre>Node &quot;postgresql-ha-postgresql-0&quot;:<br> Server role: CRITICAL (node is registered as primary but running as standby)<br> Replication lag: OK (0 seconds)<br> WAL archiving: OK (0 pending archive ready files)<br> Upstream connection: CRITICAL (node &quot;postgresql-ha-postgresql-0&quot; (ID: 1000) is a standby but no upstream record found)<br> Downstream servers: OK (this node has no downstream nodes)<br> Replication slots: OK (node has no physical replication slots)<br> Missing physical replication slots: OK (node has no missing physical replication slots)<br> Configured data directory: OK (configured &quot;data_directory&quot; is &quot;/bitnami/postgresql/data&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UVxH4bt2m9JqX-Rtaa_LvQ.png" /></figure><p>Ok, confirmed, we have issues, stop messing about and run the repair command. Let’s promote this server from Standby to Primary.</p><pre>/opt/bitnami/scripts/postgresql-repmgr/entrypoint.sh repmgr -f /opt/bitnami/repmgr/conf/repmgr.conf standby promote</pre><p>Which gives the output</p><pre>postgresql-repmgr 18:42:13.44 <br>postgresql-repmgr 18:42:13.44 Welcome to the Bitnami postgresql-repmgr container<br>postgresql-repmgr 18:42:13.44 Subscribe to project updates by watching <a href="https://github.com/bitnami/bitnami-docker-postgresql-repmgr">https://github.com/bitnami/bitnami-docker-postgresql-repmgr</a><br>postgresql-repmgr 18:42:13.45 Submit issues and feature requests at <a href="https://github.com/bitnami/bitnami-docker-postgresql-repmgr/issues">https://github.com/bitnami/bitnami-docker-postgresql-repmgr/issues</a><br>postgresql-repmgr 18:42:13.45 <br>postgresql-repmgr 18:42:13.46 DEBUG ==&gt; Configuring libnss_wrapper...</pre><pre>NOTICE: promoting standby to primary<br>DETAIL: promoting server &quot;postgresql-ha-postgresql-0&quot; (ID: 1000) using &quot;/opt/bitnami/postgresql/bin/pg_ctl  -w -D &#39;/bitnami/postgresql/data&#39; promote&quot;<br>waiting for server to promote..... done<br>server promoted<br>NOTICE: waiting up to 60 seconds (parameter &quot;promote_check_timeout&quot;) for promotion to complete<br>NOTICE: STANDBY PROMOTE successful<br>DETAIL: server &quot;postgresql-ha-postgresql-0&quot; (ID: 1000) was successfully promoted to primary<br>[REPMGR EVENT] Node id: 1000; Event type: standby_promote; Success [1|0]: 1; Time: 2020-09-16 18:42:14.771401+00;  Details: server &quot;postgresql-ha-postgresql-0&quot; (ID: 1000) was successfully promoted to primary<br>Looking for the script: /opt/bitnami/repmgr/events/execs/standby_promote.sh<br>[REPMGR EVENT] will execute script &#39;/opt/bitnami/repmgr/events/execs/standby_promote.sh&#39; for the event<br>[REPMGR EVENT::standby_promote] Node id: 1000; Event type: standby_promote; Success [1|0]: 1; Time: 2020-09-16 18:42:14.771401+00;  Details: server &quot;postgresql-ha-postgresql-0&quot; (ID: 1000) was successfully promoted to primary<br>[REPMGR EVENT::standby_promote] Locking primary...<br>[REPMGR EVENT::standby_promote] Unlocking standby...</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oEo4SvQFlkDiPF7niWveIA.png" /></figure><p>And now if we look at the cluster it looks a lot better</p><pre>/opt/bitnami/scripts/postgresql-repmgr/entrypoint.sh repmgr -f /opt/bitnami/repmgr/conf/repmgr.conf cluster show</pre><p>Which shows us</p><pre>postgresql-repmgr 18:42:28.78 <br>postgresql-repmgr 18:42:28.78 Welcome to the Bitnami postgresql-repmgr container<br>postgresql-repmgr 18:42:28.79 Subscribe to project updates by watching <a href="https://github.com/bitnami/bitnami-docker-postgresql-repmgr">https://github.com/bitnami/bitnami-docker-postgresql-repmgr</a><br>postgresql-repmgr 18:42:28.79 Submit issues and feature requests at <a href="https://github.com/bitnami/bitnami-docker-postgresql-repmgr/issues">https://github.com/bitnami/bitnami-docker-postgresql-repmgr/issues</a><br>postgresql-repmgr 18:42:28.79 <br>postgresql-repmgr 18:42:28.80 DEBUG ==&gt; Configuring libnss_wrapper...</pre><pre>ID   | Name                       | Role    | Status    | Upstream | Location | Priority | Timeline | Connection string                                                                                                                                                         <br>------+----------------------------+---------+-----------+----------+----------+----------+----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br> 1000 | postgresql-ha-postgresql-0 | primary | * running |          | default  | 100      | 5        | user=repmgr password=sMOIs9AZow host=postgresql-ha-postgresql-0.postgresql-ha-postgresql-headless.lsd-automate.svc.cluster.local dbname=repmgr port=5432 connect_timeout=5<br> 1001 | postgresql-ha-postgresql-1 | primary | - failed  | ?        | default  | 100      |          | user=repmgr password=sMOIs9AZow host=postgresql-ha-postgresql-1.postgresql-ha-postgresql-headless.lsd-automate.svc.cluster.local dbname=repmgr port=5432 connect_timeout=5</pre><pre>WARNING: following issues were detected<br>  - unable to connect to node &quot;postgresql-ha-postgresql-1&quot; (ID: 1001)</pre><pre>HINT: execute with --verbose option to see connection error messages</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m8lrznFj9Go3DXTSCM813g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*qdLrdghGwvW69W4dNg1Bew.jpeg" /></figure><h3>5) Return everything back the way you found it</h3><p>Edit your statefulset again</p><pre>kubectl edit statefulsets.apps postgresql-ha-postgresql</pre><p>And remove</p><pre>command: [&#39;sh&#39;, &#39;-c&#39;, &#39;echo The app is running! &amp;&amp; sleep 3600&#39;]</pre><p>And return livenessProbe</p><pre>livenessProbe:<br>          exec:<br>            command:<br>            - sh<br>            - -c<br>            - PGPASSWORD=$POSTGRES_PASSWORD psql -w -U &quot;postgres&quot; -d &quot;postgres&quot;  -h<br>              127.0.0.1 -c &quot;SELECT 1&quot;<br>          failureThreshold: 6<br>          initialDelaySeconds: 30<br>          periodSeconds: 10<br>          successThreshold: 1<br>          timeoutSeconds: 5</pre><p>And return readinessProbe</p><pre>readinessProbe:<br>          exec:<br>            command:<br>            - sh<br>            - -c<br>            - PGPASSWORD=$POSTGRES_PASSWORD psql -w -U &quot;postgres&quot; -d &quot;postgres&quot;  -h<br>              127.0.0.1 -c &quot;SELECT 1&quot;<br>          failureThreshold: 6<br>          initialDelaySeconds: 5<br>          periodSeconds: 10<br>          successThreshold: 1<br>          timeoutSeconds: 5</pre><p>Finally, restarting your pods</p><pre>kubectl delete po postgresql-ha-postgresql-1 postgresql-ha-postgresql-0 postgresql-ha-pgpool-74c8574dbf-vnjtb</pre><p>I hope this helps someone else in the future so you don’t have to go through what I did. And as always if there is a better way to do this, add a comment.</p><h3>What is LSD?</h3><p>If you saw the words “LSD” and were curious….well….</p><p><a href="https://lsd.co.za/">LSD</a> is an open source focused company that helps companies along their journey into the cloud, or what we like to call the LSD Trip.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c523c986f85" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Migrating Hashicorp Vault from Virtual Machines to Openshift/Kubernetes and changing from consul…]]></title>
            <link>https://neilcameronwhite.medium.com/migrating-hashicorp-vault-from-virtual-machines-to-openshift-kubernetes-and-changing-from-consul-c5eb68f4b05c?source=rss-807665507843------2</link>
            <guid isPermaLink="false">https://medium.com/p/c5eb68f4b05c</guid>
            <category><![CDATA[hashicorp-vault]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[openshift]]></category>
            <dc:creator><![CDATA[Neil White]]></dc:creator>
            <pubDate>Tue, 21 Jul 2020 17:52:18 GMT</pubDate>
            <atom:updated>2020-07-21T18:07:36.992Z</atom:updated>
            <content:encoded><![CDATA[<h3>Migrating Hashicorp Vault from Virtual Machines to Openshift/Kubernetes and changing from consul to raft and making it highly available</h3><p>Damn, that title is a mouthful. Oh yeah, I wrote this in July 2020. If you are currently in the year 2022, this might be outdated…</p><p>I recently received a request from a customer to migrate their <a href="https://www.vaultproject.io/">Hashicorp Vault</a> installation from a traditional virtual machine deployment, onto a newly deployed Openshift (think Kubernetes with more bells and whistles) platform. They recently tasted of the benefits of Kubernetes and wanted their Vault setup to also benefit from it. Their existing Vault installation suffered from the usual bane of IT, such as:</p><ul><li>Load Balancing wasn’t completed 100%</li><li>Upgrades were not maintained</li><li>The original guy who installed it has left</li><li>No one really understands how it was installed or how it works</li></ul><p>These problems, and more, can be solved when using a technology like Kubernetes. I know Kubernetes isn’t the silver bullet, but I felt in this case I could show the guys the benefits because:</p><ul><li>Kubernetes automatically has <a href="https://kubernetes.io/docs/concepts/services-networking/service/">internal load-balancers</a></li><li>Upgrades can be easily done using <a href="https://helm.sh/docs/helm/helm_upgrade/">Helm</a></li><li>All deployment is now completed via code (GitOps, hooraa!)</li></ul><p>The team I would be working with was primarily operations, so they care about stability and having to maintain the setup for the long haul.</p><p>This is was the existing setup:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/176/1*Hkr2DAydVKBVyzX3VfeBGQ.png" /></figure><p>1x Virtual Machine, running Hashicorp Vault version 0.8.2 and Consul v1.0.0, on RHEL 7.</p><p>They had a separate Vault for:</p><ul><li>Development</li><li>QA</li><li>Pre Production</li><li>Production (3 Vaults with no Load Balancer in the front)</li></ul><p>This is what they wanted:</p><blockquote>“All Vaults to be highly available, running on Openshift”</blockquote><h3>What’s the Plan Batman!</h3><p>I wanted the new solution to be simple, so I decided to go with the <a href="https://helm.releases.hashicorp.com/">Hashicorp Helm charts</a> to deploy the Vaults and for their storage backend to use the new <a href="https://www.vaultproject.io/docs/configuration/storage/raft">Raft storage backend</a>.</p><p>Helm, so that they don’t need to care about Kubernetes deployments, services, ports and images.</p><p>Raft so that the data doesn’t require an external data source like Consul.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*JdtL6zWD5zroifp73scPUw.png" /></figure><p>The high level steps would be:</p><ul><li>Extract data from existing Consul, running on VM</li><li>Setup Helm and projects/namespaces on Openshift</li><li>Deploy new Consul, running on Openshift</li><li>Restore data to new Consul, running on Openshift</li><li>Convert data from Consul to Raft storage backend format</li><li>Deploy Highly Available Vault</li><li>Restore Raft snapshot to new Highly Available Vault</li><li>Join Rafts Together</li></ul><p>This would insure that all tokens, secrets &amp; policies remain the same.</p><blockquote><strong>Before you being you must have the unseal keys and root token of the Vault you are migrating</strong></blockquote><p>If you don’t have that, then don’t bother wasting your time. Unless you have a way to unseal a vault…..then…..email me….</p><h3>Extract data from existing Consul</h3><p>For this you will need ssh access to the server. I didn’t have root access which didn’t matter. In this setup Consul and Vault run on the same server. I will be calling this server “old-dev-vault”.</p><p>ssh into the sever and verify Consul is running on port 8500</p><pre>$ ssh old-dev-vault<br>Last login: Thu Jul 16 11:30:50 2020 from 10.36.10.82</pre><pre>$ ss -lntup | grep 8500<br>tcp    LISTEN     0      128    127.0.0.1:8500                  *:*</pre><p>Take a snapshot of Consul</p><pre>$ /opt/consul/consul snapshot save 14072020.consul.snapshot<br>Saved and verified snapshot to index 6092857</pre><p>Inspect the snapshot, making sure it is all good</p><pre>$ /opt/consul/consul snapshot inspect 14072020.consul.snapshot<br>ID           15-6092857-1594720624826<br>Size         518755<br>Index        6092857<br>Term         15<br>Version      1</pre><p>We will come back to that file later on</p><h3>Setup Helm and projects/namespaces on Openshift</h3><p>This guide assumes you already understand the basics of Openshift/Kubernetes.</p><p>We will need to create a temporary project, that will be using to get the data into Raft format.</p><p>Create a new project called “vault-migrator”</p><pre>$ oc new-project vault-migrator --display-name=&quot;Temporary Restore for Hashicorp Vault&quot; --description=&quot;Temporary Restore for Hashicorp Vault&quot;</pre><p>Add the Hashicorp Helm repo. I am using helm v3.0.3</p><p>If you do not have Helm 3 it is incredibly difficult to install, but I will try do my best on how to quickly get it up and running.</p><pre>$ wget https://get.helm.sh/helm-v3.2.1-linux-amd64.tar.gz<br>$ tar zxvf helm-v3.2.1-linux-amd64.tar.gz<br>$ sudo mv linux-amd64/helm /usr/local/bin/<br>$ rm helm-v3.2.1-linux-amd64.tar.gz linux-amd64/ -rf</pre><p>Hope that installation of Helm didn’t blow your mind to much. Right, add the repo.</p><pre>$ helm repo add hashicorp https://helm.releases.hashicorp.com<br>$ helm repo update</pre><p>Verify it is is all good</p><pre>$ helm search repo</pre><pre>NAME             CHART VERSION APP VERSION DESCRIPTION                    <br>hashicorp/consul 0.23.0        1.8.0       Official HashiCorp Consul Chart<br>hashicorp/vault  0.6.0         1.4.2       Official HashiCorp Vault Chart</pre><h3>Deploy new Consul</h3><p>We are now going to deploy Consul, using Helm, in the project/namespace “vault-migrator” on Openshift</p><p>First, we are going to create a custom Helm value file for Consul</p><p>Create a file called: consul-migrator.values.yaml</p><pre>global:<br>  enabled: true<br>  domain: consul<br>  image: &quot;consul:1.8.0&quot;<br>  imageK8S: &quot;hashicorp/consul-k8s:0.17.0&quot;<br>  imageEnvoy: &quot;envoyproxy/envoy-alpine:v1.14.2&quot;<br>  datacenter: dc1<br>  enablePodSecurityPolicies: false<br>  tls:<br>    enabled: false<br>  acls:<br>    manageSystemACLs: false<br>    enabled: false<br>    createFederationSecret: false<br>server:<br>  enabled: true<br>  replicas: 1<br>  bootstrapExpect: 1<br>  storage: 10Gi<br>  connect: true<br>  resources:<br>    requests:<br>      memory: &quot;100Mi&quot;<br>      cpu: &quot;100m&quot;<br>    limits:<br>      memory: &quot;100Mi&quot;<br>      cpu: &quot;100m&quot;<br>  updatePartition: 0<br>  disruptionBudget:<br>    enabled: false<br>    maxUnavailable: null<br>externalServers:<br>  enabled: false<br>client:<br>  enabled: true<br>  join: false<br>  grpc: true<br>  exposeGossipPorts: false<br>  resources:<br>    requests:<br>      memory: &quot;100Mi&quot;<br>      cpu: &quot;100m&quot;<br>    limits:<br>      memory: &quot;100Mi&quot;<br>      cpu: &quot;100m&quot;</pre><p>Adjust the values appropriately, if you really need to. Those setting should work in most cases.</p><p>Now, one of the features of Openshift is to lock down applications be default, preventing them from having root privileges, or running as a the root user inside of a pod. Openshift calls this <a href="https://docs.openshift.com/container-platform/4.5/authentication/managing-security-context-constraints.html">Security Context Constraints</a> (SCC).</p><p>Before we deploy Consul, we need to allow the serviceaccount to run as any user id.</p><pre>$ oc adm policy add-scc-to-user anyuid -z consul-consul-client -n vault-migrator</pre><pre>$ oc adm policy add-scc-to-user anyuid -z consul-consul-server -n vault-migrator</pre><p>There we are saying, allow the “anyuid” SCC to the serviceaccount “consul-consul-client” in the project/namespace “vault-migrator”. And then we do it again for the “consul-consul-server” serviceaccount.</p><p>This will allow the Consul pods to run as any user they want</p><p>Once complete, install Consul</p><pre>$ helm install consul hashicorp/consul -n vault-migrator --values consul-migrator.values.yaml</pre><pre>NAME: consul<br>LAST DEPLOYED: Thu Jul 16 15:28:28 2020<br>NAMESPACE: vault-migrator<br>STATUS: deployed<br>REVISION: 1<br>NOTES:<br>Thank you for installing HashiCorp Consul!</pre><p>That will install a deployment called “consul”, from the repo “hashicorp/consul” in the project/namespace “vault-migrator” using the value files “consul-migrator.values.yaml”</p><p>You can verify all is good by seeing if a single pod is running in the “vault-migrator” namespace</p><pre>$ oc get po -n vault-migrator </pre><pre>NAME                     READY   STATUS    RESTARTS   AGE<br>consul-consul-server-0   1/1     Running   0          78s</pre><p>And it should have created a Persistent Volume Claim (sorry about the weird wrapping)</p><pre>$ oc get pvc -n vault-migrator</pre><pre>NAME                                         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE<br>data-vault-migrator-consul-consul-server-0   Bound    pvc-24296a30-2a3e-4d95-92b0-1badb7c01b92   10Gi       RWO            thin           2m5s</pre><p>Verify you are running on the new Consul version.</p><pre>$ oc project vault-migrator<br>$ oc exec consul-consul-server-0 -it -- consul version</pre><pre>Consul v1.8.0<br>Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol &gt;2 when speaking to compatible agents)</pre><h3>Restore data to new Consul</h3><p>With the pod up and running, you now need to copy the Consul snapshot onto this pod. Remember this file “<strong>14072020.consul.snapshot</strong>”</p><p>Make a directory on that pod and copy the snapshot file onto the pod</p><pre>$ oc exec consul-consul-server-0 -it -- mkdir /consul/data/snapshots  </pre><pre>$ oc cp <strong>14072020.consul.snapshot</strong> consul-consul-server-0:/consul/data/snapshots/</pre><p>And then restore the snapshot onto that Consul pod</p><pre>$ oc exec consul-consul-server-0 -it -- consul snapshot restore /consul/data/snapshots/<strong>14072020.consul.snapshot</strong></pre><pre>Restored snapshot</pre><p>Well done. You know have your Consul restored onto a new pod in Openshift/Kubernetes.</p><h3>Convert data from Consul to Raft storage backend format</h3><p>To <a href="https://learn.hashicorp.com/vault/operations/migrate-to-raft">migrate to the Raft backend storage</a>, we will be deploying a Vault pod and then execute the migration command from within that pod Vault.</p><p>The purpose of this Vault is only to migrate the data. We will be deleting this Vault once we have our new Raft data</p><p>Once again, create a values file: vault-migrator.values.yaml</p><pre>global:<br>  enabled: true<br>  tlsDisable: true<br>  openshift: true<br>injector:<br>  enabled: false<br>  image:<br>    repository: &quot;hashicorp/vault-k8s&quot;<br>    tag: &quot;0.4.0&quot;<br>    pullPolicy: IfNotPresent<br>  agentImage:<br>    repository: &quot;vault&quot;<br>    tag: &quot;1.4.2&quot;<br>  authPath: &quot;auth/kubernetes&quot;<br>  logLevel: &quot;info&quot;<br>  logFormat: &quot;standard&quot;<br>  revokeOnShutdown: false<br>  resources:<br>    requests:<br>      memory: 256Mi<br>      cpu: 250m<br>    limits:<br>      memory: 256Mi<br>      cpu: 250m<br>server:<br>  image:<br>    repository: &quot;vault&quot;<br>    tag: &quot;1.4.2&quot;<br>    pullPolicy: IfNotPresent<br>  updateStrategyType: &quot;OnDelete&quot;<br>  resources:<br>    requests:<br>      memory: 256Mi<br>      cpu: 250m<br>    limits:<br>      memory: 256Mi<br>      cpu: 250m<br>  ingress:<br>    enabled: false<br>  route:<br>    enabled: false<br>  authDelegator:<br>    enabled: false<br>  readinessProbe:<br>    enabled: true<br>  livenessProbe:<br>    enabled: false<br>    path: &quot;/v1/sys/health?standbyok=true&quot;<br>    initialDelaySeconds: 60<br>  preStopSleepSeconds: 5<br>  service:<br>    enabled: true<br>    type: ClusterIP<br>    port: 8200<br>    targetPort: 8200<br>  dataStorage:<br>    enabled: true<br>    size: 10Gi<br>    accessMode: ReadWriteOnce<br>  auditStorage:<br>    enabled: false<br>  dev:<br>    enabled: false<br>  standalone:<br>    enabled: true<br>    config: |<br>      ui = true<br>      listener &quot;tcp&quot; {<br>        tls_disable = 1<br>        address = &quot;[::]:8200&quot;<br>        cluster_address = &quot;[::]:8201&quot;<br>      }<br>      storage &quot;consul&quot; {<br>        path = &quot;vault&quot;<br>        address = &quot;<strong>consul-consul-server</strong>:8500&quot;<br>      }<br>  ha:<br>    enabled: false<br>ui:<br>  enabled: true<br>  serviceType: &quot;ClusterIP&quot;<br>  externalPort: 8200</pre><p>The highlights here is that we are not enabling HA and we are pointing it to the Consul pod, via the service, address = “<strong>consul-consul-server</strong>:8500”</p><p>Install temporary Vault</p><pre>$ helm install vault-migrator hashicorp/vault -n vault-migrator --values vault-migrator.values.yaml</pre><pre>LAST DEPLOYED: Thu Jul 16 15:48:30 2020<br>NAMESPACE: vault-migrator<br>STATUS: deployed<br>REVISION: 1<br>TEST SUITE: None<br>NOTES:<br>Thank you for installing HashiCorp Vault!</pre><p>Now, if you have done everything correctly, the new Vault should have connected to the Consul pod and picked up the data in there. Meaning if we ask the Vault for a status, it should show information regarding the Vault.</p><blockquote><em>Quick Note:<br>$ = command is run from Linux Host/Node<br>/ $ = command is run from inside of pod</em></blockquote><p>Like this:</p><pre>$ oc rsh vault-migrator-0 </pre><pre>/ $ vault status<br>Key                Value<br>---                -----<br>Seal Type          shamir<br>Initialized        true<br>Sealed             true<br>Total Shares       5<br>Threshold          3<br>Unseal Progress    0/3<br>Unseal Nonce       n/a<br>Version            1.4.2<br>HA Enabled         true</pre><p>That indicates the Vault is reading the restored Consul data.</p><p>Now time to convert the Consul data into a Raft backend</p><p>Remaining inside of that Vault pod create a migration file as follows:</p><blockquote>Pro Tip! Vault container contains the “vi” command</blockquote><blockquote>vi /tmp/migrate.hcl</blockquote><pre>storage_source &quot;consul&quot; {<br>  address = &quot;consul-consul-server:8500&quot;<br>  path    = &quot;vault&quot;<br>}</pre><pre>storage_destination &quot;raft&quot; {<br>  path = &quot;/vault/data&quot;<br>}</pre><pre>cluster_addr = &quot;<a href="http://127.0.0.1:8201/">http://127.0.0.1:8201</a>&quot;</pre><p>And now we migrate the data!!!</p><pre>/ $ vault operator migrate -config /tmp/migrate.hcl</pre><p>If everything went well you should see this following</p><pre>...<br>2020-07-16T14:18:26.995Z [INFO]  copied key: path=sys/token/parent/3561bb287dfd3983d4dc9ab66dea603c390d95d5/fbe49dd49c384e2ae6c7c2f1b1eda0d0310972c9<br>2020-07-16T14:18:27.003Z [INFO]  copied key: path=sys/token/parent/3561bb287dfd3983d4dc9ab66dea603c390d95d5/fd93a9c4670a29792886e61e61a0d38dcced1db5<br>2020-07-16T14:18:27.007Z [INFO]  copied key: path=sys/token/parent/3561bb287dfd3983d4dc9ab66dea603c390d95d5/fdea7a4910319f962d902156ae4349795f2a4d1b<br>2020-07-16T14:18:27.011Z [INFO]  copied key: path=sys/token/salt<br>Success! All of the keys have been migrated.</pre><p>Aaaand (but wait there is more) you will be new files created under /vault/data/</p><pre>/ $ ls -1 /vault/data/</pre><pre>lost+found<br>node-id<br>raft<br>vault.db</pre><p>But remember, we have only migrated the data. We are not using it yet. This Vault is still using Consul. To verify that, try running the following command once you have unsealed the Vault and logged into it.</p><pre>/ $ vault operator raft list-peers</pre><pre>No raft cluster configuration found</pre><p>Next step is to configure Vault to use the Raft backend and not the Consul backend.</p><p>Exit the Vault shell (exit or Crtl+d)</p><p>And edit the configmap for Vault</p><pre>$ oc edit configmaps vault-migrator-config -n vault-migrator</pre><p>Change this section</p><pre>storage &quot;consul&quot; {<br>      path = &quot;vault&quot;<br>      address = &quot;consul-consul-server:8500&quot;<br>    }</pre><p>To this</p><pre>storage &quot;raft&quot; {<br>      path = &quot;/vault/data&quot;<br>    }</pre><p>With that configmap change, delete the pod</p><pre>$ sleep 120 ; oc delete po vault-migrator-0</pre><blockquote><em>Why the sleep? </em><a href="https://github.com/kubernetes/kubernetes/issues/30189"><em>Because configmaps do not update instantaneously</em></a></blockquote><p>With the pod restarted, list try run our Raft command again. First remote shell into the pod, then unseal and login.</p><p>You can even seen the new data once you have finished unsealing the Vault</p><pre>$ oc rsh vault-migrator-0</pre><pre>/ $ vault operator unseal<br>Unseal Key (will be hidden): <br>Key                     Value<br>---                     -----<br>Seal Type               shamir<br>Initialized             true<br>Sealed                  false<br>Total Shares            5<br>Threshold               3<br>Version                 1.4.2<br>Cluster Name            vault-cluster-1acdbff4<br>Cluster ID              5302c4ed-34a8-cdf4-a5ba-3e0e60df2565<br>HA Enabled              true<br>HA Cluster              n/a<br>HA Mode                 standby<br>Active Node Address     &lt;none&gt;<br>Raft Committed Index    676<br>Raft Applied Index      676</pre><p>Check out the Raft Committed Index!!! Such Wow!</p><pre>/ $ vault operator raft list-peers</pre><pre>Node                                    Address           State     Voter<br>----                                    -------           -----     -----<br>b45a5f4e-bfe0-8420-248f-8ed9cea285c2    127.0.0.1:8201    leader    true</pre><p>Now that we finally have our Vault data converted from Consul to Raft we can do a snapshot</p><pre>/ $ vault operator raft snapshot save /tmp/14072020.raft.snapshot</pre><p>Copy that file from the pod, to import into the real Highly Available Vault we actually want to build. Exit the container and copy the file from inside of the container onto your host.</p><pre>/ $ exit</pre><pre>$ oc cp vault-migrator-0:/tmp/14072020.raft.snapshot 14072020.raft.snapshot</pre><h3>Deploying a Highly Available Vault</h3><p>Finally! I can get a new shiny Vault in Openshift. Enough talking! Let’s do this!</p><p>Create a new Project</p><pre>$ oc new-project takeoverworld-app-dev --display-name=&quot;App to take over the World&quot; --description=&quot;Genetically modify 2 lab mice and harness the power of their brains&quot;</pre><p>Next we are going to create a new Vault values file that has a few modifications:</p><ul><li>High Availability Enabled</li><li>Anti Affinity Enabled</li><li>Auditing enabled</li><li>Raft storage backend</li></ul><p>We will have 3 pods running (High Availability), on 3 separate nodes (Anti Affinity), using the Raft storage backend thereby eliminating the need to deploy Consul.</p><p>In the values below we made the PVC 1Gb. Adjust the size as you require.</p><p>Create a file called: vault.values.yaml</p><pre>global:<br>  enabled: true<br>  tlsDisable: true<br>  openshift: true<br>injector:<br>  enabled: true<br>  resources:<br>    requests:<br>      memory: 256Mi<br>      cpu: 50m<br>    limits:<br>      memory: 256Mi<br>      cpu: 500m<br>server:<br>  resources:<br>    requests:<br>      memory: 256Mi<br>      cpu: 50m<br>    limits:<br>      memory: 256Mi<br>      cpu: 500m<br>  authDelegator:<br>    enabled: true<br>  readinessProbe:<br>    enabled: true<br>  livenessProbe:<br>    enabled: false<br>    path: &quot;/v1/sys/health?standbyok=true&quot;<br>    initialDelaySeconds: 60<br>  service:<br>    enabled: true<br>    type: ClusterIP<br>  dataStorage:<br>    enabled: true<br>    size: 1Gi<br>    storageClass: null<br>    accessMode: ReadWriteOnce<br>  auditStorage:<br>    enabled: true<br>    size: 1Gi<br>    storageClass: null<br>    accessMode: ReadWriteOnce<br>  route:<br>    enabled: false<br>  affinity: |<br>    podAntiAffinity:<br>      requiredDuringSchedulingIgnoredDuringExecution:<br>        - labelSelector:<br>            matchLabels:<br>              app.kubernetes.io/name: {{ template &quot;vault.name&quot; . }}<br>              app.kubernetes.io/instance: &quot;{{ .Release.Name }}&quot;<br>              component: server<br>          topologyKey: kubernetes.io/hostname<br>  ha:<br>    enabled: true<br>    replicas: 3<br>    raft:<br>      enabled: true<br>      setNodeId: false<br>      config: |<br>        ui = true<br>        listener &quot;tcp&quot; {<br>          tls_disable = 1<br>          address = &quot;[::]:8200&quot;<br>          cluster_address = &quot;[::]:8201&quot;<br>        }<br>        storage &quot;raft&quot; {<br>          path = &quot;/vault/data&quot;<br>        }<br>        service_registration &quot;kubernetes&quot; {}<br>  dev:<br>    enabled: false<br>ui:<br>  enabled: true<br>  serviceType: &quot;ClusterIP&quot;</pre><p>Install a new Vault using that values file into the new project</p><pre>$ helm install pinky hashicorp/vault -n takeoverworld-app-dev --values vault.values.yaml</pre><p>That will create your Vault pods for you</p><pre>oc get po -n takeoverworld-app-dev</pre><pre>NAME                                          READY   STATUS    RESTARTS   AGE<br>pinky-vault-0                                 0/1     Running   0          2m32s<br>pinky-vault-1                                 0/1     Running   0          2m32s<br>pinky-vault-2                                 0/1     Running   0          2m31s<br>pinky-vault-agent-injector-6d6df4cff5-r9dpp   1/1     Running   0          2m32s</pre><p>Notice the name “<strong>pinky</strong>”. You cannot install multiple Vaults using the same name. If I created a completely separate project and tried to install a Vault with the name “pinky”, it would fail…….like this…..</p><pre>$ oc new-project takeoverworld-app-qa<br>$ helm install pinky hashicorp/vault -n takeoverworld-app-qa --values vault.values.yaml</pre><pre>Error: rendered manifests contain a resource that already exists. Unable to continue with install: existing resource conflict: kind: ClusterRole, namespace: , name: pinky-vault-agent-injector-clusterrole</pre><p>However, if I changed the name, the Vault will install</p><pre>helm install brain hashicorp/vault -n takeoverworld-app-qa --values vault.values.yaml<br>NAME: brain<br>LAST DEPLOYED: Tue Jul 21 15:20:16 2020<br>NAMESPACE: takeoverworld-app-qa<br>STATUS: deployed<br>REVISION: 1<br>TEST SUITE: None<br>NOTES:<br>Thank you for installing HashiCorp Vault!</pre><p>This is important to remember if you are planning on installing multiple Vaults on the same cluster.</p><h3>Restore Raft snapshot to new Highly Available Vault</h3><p>Now that we have a new running Vault we can restore the data from the file <strong>14072020.raft.snapshot</strong>. Here are the steps</p><p>Copy the file into the first pod (anybody is really fine)</p><pre>$ oc cp 14072020.raft.snapshot pinky-vault-0:/tmp/14072020.raft.snapshot</pre><p>Remote shell into the pod and initialize the vault. This information is not important as we will be overwriting it once we restore from the snapshot.</p><pre>$ oc rsh pinky-vault-0</pre><pre>/ $ vault operator init</pre><pre>Unseal Key 1: eP8UV0T2/I6ewEbJ1zbd3N2aL7pn3GAsUIK11DZhdyns<br>Unseal Key 2: 7qT1a4oEJLi0Lvh9Z5YLt+CGtkdUlgIdAn901R+mOlON<br>Unseal Key 3: j7a2vXPT8DAlG/9JsDcgHVRfhtgvWNygYIU3P0csyp9H<br>Unseal Key 4: QQBc72pwxTGRZ/u1DpdKLAQn5tfY4vmvIfxLn6JAIQQf<br>Unseal Key 5: rnvZLph8cZqL3+kyU5SVb+2UoK/XXbXcDctu8hPfzGd/</pre><pre>Initial Root Token: s.NCjg9ZbclDTfetGkSG0BvKIu</pre><pre>Vault initialized with 5 key shares and a key threshold of 3. Please securely<br>distribute the key shares printed above. When the Vault is re-sealed,<br>restarted, or stopped, you must supply at least 3 of these keys to unseal it<br>before it can start servicing requests.</pre><pre>Vault does not store the generated master key. Without at least 3 key to<br>reconstruct the master key, Vault will remain permanently sealed!</pre><pre>It is possible to generate new unseal keys, provided you have a quorum of<br>existing unseal keys shares. See &quot;vault operator rekey&quot; for more information.</pre><p>Unseal the Vault with the information you just received</p><pre>/ $ vault operator unseal</pre><pre>Unseal Key (will be hidden): <br>Key                     Value<br>---                     -----<br>Seal Type               shamir<br>Initialized             true<br>Sealed                  false<br>Total Shares            5<br>Threshold               3<br>Version                 1.4.2<br>Cluster Name            vault-cluster-953f7dab<br>Cluster ID              c2bd40d0-5b94-cdd8-88e6-aa07f50793dc<br>HA Enabled              true<br>HA Cluster              n/a<br>HA Mode                 standby<br>Active Node Address     &lt;none&gt;<br>Raft Committed Index    24<br>Raft Applied Index      24</pre><p>Log into the Vault using the Root Token</p><pre>/ $ vault login</pre><pre>Token (will be hidden): <br>Success! You are now authenticated. The token information displayed below<br>is already stored in the token helper. You do NOT need to run &quot;vault login&quot;<br>again. Future Vault requests will automatically use this token.</pre><pre>Key                  Value<br>---                  -----<br>token                s.NCjg9ZbclDTfetGkSG0BvKIu<br>token_accessor       PKnSprUq4D5X6macj6V2QL1x<br>token_duration       ∞<br>token_renewable      false<br>token_policies       [&quot;root&quot;]<br>identity_policies    []<br>policies             [&quot;root&quot;]</pre><p>Finally! Finally we have arrived!</p><p>Restore the snapshot!!!</p><pre>/ $ vault operator raft snapshot restore -force /tmp/14072020.raft.snapshot</pre><p>And you get zero output. Lacks a bit of flair, I agree, oh well.</p><p>Log out and delete all the pods</p><pre>/ $ exit</pre><pre>$ oc delete po --all -n takeoverworld-app-dev</pre><p>Once all the pods have restarted, remote console into the Vault and Unseal it using the original keys from the restored snapshot.</p><pre>$ oc rsh pinky-vault-0</pre><pre>/ $ vault operator unseal</pre><pre>Unseal Key (will be hidden): <br>Key                     Value<br>---                     -----<br>Seal Type               shamir<br>Initialized             true<br>Sealed                  false<br>Total Shares            5<br>Threshold               3<br>Version                 1.4.2<br>Cluster Name            vault-cluster-1acdbff4<br>Cluster ID              5302c4ed-34a8-cdf4-a5ba-3e0e60df2565<br>HA Enabled              true<br>HA Cluster              <a href="https://vault-migrator-0.vault-migrator-internal:8201/">https://vault-migrator-0.vault-migrator-internal:8201</a><br>HA Mode                 standby<br>Active Node Address     <a href="http://10.30.18.11:8200/">http://10.30.18.11:8200</a><br><strong>Raft Committed Index    691<br>Raft Applied Index      691</strong></pre><p>Check out the new Raft Committed Index! That’s 691 baby!</p><h3>Join Rafts Together</h3><p>Time to join the other pods together</p><p>Remaining in the pod, list the raft peers</p><pre>/ $ vault operator raft list-peers</pre><pre>Node                                    Address                                    State     Voter<br>----                                    -------                                    -----     -----<br>68e7d28e-aa44-f2d8-3be0-7e624b9ecf32    pinky-vault-0.pinky-vault-internal:8201    leader    true</pre><p>There we are given the name we need to join the other pods to this first pod</p><blockquote>Take note the port is <strong>8200</strong> and the protocol is <strong>http</strong></blockquote><pre>$ oc exec -ti pinky-vault-1 -- vault operator raft join <a href="http://pinky-vault-0.pinky-vault-internal:8200/">http://pinky-vault-0.pinky-vault-internal:8200</a></pre><pre>Key       Value<br>---       -----<br>Joined    true</pre><p>And again</p><pre>$ oc exec -ti pinky-vault-2 -- vault operator raft join <a href="http://pinky-vault-0.pinky-vault-internal:8200/">http://pinky-vault-0.pinky-vault-internal:8200</a></pre><pre>Key       Value<br>---       -----<br>Joined    true</pre><p>Now unseal pinky-vault-1</p><pre>$ oc exec -ti pinky-vault-1 -- vault operator unseal</pre><pre>Unseal Key (will be hidden): <br>Key                     Value<br>---                     -----<br>Seal Type               shamir<br>Initialized             true<br>Sealed                  false<br>Total Shares            5<br>Threshold               3<br>Version                 1.4.2<br>Cluster Name            vault-cluster-1acdbff4<br>Cluster ID              5302c4ed-34a8-cdf4-a5ba-3e0e60df2565<br>HA Enabled              true<br>HA Cluster              <a href="https://pinky-vault-0.pinky-vault-internal:8201/">https://pinky-vault-0.pinky-vault-internal:8201</a><br>HA Mode                 standby<br>Active Node Address     <a href="http://10.30.6.14:8200/">http://10.30.6.14:8200</a><br>Raft Committed Index    770<br>Raft Applied Index      770</pre><p>You will now be able to see this pod as a follower on the raft</p><pre>$ oc exec -ti pinky-vault-0 -- vault operator raft list-peers</pre><pre>Node                                    Address                                    State       Voter<br>----                                    -------                                    -----       -----<br>68e7d28e-aa44-f2d8-3be0-7e624b9ecf32    pinky-vault-0.pinky-vault-internal:8201    leader      true<br>3f9eaff0-ae8c-3470-0ca9-7e68eeedf6c8    pinky-vault-1.pinky-vault-internal:8201    follower    true</pre><p>And you will see the pod is now ready to receive traffic</p><pre>$ oc get po -n takeoverworld-app-dev</pre><pre>NAME                                          READY   STATUS    RESTARTS   AGE<br>pinky-vault-0                                 1/1     Running   0          39m<br>pinky-vault-1                                 1/1     Running   0          39m<br>pinky-vault-2                                 0/1     Running   0          39m<br>pinky-vault-agent-injector-6d6df4cff5-6j7rz   1/1     Running   0          39m</pre><p>Finally (I know I’ve said that a bunch of times), unseal the last Vault</p><pre>$ oc exec -ti pinky-vault-2 -- vault operator unseal</pre><pre>Unseal Key (will be hidden): <br>Key                     Value<br>---                     -----<br>Seal Type               shamir<br>Initialized             true<br>Sealed                  false<br>Total Shares            5<br>Threshold               3<br>Version                 1.4.2<br>Cluster Name            vault-cluster-1acdbff4<br>Cluster ID              5302c4ed-34a8-cdf4-a5ba-3e0e60df2565<br>HA Enabled              true<br>HA Cluster              <a href="https://pinky-vault-0.pinky-vault-internal:8201/">https://pinky-vault-0.pinky-vault-internal:8201</a><br>HA Mode                 standby<br>Active Node Address     <a href="http://10.30.6.14:8200/">http://10.30.6.14:8200</a><br>Raft Committed Index    777<br>Raft Applied Index      777</pre><p>We can now see all our pods talking to each other</p><pre>$ oc exec -ti pinky-vault-0 -- vault operator raft list-peers</pre><pre>Node                                    Address                                    State       Voter<br>----                                    -------                                    -----       -----<br>68e7d28e-aa44-f2d8-3be0-7e624b9ecf32    pinky-vault-0.pinky-vault-internal:8201    leader      true<br>3f9eaff0-ae8c-3470-0ca9-7e68eeedf6c8    pinky-vault-1.pinky-vault-internal:8201    follower    true<br>c4330b82-8508-73c2-6d0c-f14bdedb3e2e    pinky-vault-2.pinky-vault-internal:8201    follower    true</pre><h3>Creating an external Route</h3><p>For the peeps that were paying attention, when we created the Helm values file for the highly available Vault deployment, we told it not to create an Openshift route. When telling the Vault values file to create the route for Openshift, it setup the route as “passthrough” which was not working for me, which is why I disabled it.</p><p>To create a working route, with the certificate termination occurring on the Openshift routers, you can run the following command</p><pre># oc create route edge pinky-vault --service=pinky-vault-active --insecure-policy=Redirect --hostname=pinky-vault.lsd.co.za</pre><p>And boom, we have a UI</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KNAkffKHeQiVwL6FdKSHIg.png" /></figure><h3>Testing out High Availability</h3><p>Let’s make sure this Vault HA stuff actually works</p><p>Check the current status of the Vault</p><pre>$ oc exec -ti pinky-vault-0 -- vault status</pre><pre>Key                     Value<br>---                     -----<br>Seal Type               shamir<br>Initialized             true<br>Sealed                  false<br>Total Shares            5<br>Threshold               3<br>Version                 1.4.2<br>Cluster Name            vault-cluster-1acdbff4<br>Cluster ID              5302c4ed-34a8-cdf4-a5ba-3e0e60df2565<br>HA Enabled              true<br>HA Cluster              <a href="https://pinky-vault-0.pinky-vault-internal:8201/">https://pinky-vault-0.pinky-vault-internal:8201</a><br>HA Mode                 <strong>active</strong><br>Raft Committed Index    812<br>Raft Applied Index      812</pre><p>We can see “pinky-vault-0” is Active node</p><p>If we look at the other pods, they are in Standby mode</p><pre>$ oc exec -ti pinky-vault-1 -- vault status</pre><pre>Key                     Value<br>---                     -----<br>Seal Type               shamir<br>Initialized             true<br>Sealed                  false<br>Total Shares            5<br>Threshold               3<br>Version                 1.4.2<br>Cluster Name            vault-cluster-1acdbff4<br>Cluster ID              5302c4ed-34a8-cdf4-a5ba-3e0e60df2565<br>HA Enabled              true<br>HA Cluster              <a href="https://pinky-vault-0.pinky-vault-internal:8201/">https://pinky-vault-0.pinky-vault-internal:8201</a><br>HA Mode                 standby<br>Active Node Address     <a href="http://10.30.6.14:8200/">http://10.30.6.14:8200</a><br>Raft Committed Index    815<br>Raft Applied Index      815</pre><p>Let’s delete “pinky-vault-0” and see what happens</p><pre>$ oc delete po pinky-vault-0 <br>pod &quot;pinky-vault-0&quot; deleted</pre><pre>$ oc get po </pre><pre>NAME                                          READY   STATUS    RESTARTS   AGE<br>pinky-vault-0                                 0/1     Running   0          2m34s<br>pinky-vault-1                                 1/1     Running   0          63m<br>pinky-vault-2                                 1/1     Running   0          63m<br>pinky-vault-agent-injector-6d6df4cff5-6j7rz   1/1     Running   0          63m</pre><p>We can see “pinky-vault-0” is not ready. That is because the Vault is still sealed</p><pre>$ oc exec -ti pinky-vault-0 -- vault status</pre><pre>Key                Value<br>---                -----<br>Seal Type          shamir<br>Initialized        true<br>Sealed             true<br>Total Shares       5<br>Threshold          3<br>Unseal Progress    0/3<br>Unseal Nonce       n/a<br>Version            1.4.2<br>HA Enabled         true</pre><p>We can determine the new leader by asking one of the other pods</p><pre>$ oc exec -ti pinky-vault-2 -- vault operator raft list-peers</pre><pre>Node                                    Address                                    State       Voter<br>----                                    -------                                    -----       -----<br>68e7d28e-aa44-f2d8-3be0-7e624b9ecf32    pinky-vault-0.pinky-vault-internal:8201    follower    true<br>3f9eaff0-ae8c-3470-0ca9-7e68eeedf6c8    pinky-vault-1.pinky-vault-internal:8201    leader      true<br>c4330b82-8508-73c2-6d0c-f14bdedb3e2e    pinky-vault-2.pinky-vault-internal:8201    follower    true</pre><p>We can see “pinky-vault-1” became leader.</p><p>Let’s verify that</p><pre>$ oc exec -ti pinky-vault-1 -- vault status</pre><pre>Key                     Value<br>---                     -----<br>Seal Type               shamir<br>Initialized             true<br>Sealed                  false<br>Total Shares            5<br>Threshold               3<br>Version                 1.4.2<br>Cluster Name            vault-cluster-1acdbff4<br>Cluster ID              5302c4ed-34a8-cdf4-a5ba-3e0e60df2565<br>HA Enabled              true<br>HA Cluster              <a href="https://pinky-vault-1.pinky-vault-internal:8201/">https://pinky-vault-1.pinky-vault-internal:8201</a><br>HA Mode                 active<br>Raft Committed Index    833<br>Raft Applied Index      833</pre><p>That confirms we have a new leader, and you can double verify by looking at some information from the Web UI</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*14ZQByQKdF_FO6p_qNcXeg.png" /></figure><p>We have HA!</p><p>Are you not entertained!</p><p>Finally (yes, I know) unseal the Vault we just killed</p><pre>$ oc exec -ti pinky-vault-0 -- vault operator unseal</pre><p>You know have a Highly Available Vault, using the Raft backend Storage, running on Openshift.</p><p><a href="https://www.youtube.com/watch?v=79DijItQXMM">You’re Welcome!</a></p><h3>Cleaning Up</h3><p>You can clean up your Helm deployments from the vault-migrator project as follows</p><pre>$ helm uninstall consul -n vault-migrator -n vault-migrator<br>$ helm uninstall vault-migrator -n vault-migrator -n vault-migrator</pre><p>Helm does not remove persistent volume claims</p><pre>$ oc delete pvc data-vault-migrator-consul-consul-server-0 -n vault-migrator<br>$ oc delete pvc data-vault-migrator-0 -n vault-migrator</pre><p>Or if you wanna whack the entire project</p><pre>$ oc delete project vault-migrator</pre><h3>Troubleshooting</h3><p>If your Consul pods are not starting up, and you are seeing this error</p><pre>LAST SEEN   TYPE      REASON                  OBJECT                                                             MESSAGE<br>8s          Normal    SuccessfulCreate        statefulset/consul-consul-server                                   create Claim data-vault-migrator-consul-consul-server-0 Pod consul-consul-server-0 in StatefulSet consul-consul-server success<br>2s          Warning   FailedCreate            statefulset/consul-consul-server                                   create Pod consul-consul-server-0 in StatefulSet consul-consul-server failed error: pods &quot;consul-consul-server-0&quot; is forbidden: unable to validate against any security context constraint: [fsGroup: Invalid value: []int64{1000}: 1000 is not an allowed group]<br>2s          Warning   FailedCreate            daemonset/consul-consul                                            Error creating: pods &quot;consul-consul-&quot; is forbidden: unable to validate against any security context constraint: [spec.containers[0].securityContext.containers[0].hostPort: Invalid value: 8500: Host ports are not allowed to be used spec.containers[0].securityContext.containers[0].hostPort: Invalid value: 8502: Host ports are not allowed to be used]<br>6s          Normal    ProvisioningSucceeded   persistentvolumeclaim/data-vault-migrator-consul-consul-server-0   Successfully provisioned volume pvc-bbf06b8f-f004-4025-8ec8-39d258c282d5 using kubernetes.io/vsphere-volume</pre><p>Then you haven’t enabled the SecurityContextConstraint (SCC) to allow Consul to run as any user ID (anyuid)</p><p>If the Helm install fails with an error like below</p><pre>$ helm install pinky hashicorp/vault -n takeoverworld-app-dev --values vault.values.yaml</pre><pre>Error: Operation cannot be fulfilled on resourcequotas &quot;resources&quot;: the object has been modified; please apply your changes to the latest version and try again</pre><p>I simply uninstalled the chart and tried again</p><pre>$ helm uninstall pinky -n takeoverworld-app-dev<br>$ helm install pinky hashicorp/vault -n takeoverworld-app-dev --values vault.values.yaml</pre><pre>NAME: pinky<br>LAST DEPLOYED: Tue Jul 21 15:14:41 2020<br>NAMESPACE: takeoverworld-app-dev<br>STATUS: deployed<br>REVISION: 1<br>TEST SUITE: None<br>NOTES:<br>Thank you for installing HashiCorp Vault!</pre><p>If you receive the following error while trying to migrate data when running the command</p><pre>$ vault operator migrate -config migrate.hcl<br>...<br>...<br>...<br>...<br>Error migrating: error reading entry: Unexpected response code: 400</pre><p>Then you have a corrupt key/value in our original Consul. I had to delete the key from the original server, export again and try the migration again.</p><h3>What is LSD?</h3><p>If you saw lsd.co.za and were curious….well….</p><p><a href="https://lsd.co.za/">LSD</a> is an open source focused company that helps companies along their journey into the cloud, or what we like to call the LSD Trip.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c5eb68f4b05c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>