<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Michael Courcy on Medium]]></title>
        <description><![CDATA[Stories by Michael Courcy on Medium]]></description>
        <link>https://medium.com/@michael.courcy?source=rss-62bfc22e8124------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*lcJeO3ydOrSM1ta3.</url>
            <title>Stories by Michael Courcy on Medium</title>
            <link>https://medium.com/@michael.courcy?source=rss-62bfc22e8124------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 08:16:58 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@michael.courcy/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Kubernetes is not stateless, you need a backup tool]]></title>
            <link>https://medium.com/@michael.courcy/kubernetes-is-not-stateless-you-need-a-backup-tool-4299478a07fd?source=rss-62bfc22e8124------2</link>
            <guid isPermaLink="false">https://medium.com/p/4299478a07fd</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[backup]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Michael Courcy]]></dc:creator>
            <pubDate>Sat, 11 Nov 2023 14:23:36 GMT</pubDate>
            <atom:updated>2023-11-13T16:20:56.602Z</atom:updated>
            <content:encoded><![CDATA[<h3>A strange assumption</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QsQYoYStFkM1s_lgN30SEA.jpeg" /></figure><p>We often hear from customer and prospects using Kubernetes this strange assumption:</p><p><strong><em>With Kubernetes everything is Gitops and Stateless now!</em></strong></p><p>And consequently:</p><p><strong><em>Do I need a Kubernetes backup tool at all now that everything is “Gitops” and that all my workloads are “Stateless”?</em></strong></p><p>I want to show you that this is a terrible misbelief that user at the beginning of their journey on Kubernetes made up. They have the hope that now Disaster Recovery management is just a matter of relaunching a tool chain and they don’t need to invest in any backup tool.</p><blockquote>There is a confusion between serverless and stateless, from the developer standpoint kubernetes is serverless but absolutely not stateless...</blockquote><p>But before digging into this let’s be clear on these different words: Gitops, Stateless and Toolchain.</p><h3>Gitops and Stateless</h3><p>Gitops is a Devops practice of using GIT and CI/CD tool to apply infrastructure automation. You declare your infrastructure by committing new code change in GIT and the CI/CD tool deploy/apply your change automatically.</p><p>Stateless means application doesn’t have persistent value, if you redeploy the application from scratch it continues to work as before. A stateless application does not maintain data on any storage media.</p><p>Toolchain is the is the set of tools that fetch the code from GIT and apply different actions with this code that build up your infrastructure. The toolchain is said idempotent if it produces a deterministic result that does not depend on the state of the previous execution. A good toolchain should be idempotent.</p><h3>Containers enforce the feeling of Stateless</h3><p>Containerization enforce this idea of stateless, because a container “contains” all the dependency needed for the application to run. The image defines this list of dependencies, and a container is an ephemeral instance of this image. If you lose the machine that run the container, it’s a non-event, just redeploy from the image a new container instance in another machine. The container runtime will recreate all the files from the image definition, and you’re done for another long run.</p><p>Well, this is not true if the container use a volume. For instance a database container will use a a volume to write its data. In this case the container is stateful. If you loose your volume your database will restart empty.</p><blockquote>Containers are stateless, unless they are stateful. It sounds stupid ? I agree …</blockquote><p>You may be surprised to discover that the <a href="https://www.datadoghq.com/container-report/#6">recent 2023 Datadog report (fact 6)</a> shows that :</p><blockquote>Databases and web servers are the leading workload categories for containers.</blockquote><p>Yes you read it, the leading workload categories for containers are stateful !</p><h3>Kubernetes and the cloud reinforce this feeling</h3><p>With Kubernetes the feeling of “stateless” had reach another threshold, now you don’t even need to remember on which machine you deploy which containers because Kubernetes do it for you and dynamically handle your desired state. Kubernetes give you the strong feeling that you can completely abstract away from the infrastructure only the code matters.</p><p>You still have to define the “desired state” in Kubernetes, like the load balancers to expose your application, the number of replicas, the memory and cpu, the secrets, the config files, etc. But all that is defined in yaml file you apply to Kubernetes and that you maintain in GIT.</p><p>But wait! We still have to build and protect the Kubernetes cluster; this is a complex task right?</p><p>Not anymore! Now the cloud let you build a Kubernetes cluster in a minute. A simple click in the AWS or Azure console, a single “glcoud containers create …” command or just a new cluster definition in GIT with a CI/CD tool connected to the cloud API and you’re good.</p><h3><strong>Now the feeling that everything is “stateless” is skyrocketing !</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nJDSnkoxD1CIPN91mecuMA.jpeg" /></figure><p>And we meet prospects that sincerely looks puzzled when we speak about backup on Kubernetes…</p><p>Time now to touch the ground again and speak of the reality.</p><h3>Stateless application does not exist in reality</h3><p>If you take the application as a whole, you quickly realize that stateless application does not exist in reality. Try to imagine an online shop that does not maintain the orders, that does not maintain the address of the customers. Imagine a bank application that does not manage the transactions. Said like this it may sound ridiculous and obvious but it’s important to reattach to the reality.</p><blockquote><strong>If an application was really stateless, most likely it would be useless.</strong></blockquote><p>So why are we speaking of stateless then? Because part of the application is stateless. For instance, a frontend in nodejs which is stateless is making request to a postgresql database which is stateful. From a functional point of view the whole application is pretty stateful. The fact that you split the applications in two parts one stateless and the other stateful does not mean that you don’t have to manage the data anymore.</p><p><em>Yes but my databases are outside of Kubernetes, my pattern still work right?</em></p><p>If your databases are outside of the Kubernetes cluster, you’re going to face some real challenges that will hit badly your Gitops approach. Let’s see them in detail.</p><blockquote>You want your database to become a Kubernetes citizen as the rest of your components.</blockquote><h3>The co-localization challenges</h3><p>If the database is outside of the Kubernetes cluster, you’re going to face a co-localization challenge that will break your “stateless” approach. Indeed, you can’t afford to put the database far away from your workloads, otherwise you’re going to have serious performance issue.</p><p>So, your database and your kubernetes cluster are on the same network for good reasons. Now you have a disaster that break your infra. It’s easy to rebuild your kubernetes cluster somewhere else (remember it’s full stateless and gitops) but what about your databases? <strong>You have to instantiate new databases machines and reapply your dump. Not very clean, not very “gitops”.</strong></p><p>So what ? Your gitops practice stops where your database starts ? Devops means that dev and ops share their preoccupations, are you not breaking this rule ?</p><h3>The migration challenges</h3><p>Not because of a disaster but you want to migrate to another provider for saving money, the Kubernetes part is easy, but the database part is very risky because you continue to manage this part the old way. <strong>You’re going to balance the money you save with the migration against the risk you take with the databases</strong>. This architecture is really reducing your freedom of choice.</p><h3>The Testability challenges</h3><p>Your developer and QA team need to test the applications with real data, you need to bring back a copy of the database in another machine or group of machines and make sure that the configuration of the test instance is not pointing to the production database. Now you want to multiply the number of dev and QA team, you need to multiply the number of machines and configuration changes. <strong>If the databases were managed in Kubernetes in the same namespace than the application, you would not even think about this problem</strong>. The backup tool would let you restore your application in another place in a minute.</p><h3>The database/application version mismatch challenges</h3><p>You’ll also have to map your image version with your database scheme version. This is not so easy to manage, in my developer career I have seen so many mismatches between a database scheme and an application version. <strong>Unexpected schema change and data transformation corrupts your data and may have dramatic consequences</strong>.</p><h3>The Micro-services challenges</h3><p>Your dev team is very agile and want to evolve to a microservices architecture. This architecture requires to build several databases often with different purpose (for instance elasticsearch, redis, mongo and postgres provide different capacities), but if your databases are managed the old way then it’s very hard to accept this diversity. It multiplies all the challenge we list before by the number of databases and database type. <strong>Most likely you’re going to refuse this change and reinforce the monolithic aspect of your application by making the database the actual monolith.</strong></p><p><strong>If you don’t move your database in Kubernetes you’re going to monolith even more your application as it evolves.</strong></p><h3>The cost challenges</h3><p>Deploying your applications on Kubernetes reduce the cost of the application to a great magnitude. But that won’t be true for the database part. <strong>Kubernetes optimize your compute resources and why database would be an exception ?</strong></p><h3>What we observe on the field</h3><p>For all this reasons, gradually databases will enter your Kubernetes cluster. That’s what we observe in the field.</p><p>A first step is made for the tests and development to allow databases to be deployed in Kubernetes, it’s much cheaper and much easier to manage.</p><p>Then the team notice it works very well and don’t see the points to maintain the database outside of Kubernetes anymore. They want to work with other databases with different capacities, waiting for the DBA team to sync up with them is often too long and they create the new database directly in their application namespace.</p><p>Quickly you find yourself maintaining a “schizophrenic” pattern, part of the database is outside of Kubernetes and the other part is inside.</p><blockquote>You’ll finish by moving most of your databases inside Kubernetes, this is inevitable.</blockquote><p>Now you need a strong Kubernetes Backup tool…</p><h3>Everything is Gitops is …. not true (most of the time)</h3><p>In theory everything is code, at all the level you do automation with the “As Code” spirit, in other words you try to be 100% declarative.</p><p>For instance:</p><ul><li>You use terraform code to create your network, cloud services, Kubernetes clusters and so on</li><li>You use Argo CD to deploy your main kubernetes tool like cert-manager, ISTIO and so on.</li><li>You use Tekton to build, test and push your applications images</li><li>You use helm chart to deploy your applications with their specific configurations</li></ul><p>All that is great and of course we can only approve the enforcement of those practices. But the reality is not as bright as it looks …</p><ul><li>Building all this chain tools is a big effort; you don’t necessarily have all the human resources for that</li><li>sometime hotfix within an hour is absolutely needed and the chain tool does not handle this case</li><li>Your toolchain has been designed for redeploying too many components at the same time and you cannot allow that, you just want to redeploy a specific component, so you do it manually</li><li>Now you’ve been asked to deploy multiple instances of the same infrastructure but some of the parametrization has not been thought for that, reworking the whole tool chain vs doing the change manually make you choose the later</li><li>The application does not evolve anymore and only one developer is sometime required to fix some bugs; you won’t re-invest in the tool chain. The developer will apply the change manually, this can last for years sometime.</li><li>Many tool chains (maintained by different teams) are targeting a single one application, as the code of the different tool chains evolves it’s not so easy to rebuild the exact state of an application at a given point in time.</li></ul><p>This list is not exhaustive and each time I was seriously digging in any projects, at different level I was able to see that not everything is “As code”. There is always a piece (sometime a big one) of exception that break this theoretical process.</p><blockquote>At the end the source of truth is Kubernetes, and you need a tool that capture that properly.</blockquote><h3>Gitops may break, it happens more than you can imagine</h3><p>All those toolchains (terraform, argocd, tekton … ) even the one provided by the cloud provider (azuredevops, github action, codefresh, AWS …) are just program that execute on machines and they can also break for many reasons. They may also be broken just by human error or by dependencies that don’t work anymore.</p><p>For instance, I remember of a toolchain that was scanning for vulnerabilities in the docker images, this tool must pass all the images to allow the process of deployment to go further. Unfortunately, this tool was temporary broken, and the application has to be restored because the cluster broke for another reason (you know disaster always come together…). Nobody at the time of this urgency knew how to rebuild the tool chain without the security scans. The fact the application has already been deployed did not make any difference if you want to deploy again you had to pass this step.</p><p>It was impossible to restore the application, team had to wait for someone to find out how to rebuild the toolchain without the security scan. At the end the SLA requirement was not met.</p><blockquote><strong>And the team decided to invest in a backup tool that can reinstall the app independently of the toolchain.</strong></blockquote><p>Also hacker knows very well the importance of the GIT repository and of the tools chain, they may decide to corrupt or destroy them. If that happens you’ll have to repair/rebuild them before being able to reuse. This can have a serious impact on your RTO. If you loose completely your git repo you’ll have to wake up one of your developer in the middle of the night and ask them if by any chance they still have the main branch on their laptop. If you think that never happens, think twice …</p><blockquote>The Gitops tool chain is for development and deployment it’s not a backup tool.</blockquote><p>Kasten for instance feature immutable backup that guarantee that even a rogue admin won’t be able to destroy your backups. GIT and the Tools chain have not been designed for that. <strong>Remember most of the attacks are internal.</strong></p><h3>Operators are changing the game</h3><p>Operator are controller on Kubernetes dedicated to databases. Nearly all major database vendors are now proposing their operator version. The operator encapsulates the knowledge of database expert and make database management on Kubernetes very simple and <strong>supported</strong>.</p><p>The operator will help you to scale in or scale out. You’ll just change a field in a custom resource (for instance the number of replicas) and the operator will do all the complex operation to meet the desired state without interrupting the service.</p><p><strong>With Operator there is no reason anymore to not move your database in Kubernetes if you trust the vendor.</strong> But one thing must be observed, is that operator create a lot of changes in the Kubernetes (secrets, certificates, PVC …), and more than ever you need a backup tool that can work with the operator API. <strong>Kasten feature an extension mechanism based on Kanister that make this coordination between the operator API and the backup action very easy.</strong></p><h3>Kasten is not denying Gitops practice, on the contrary !</h3><p>To conclude on this topic the goal was not to deny the value brought by Gitops practices. At Kasten we deploy every 2 weeks running tons of automated deployments and automated tests. All that would not be possible if we did not embrace the Devops (including Gitops) practices.</p><p>I also show in this <a href="https://github.com/michaelcourcy/kasten-tekton#test-backup-before-deploy">Tekton demo</a> how to include a Kasten Backup Action to capture a snapshot of your application before deploying a new version. There is also an <a href="https://github.com/michaelcourcy/pacman/blob/main/azure-pipelines.yml">azuredevops example</a> that does the same think but with azuredevops task. So Kasten work very well with Gitops practices.</p><p>But it would be a mistake to think that now because you are Gitops you don’t need a backup tool on kubernetes:</p><h3>Key takeaways</h3><ul><li>You still need to capture the source of truth which is at the end Kubernetes and nothing else.</li><li>Your database will end up sooner or later in Kubernetes because it makes your application and data management much simpler to operate and reduce the cost. Hence, you’ll back it up with the rest of the application stack.</li><li>If everything breaks at the same time you need a plan B to rebuild somewhere else quickly without depending on development resources.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4299478a07fd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My first 2 hours with Elixir]]></title>
            <link>https://medium.com/@michael.courcy/my-first-2-hours-with-elixir-9480cd985b40?source=rss-62bfc22e8124------2</link>
            <guid isPermaLink="false">https://medium.com/p/9480cd985b40</guid>
            <category><![CDATA[elixir]]></category>
            <dc:creator><![CDATA[Michael Courcy]]></dc:creator>
            <pubDate>Sun, 29 Dec 2019 02:21:56 GMT</pubDate>
            <atom:updated>2019-12-29T02:24:39.162Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*cXXqb-5prpRa5fKFonjIOQ.jpeg" /></figure><p>I read a huge buzz about Elixir. It’s supposed to be very easy to read, fun to write, scaling very well. This language also claims to be purely functional and its data structure are immutable making it naturally thread safe and parallel safe. Beside the learning curve is not steep.</p><p>Every time I see this kind of praise about a new language I don’t give too much credit thinking “yet another language”, “too good to be true”. But laudatory comments were repeating very often and this morning after reading another very enthusiastic experience on <a href="https://www.quora.com/What-makes-Elixir-better">quora</a> I decided to give it a try.</p><p>When I try a new language I give it 2 hours reading and I test how I can implement the Guess Number game.</p><p>Guess Number is a simple game where the program choose randomly a number between 1 and 1000 and you have a maximum of 10 attempts to find it. This program is not complicated to write but you have to manage loop, exit condition, input and output from the user. How elegant you can produce the solution after 2 hours learning the language is often a good sign of its “developper friendly” nature.</p><p>And I must say that actually I was positively surprised . I immediately use pattern matching and pipe. Pattern matching help me to separate game rule from control flow in two distinct functions:</p><pre>def tryAttempt(attempt, actual) when attempt &lt; 10 do<br>  {guess, _} = IO.gets(&quot;enter your number : &quot;) <strong>|&gt;</strong> Integer.parse<br>  case evaluate(guess, actual) do<br>       {<strong>:good</strong>, message} -&gt; success(message, attempt)<br>       {<strong>:bad</strong>, message} -&gt; fail(message, attempt, actual)<br>   end<br>end</pre><pre>def evaluate(guess, actual) do<br>  cond do<br>     guess == actual -&gt; {<strong>:good</strong> ,&quot;You found it&quot;}<br>     guess &gt; actual -&gt; {<strong>:bad</strong> ,&quot;The number is smaller&quot;}<br>     guess &lt; actual -&gt; {<strong>:bad</strong> ,&quot;The number is bigger&quot;}<br>   end<br>end</pre><p>At the end no need to use a “for” or “while” loop as I usually would do in other language, I was forced to use recursion. The bold show how the function “tryAttempt” and “fail” recursively call each other. You can do that with any other language but because i did not find in this 2 hours how to make a loop I was forced to use recursion and I was surprised to find that easier to read actually.</p><pre>def <strong>tryAttempt</strong>(attempt, actual) when attempt &lt; 10 do<br>  {guess, _} = IO.gets(&quot;enter your number : &quot;) |&gt; Integer.parse<br>  case evaluate(guess, actual) do<br>       {:good, message} -&gt; success(message, attempt)<br>       {:bad, message} -&gt; <strong>fail</strong>(message, attempt, actual)<br>   end<br>end</pre><pre>def <strong>fail</strong>(message, attempt, actual) do<br>  IO.puts(&quot;#{message}  #{10 - attempt} attempts remaining&quot;)<br>  <strong>tryAttempt</strong>(attempt + 1, actual)<br>end</pre><p>Finally the guard condition let me write another definition of tryAttempt when the number of attempt is 10, this make the code much more readable than introducing a if inside the function because you do completely different thing in this case.</p><pre>def tryAttempt(attempt, actual) <strong>when attempt == 10</strong> do<br>   IO.puts(&quot;You have reached the 10 attempts sorry the number was #{actual}&quot;)<br>end</pre><p>Of course with two hours I just scratched the surface and I did not try OTP, process and distributed tasks but the first impression was cool : yes elixir let you write easy to write and read programs.</p><p>Here is the complete code :</p><pre>defmodule GuessNumber do<br>  <br>  def tryAttempt(attempt, actual) <strong>when attempt == 10</strong> do<br>    IO.puts(&quot;You have reached the 10 attempts sorry the number was #{actual}&quot;)<br>  end</pre><pre>  def <strong>tryAttempt</strong>(attempt, actual) when attempt &lt; 10 do<br>    {guess, _} = IO.gets(&quot;enter your number : &quot;) <strong>|&gt;</strong> Integer.parse<br>    case evaluate(guess, actual) do<br>         {<strong>:good</strong>, message} -&gt; success(message, attempt)<br>         {<strong>:bad</strong>, message} -&gt; fail(message, attempt, actual)<br>     end<br>  end</pre><pre>  def evaluate(guess, actual) do<br>    cond do<br>       guess == actual -&gt; {<strong>:good</strong> ,&quot;You found it&quot;}<br>       guess &gt; actual -&gt; {<strong>:bad</strong> ,&quot;The number is smaller&quot;}<br>       guess &lt; actual -&gt; {<strong>:bad</strong> ,&quot;The number is bigger&quot;}<br>    end<br>  end</pre><pre>  def <strong>fail</strong>(message, attempt, actual) do<br>    IO.puts(&quot;#{message}  #{10 - attempt} attempts remaining&quot;)<br>    <strong>tryAttempt</strong>(attempt + 1, actual)<br>  end</pre><pre>  def success(message, attempt) do<br>    IO.puts(&quot;#{message} You dit it in #{attempt} attempts&quot;)<br>  end</pre><pre>end</pre><pre>IO.puts (&quot;Guess my number between 1 and 1000, 10 attempts possible&quot;)<br>actual = Enum.random(1..1000)<br>GuessNumber.tryAttempt(1,actual)</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9480cd985b40" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>