<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gabriel Abdalla Cavalcante on Medium]]></title>
        <description><![CDATA[Stories by Gabriel Abdalla Cavalcante on Medium]]></description>
        <link>https://medium.com/@gcavalcante8808?source=rss-e30210ba8ff9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*upQ3SLpJc7L8Z-fri3Vcrg.jpeg</url>
            <title>Stories by Gabriel Abdalla Cavalcante on Medium</title>
            <link>https://medium.com/@gcavalcante8808?source=rss-e30210ba8ff9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 10:18:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gcavalcante8808/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Semantic search, medieval monks, and why retrieval mode matters]]></title>
            <link>https://blog.sidecloud.ai/semantic-search-medieval-monks-and-why-retrieval-mode-matters-cb2986cfd051?source=rss-e30210ba8ff9------2</link>
            <guid isPermaLink="false">https://medium.com/p/cb2986cfd051</guid>
            <category><![CDATA[umberto-eco]]></category>
            <category><![CDATA[b2b-saas]]></category>
            <category><![CDATA[semantic-search]]></category>
            <category><![CDATA[vector-embeddings]]></category>
            <category><![CDATA[developer-tools]]></category>
            <dc:creator><![CDATA[Gabriel Abdalla Cavalcante]]></dc:creator>
            <pubDate>Thu, 05 Feb 2026 14:23:24 GMT</pubDate>
            <atom:updated>2026-02-05T14:23:24.816Z</atom:updated>
            <content:encoded><![CDATA[<p>This article is an introduction to semantic search, embeddings, a bit of Umberto Eco’s The Name of the Rose history and my own history, using BAAI/bge-m3 open source multi retrieval mode capabilities.</p><blockquote>This is also the first post from <a href="https://short.sidecloud.ai/medium-eco">SideCloud.ai</a> — a company that provides AI Infrastructure for Developers, whose first offering is an embeddings-as-a-service<strong>, a </strong>GDPR-compliant embeddings API built for developers who are working with semantic search but without the infrastructure headaches.</blockquote><blockquote>Our first offering exposes BGE-M3’s full capabilities: dense, sparse, and ColBERT retrieval modes through a simple API, so you can focus on building features instead of managing GPU clusters.</blockquote><p>A Python notebook is available on Google Colab, which includes free usage of a NVIDIA T4 GPU: <a href="https://short.sidecloud.ai/medium-eco-notebook">you can check it here.</a></p><p>By the end of this post, you’ll understand why running this notebook yourself is educational — but running it in production is a different story entirely.</p><p>When I was young, I found the use of erudite language combined with suspense and investigative stories fascinating; authors like Umberto Eco and Agatha Christie always left me <em>euphoric</em>.</p><p>Twenty years have passed; and in those twenty years working with infrastructure, 12 with Python and development, I came across a field that reminded me of that same sense of wonder: NLP, Word2Vec, and semantic search.</p><h3>About the Book — The Name of the Rose</h3><p>The Name of the Rose, written in 1980 by the master Umberto Eco, is an investigative novel that depicts the arrival of the Franciscan monk William of Baskerville and his apprentice at a Benedictine monastery in Italy, in 1327.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*wd3kqYHdS644jfhRUp0fOQ.png" /><figcaption>Willian of Baskerville and his apprentice on the 1986 Movie</figcaption></figure><p>The book itself brings religion, violence, eroticism, dialogues about the sacred at various points, and — nonetheless — humor.</p><h3>About Embeddings — and a quick word about Semantic Search and some technical terms</h3><p><em>Semantic search</em> is relative to delivering results more closely related to searches made by people using predictive functions (AI) and meaning instead of simple keywords.</p><p>To make the search possible, different text chunks are transformed into float vectors with big dimensions, aiming to preserve the meaning of the words in a given text — these vectors are called <em>embeddings.</em></p><p>Considering a Python snippet like the one below, which has the question &quot;What is BGE M3&quot; and uses the open source model &quot;BAAI/bge-m3&quot;:</p><pre>from FlagEmbedding import BGEM3FlagModel<br><br>model = BGEM3FlagModel(&#39;BAAI/bge-m3&#39;, use_fp16=True)<br><br>sentences_1 = [&quot;What is BGE M3?&quot;]</pre><p>The code would generate a 1024 <strong>dimension</strong> NumPy array:</p><pre>&gt;&gt;&gt; model.encode(sentences_1)<br>array([[-0.03411704, -0.04707833, -0.0008944 , ...,  0.04828537,<br>         0.00755428, -0.02961661]], shape=(1, 1024), dtype=float32)<br>&gt;&gt;&gt; len(model.encode(sentences_1)[&#39;dense_vecs&#39;][0])<br>1024</pre><p>With the embeddings generated, the search consists of find similar or related text chunks — which of course are also <em>embeddings.</em></p><blockquote><em>Simply put, texts transformed into vectors that aim to maintain contextual meaning are called </em>embeddings<em>.</em></blockquote><h3>BGE-M3: Open source Model and what we want to accomplish</h3><p>In this post’s we demonstrate the creation of <em>embeddings</em> using 5 passages from Umberto Eco’s “The Name of the Rose” and how well these passages correlate to the query “The sin of flesh and pleasure permeates the name of the rose book.”</p><p>In addition to the usual Python libraries (transformers, scikit-learn, NumPy, etc) we are also going to use an open source multilingual <em>embeddings </em>model called BGE-M3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N3eOOdFEHLarbXwqMcPQ7w.jpeg" /><figcaption>The nice and cute panda from the FlagOpen/FlagEmbedding github repository</figcaption></figure><h4>BGE-M3 Features</h4><p>BGE-M3 is a bit different from what we commonly have on the market: most models have a <em>retrieval mode </em><strong>DENSE </strong>embeddings only: the text is represented as a single dimension vector.</p><p>Besides <strong>DENSE, </strong>there are two other retrieval modes supported by BGE-M3:</p><ul><li><strong>SPARSE: </strong>assign scores for different tokens in the text in a format called<strong> </strong><a href="https://www.pinecone.io/learn/splade/"><strong>SPLADE</strong></a>, which is a keyword alike representation<strong>.</strong></li></ul><p><strong>Eg: </strong>For a question like &quot;What is BGE-M3?&quot; the token/numeric representation can be translated to:</p><pre>[{&#39;What&#39;: 0.08356, &#39;is&#39;: 0.0814, &#39;B&#39;: 0.1296, &#39;GE&#39;: 0.252, &#39;M&#39;: 0.1702, &#39;3&#39;: 0.2695, &#39;?&#39;: 0.04092}]</pre><ul><li><strong>ColBERT:</strong> this one is a heavy hitter, dedicated to capture more subtle details on texts at the expense of memory and storage costs.</li></ul><p>It&#39;s like a DENSE embedding but for each token found in a text, a full dimension vector representation will be computed.</p><p>Eg: A text with 5 tokens that would generate a 1024 vector in DENSE mode, now generates one 1024 dimension vector for each token, resulting in 5120 tokens.</p><blockquote>Being able to encode texts in different embedding types, is what makes BGE-m3 a Multi-Functional model: DENSE, SPARSE and COLBERT embedding retrieval modes are possible with a single model.</blockquote><h3>Python Code</h3><h3>Installation</h3><p>The following Python libraries are required:</p><ul><li>“transformers&lt;5.0”</li><li>“flagembedding”</li><li>“numpy”</li><li>“torch”</li></ul><p>You can install these libraries using pip or uv in a virtualenv.</p><p>If you don’t have an NVIDIA GPU, you can use CPU only: use pip install torch --extra-index-url https://download.pytorch.org/whl/cpu to install the CPU-only version of torch.</p><h3>Setup</h3><p>Ok, Now let’s move on to the actual code. Start by importing the libraries and initialize the “BAAI/bge-m3” model:</p><pre>import numpy as np<br>from FlagEmbedding import BGEM3FlagModel<br><br>model = BGEM3FlagModel(&#39;BAAI/bge-m3&#39;, use_fp16=True)</pre><h3>Data Preparation</h3><p>Proceed to set the question and the 5 text passages used:</p><pre>question = [&quot;Remembrance of sins of flesh and pleasure.&quot;]<br>text_chunks = [<br>    &quot;The day before, Benno had said he would be prepared to sin in order to procure a rare book. He was not lying and not joking. A monk should surely love his books with humility, wishing their good and not the glory of his own curiosity; but what the temptation of adultery is for laymen and the yearning for riches is for secular ecclesiastics, the seduction of knowledge is for monks.&quot;,<br>    &quot;Now I know, as the doctor says, that love can harm the lover when it is excessive. And mine was excessive. I have tried to explain what I felt then, not in the least to justify what I felt. I am speaking of what were my sinful ardors of youth. They were bad, but truth obliges me to say that at the time I felt them to be extremely good. And let this serve to instruct anyone who may fall, as I did, into the nets of temptation. Today, an old man, I would know a thousand ways of evading such seductions.&quot;,<br>    &quot;Now, as I walked in the cold, pale sun of that winter morning, surrounded by the fervor of men and animals, I began to remember my experiences in a different way. As if, from everything that had happened, my repentance and the consoling words of the penitential cleansing no longer remained, but only visions of bodies and human limbs. Into my feverish mind came abruptly the ghost of Berengar, swollen with water, and I shuddered with revulsion and pity. Then, as if to dispel that lemur, my mind turned to other images of which the memory was a fresh receptacle, and I could not avoid seeing, clear before my eyes (the eyes of the soul, but almost as if it appeared before my fleshly eyes), the image of the girl, beautiful and terrible as an army arrayed for battle.&quot;,<br>    &quot;To tell the truth, the other terrible events following my sinful encounter with the girl had caused me almost to forget that occurrence, and once I had confessed to Brother William, my spirit was relieved of the remorse I had felt on waking after my guilty lapse, so it was as if I had handed over to the monk, with my words, the burden itself of which they were the signifying voice. What is the purpose of the holy cleansing of confession-, if not to unload the weight of sin, and the remorse it involves, into the very bosom of our Lord, obtaining with absolution a new and airy lightness of soul, such as to make us forget the body tormented by wickedness? But I was not freed of everything. &quot;,<br>    &quot;In vain. I thought of the girl. My flesh had forgotten the intense pleasure, sinful and fleeting (a base thing), that union with her had given me; but my soul had not forgotten her face, and could not manage to feel that this memory was perverse: rather, it throbbed as if in that face shone all the bliss of creation.&quot;<br>]</pre><p>The question above &quot;Remembrance of sins of flesh and pleasure.&quot; is very interesting considering that the text chunks have different correlated words for &quot;pleasure&quot;, &quot;sin&quot;, &quot;seduction&quot; and so on.</p><p>This is more akin to a daily question with specific and nuanced terms that we may try to ask than a regular keyword search in a search engine. This one of the aspects that make the <em>semantic search </em>useful.</p><figure><img alt="Seduction of knowledge vs the sin of flesh and pleasure: the DENSE retrieval falls into a trap." src="https://cdn-images-1.medium.com/max/1024/1*n3Dvzo8QbYa7ZOmhi4qhtw.jpeg" /><figcaption>Seduction of knowledge vs the sin of flesh and pleasure: Image Generated by Le Chat.</figcaption></figure><h4>Evaluation</h4><blockquote><em>With question and text chunks defined, a score is assigned in a range from 1 (less relevant) to 5 (extremely relevant) for each text chunk:</em></blockquote><pre>true_relevance = np.array([[2, 1, 3, 4, 5]])<br>pairs = [[question[0], c] for c in text_chunks]</pre><p>Then, pairs of (question, text) are prepared: those will be fed to BGE-M3 compute_score<em> </em>method, which computes how much <em>similar </em>the text is to the question<em>:</em></p><pre>&gt;&gt;&gt; result = model.compute_score(pairs, weights_for_different_modes=None)<br>&gt;&gt;&gt; result<br>{&#39;colbert&#39;: [0.5435755252838135, 0.5425188541412354, 0.5512334108352661, 0.5714364647865295, 0.6577926278114319], &#39;sparse&#39;: [0.06500028818845749, 0.050365954637527466, 0.03138704597949982, 0.05373295396566391, 0.15647552907466888], &#39;dense&#39;: [0.51580810546875, 0.5027126669883728, 0.5723879337310791, 0.5443050861358643, 0.6091998815536499], &#39;sparse+dense&#39;: [0.29040420055389404, 0.27653932571411133, 0.30188748240470886, 0.2990190088748932, 0.38283771276474], &#39;colbert+sparse+dense&#39;: [0.3747946321964264, 0.36519917845726013, 0.3850027620792389, 0.38982483744621277, 0.47448936104774475]}</pre><p>Considering the best possible result would a rank were the passages receive greater scores in the order [5,4,3,1,2], we convert the output of the command above in a table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FhNDmJfoIbJe27OYlPnl-A.png" /><figcaption>Table representing the bgem3 compute_score output: Dense, Sparse, Colbert, Sparse+Dense and ALL Modes Results</figcaption></figure><p>With the example above in mind, we at SideCloud.ai believe that human language and its nuances could require more than one method at your disposal in way that different retrieval modes are an excellent resource to have on hand for those working with embeddings and semantic search.</p><blockquote>As the wise say: there is no silver bullet.</blockquote><h3>Closed Beta — Sidecloud.ai</h3><p>Well considering how frustrating it can be to work with AI-related infrastructure, SideCloud.ai is currently running a <strong>closed beta</strong> for developers and teams working with semantic search, RAG systems, or large-scale text retrieval, <strong>with a 3M Dense Tokens for free, or the corresponding amount considering other retrieval modes during the closed beta.</strong></p><p>👉 Learn more or request access at:<br> <a href="https://short.sidecloud.ai/medium-eco"><strong>https://sidecloud.ai</strong></a></p><p>The beta is intentionally small and feedback-driven.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb2986cfd051" width="1" height="1" alt=""><hr><p><a href="https://blog.sidecloud.ai/semantic-search-medieval-monks-and-why-retrieval-mode-matters-cb2986cfd051">Semantic search, medieval monks, and why retrieval mode matters</a> was originally published in <a href="https://blog.sidecloud.ai/">SideCloud.ai</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Builtin Quality for Helm Charts: unit testing to the rescue!]]></title>
            <link>https://medium.com/@gcavalcante8808/builtin-quality-for-helm-charts-unit-testing-to-the-rescue-2cb9d5c1ddc8?source=rss-e30210ba8ff9------2</link>
            <guid isPermaLink="false">https://medium.com/p/2cb9d5c1ddc8</guid>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[yaml]]></category>
            <category><![CDATA[unit-testing]]></category>
            <dc:creator><![CDATA[Gabriel Abdalla Cavalcante]]></dc:creator>
            <pubDate>Sat, 06 Feb 2021 01:16:51 GMT</pubDate>
            <atom:updated>2021-02-06T01:16:51.825Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nLY7KxCL2W4RHAAV.jpg" /></figure><p>As we dive more and more in helm charts and we become true bulwarks of yaml writing (YamlOps engineers!), sometimes there is a need to use some template logic to introduce different behaviors in our helm charts; when it occurs, we have more options to customize helm charts (and often reuse them) but still, we have another problem: with more powers, comes more bugs!</p><p>To avoid some bugs on the template logic or even have guarantees that the behavior will not be changed by accident (regression) in the future, we can leverage <strong>unit tests</strong> as we do in software (a good point here: as stated in the SRE Book from Google Team, “solving operations problems with software has always been central to DevOps” comes to mind as I type).</p><blockquote><em>But Wait! You are talking about unittests and testcases but some of us are not </em><em>developers per se; we just use yamls to describe our infrastructure!</em></blockquote><blockquote><em>In this case, my consecrated fellow, trust me: when you put even a simple </em><em>if somewhere in your code or in your YAML template, brace ourselves: eventually — at some point of the future, it will bite you back!</em></blockquote><h3>Scenario</h3><p>To leverage the concept of testing, we are going to hypothesize a scenario: a company that wants to add support for a CNCF Service Mesh called <strong>Linkerd. </strong>The linkerd is not the topic that will be discussed here, but some conditionals and template logic that we are going to use is part of our day-to-day routines on maintaining helm charts: nested ifs, ranges, ORs, and so on.</p><p>Taking the above scenario as our start point is stated that Linkerd support is achieved by making the following changes:</p><ol><li>a serviceAccount have the attribute automountServiceAccountToken defined as true, which is the default in Kubernetes, but in our pseudo company we set it to false by default for security reasons (you really should do it :D);</li><li>the deployment that will be part of the service mesh, have the label linkerd.io/inject: enabled , which we don&#39;t have in any of our resources, yet.</li></ol><p>With that in mind, we go to:</p><ol><li>Create a new helm chart;</li><li>Change the Service account to accommodate scenarios with and without linkerd support.</li></ol><p>To do the HandsOn Topic you going to need the following software:</p><ul><li>[REQUIRED] Helm v3: You can install it by following the instruction in <a href="https://helm.sh/docs/intro/install/">https://helm.sh/docs/intro/install/</a></li></ul><h3>HandsOn</h3><p>Well, we start by creating a new helm chart with the command helm create medium , which creates a mediumfolder with the following structure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/490/1*FmnOhufKrncLojTWcE_mTA.png" /></figure><p>Thinking on item 1, we need to make changes to ourtemplates/serviceaccount.yamlwhich is untouched/vanilla at this point:</p><pre>{{- if .Values.serviceAccount.create -}}<br>apiVersion: v1<br>kind: ServiceAccount<br>metadata:<br>  name: {{ include &quot;medium.fullname&quot; . }}<br>  labels:<br>    {{- include &quot;medium.labels&quot; . | nindent 4 }}<br>  {{- with .Values.serviceAccount.annotations }}<br>  annotations:<br>    {{- toYaml . | nindent 4 }}<br>  {{- end }}<br>{{- end }}</pre><p>The basic logic that comes in the default template is very simple: it will render the resource only if .Values.serviceAccount.create is validated as a true value.</p><p>As stated before, we need to do something to maintain the behavior expected(automountServiceAccountToken: false) and provide means to activate the token when using linkerd. We can achieve the behavior above with the following YAML:</p><pre>{{- if .Values.serviceAccount.create -}}<br>apiVersion: v1<br>kind: ServiceAccount<br>{{ if .Values.linkerd_support }}<br>automountServiceAccountToken: true<br>{{ else }}<br>automountServiceAccountToken: false<br>{{ end }}<br>[...]</pre><p>Now we inserted a “linkerd_support” value that will act as a switch for the configuration: if true mount service account token otherwise does not mount. Time to test, then:</p><pre>helm template . --set serviceAccount.create=true \<br>                --set linkerd_support=true \<br>                --show-only templates/serviceaccount.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/764/1*Rs4Pqq4Hpsgu4St31GNVAw.png" /></figure><p>Very nice, it works as intended: if serviceAccount.createand linkerd_support are true, then a service account with automountServiceAccountToken:true will be created. At this point, we could start by changing the deployment to support item #2, but we going to stop here and think about some points:</p><blockquote><em>What gives you guarantees that in the future the conditional expressed here will keep working as intended or that a change will not inject some sort of </em><em>regression on this behaviour?</em></blockquote><blockquote><em>That every person that changes the helm chart will run the helm template and check for this (and others) results?</em></blockquote><p>If you came this far in this article, maybe you’re thinking the same as me: unit tests and continuous testing for the rescue!</p><h3>Time to write some tests</h3><p>Before we can dive into the tests we need some other tools than helm3 installed:</p><ul><li>helm unit test plugin: helm plugin install <a href="https://github.com/anikin-aa/helm-unittest">https://github.com/anikin-aa/helm-unittest</a></li></ul><p>With all tooling installed, it&#39;s time to work.</p><p>First, create a folder called tests inside your helm chart root folder; then, inside this new folder, create a file called linkerd_support_test.yaml, with the following content:</p><pre>suite: test that our chart have support for linkerd<br>templates:<br>  - serviceaccount.yaml</pre><p>We start by defining the suite name which is the test case Name and the templates that will compose the test; in this case, serviceaccount.yaml — the template that we changed earlier.</p><p>Before we can test the changes linked with linkerd (Uh-Oh! Bad joke), we going to take into consideration some knowledge spread by Uncle bob and the boy scout rule: “<strong><em>Always leave the campground cleaner than you found it”</em></strong>.</p><p>If we take a look at serviceAccount, the first if of the code can be tested and that will be our first test.</p><p>For that, we going through writing tests by using <a href="https://freecontent.manning.com/making-better-unit-tests-part-1-the-aaa-pattern/">AAA</a> pattern as much as possible. Let&#39;s see it in action:</p><pre>tests:<br>  - it: should have serviceAccount when sa support is ON<br>    set:<br>      serviceAccount:<br>        create: true<br>    asserts:<br>      - hasDocuments:<br>          count: 1<br>      - isKind:<br>          of: ServiceAccount<br>      - equal:<br>          path: automountServiceAccountToken<br>          value: false</pre><p>With the test above, we are doing the following:</p><ol><li>Defining/Arranging some values for the test (which translates roughly to helm template . --set serviceAccount.create=true command);</li><li>Making some assertions that:</li></ol><p>a. There is a document rendered;</p><p>b. The Kind of the resource is a ServiceAccount.</p><p>3. That the automountServiceAccountTokenis false by default when serviceAccountis enabled but linkerd is not.</p><p>Let&#39;s run the tests with helm unittest command:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w4joHoobohFVqCQKOt2a4A.png" /></figure><p>Good, our first test is working.</p><p>Time to create another test, this time we gonna test that automountServiceAccountToken is activated when linkerd_support is true by creating another test (right after the first, the test section expect a list of tests):</p><pre>- it: should have enabled automountServiceAccountToken when linkerd support is ON<br>    set:<br>      linkerd_support: true<br>      serviceAccount:<br>        create: true<br>    asserts:<br>      - hasDocuments:<br>          count: 1<br>      - isKind:<br>          of: ServiceAccount<br>      - equal:<br>          path: automountServiceAccountToken<br>          value: true</pre><p>If we run helm unittest . command again, it works:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xFfa_5BHP7WBc9fdP8GNPQ.png" /></figure><p>A good point here is: we need to set both serviceAccount and linkerd_support to true to comply with the scenario that we described earlier.</p><p>Some questions that occur at this time:</p><blockquote><em>What happens if </em><em>serviceAccount.enabled is false but </em><em>linkerd_support is set to </em><em>true?</em></blockquote><blockquote><em>Should we separate the logic for linkerd support in a manner which it doesn’t interact with other resource logic like the one in the service account?</em></blockquote><blockquote><em>Should we separate a resource of each type, augumented or not with linkerd needs?</em></blockquote><blockquote><em>What if we add support for other service meshes, which often include their own labels and other changes on resources? More “elifs”?</em></blockquote><p>Whatever is the answer to the questions above, you can now use the tests to prove each of these scenarios.</p><p>Additionally, if you have helm and helm-unittest installed in your CI environment, then you can run it continuously with your pipelines by using the same commands that we used during the article.</p><h3>Final Considerations</h3><p>As we need to add more checks or behaviors to our helm templates, we should do it more securely by writing unit tests, and maybe we can avoid some nasty bugs here and there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*NDjrzGEHiS-HyKhR.gif" /></figure><p>The situation presented in the scenario can happen at any point of your helm template; in truth, more checks that you put into your yamls— nesting ifs, ranges, and/or/xor and some other TonyHawk Moves ™️, higher is the chance that you’ll hit an unpredicted behavior.</p><p>Some final thoughts about this article:</p><ol><li>If you are nesting too many ifs or some other conditionals, maybe its time to write a function inside your helpers.tpl file with Single Responsibility in mind to take out complexity;</li><li>What to test and what not? I think the same rule for code can be considered for helm charts: write tests for what makes you more comfortable in terms of security to change or refactor a helm chart; avoid testing third-party code (like the other functions inside helpers.tplfile) unless you make changes to them.</li></ol><p>That’s All Folks!</p><blockquote>Home Exercise: can you implement item #2 of our scenario covered with unittests? Can you do it in a TDD way?</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2cb9d5c1ddc8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Secrets as a code with Mozilla SOPS and AWS KMS]]></title>
            <link>https://medium.com/mercos-engineering/secrets-as-a-code-with-mozilla-sops-and-aws-kms-d069c45ae1b9?source=rss-e30210ba8ff9------2</link>
            <guid isPermaLink="false">https://medium.com/p/d069c45ae1b9</guid>
            <dc:creator><![CDATA[Gabriel Abdalla Cavalcante]]></dc:creator>
            <pubDate>Sat, 01 Feb 2020 12:28:35 GMT</pubDate>
            <atom:updated>2024-07-20T15:07:56.296Z</atom:updated>
            <cc:license>https://creativecommons.org/publicdomain/mark/1.0/</cc:license>
            <content:encoded><![CDATA[<h3>Secrets as code with Mozilla SOPS and AWS KMS</h3><p>How to securely manage your secrets inside your application workflow</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*youH_J-7OufF1mxNOrVAtA.jpeg" /></figure><p>A simple question that pops in every application development cycle is: where do I keep the credentials/secrets used by my application?</p><p>Before we dive into the “world of secrets” lets align some concepts first: in the context of this post, what is a secret?</p><p><em>We’re going to assume that a secret is a piece of information that will be used to access some resource (like databases, apis, etc.) and cannot be made public for security reasons.</em></p><p>In this way, when we say “Secrets as Code”, we’re talking about one of possible methods on how to keep and retrieve those secrets to be used by an application. Before we dive in the main topic, let’s see some ways on how to manage the secrets:</p><ol><li>By writing them directly on source code(Hard Coded):<strong> the worst possible option</strong>, it’s the poor’s man’s choice on how to do it and represents two main problems:</li></ol><ul><li>Security itself, everyone that has repository access can see all credentials and can use them freely if they wish (in some cases people tend to rely on enforced access by IP or other methods to feel ‘safer’);</li><li>Reusing or making the application easily “deployable” into more environments becomes a nightmare, because you need to edit source code every time.</li></ul><p>2. By keeping it on a different place than the source code: In this approach your application expects that credentials are available on filesystem or as environment variables but they aren’t related to the application’s lifecycle; someone/something somewhere keeps the credentials updated.</p><p>3. By keeping it in the same repo of your source code (and read them from the application or inject as environment variables): now you’re saying that your secrets have a lifecycle that is almost transparent and you need to open a merge/pull request to make the changes in a gitflow based workflow.</p><p>In this way, you move into making your VCS as the source of truth for all your code and its secrets.</p><p><em>In this post, we’re going to see how to leverage Mozilla SOPS and achieve a state where you can commit and manage your secrets in conjunction with the source code.</em></p><h4>What is Mozilla SOPS?</h4><p>Mozilla SOPS is a cli tool to works with filetypes that relies on key:value format (json, yaml, env) and does that by **encrypting only the values**, allowing us to see the keys and thereby comprehend the set of secrets that are present on that file without leaking the values.</p><p>An simple but effective example on how to use can be viewed here (from sops github page):</p><figure><img alt="Oficial git from sops github page" src="https://cdn-images-1.medium.com/max/800/0*1yNPMe-anVP1ZQxG" /></figure><h3>What do we need?</h3><p>To make things work we’re going to need at least:</p><ul><li>An AWS account with privileges to create an IAM User and a KMS Key using cloudformation templates;</li><li>AWS Cli installed and configured (<a href="https://docs.aws.amazon.com/cli/latest/userguide/install-cliv1.html">https://docs.aws.amazon.com/cli/latest/userguide/install-cliv1.html</a>);</li><li>A key to encrypt the values: in our case, we’re going to use a AWS KMS because it is a simple way to have an certified key (FIPS 140–2), works well programatically and is supported by SOPS.</li><li>The sops binary, which can be download from <a href="https://github.com/mozilla/sops/releases">https://github.com/mozilla/sops/releases</a> or through brew install sops if you’re using MacOS.</li></ul><h4>Creating the IAM User</h4><p>An IAM user will be used to encrypt and decrypt the secrets. The first thing that we need is to create this user. You can create it by using the following cloudformation template:</p><pre>Resources:<br>  OperatorUser:<br>    Type: &#39;AWS::IAM::User&#39;<br>    Properties:<br>      UserName: &#39;kms.user&#39;</pre><pre>  OperatorAccessKey:<br>    Type: &#39;AWS::IAM::AccessKey&#39;<br>    Properties:<br>      UserName:<br>        !Ref OperatorUser</pre><pre>Outputs:<br>  OperatorAccessKeyOutput:<br>    Value:<br>      !Ref OperatorAccessKey<br>  OperatorSecretKeyOutput:<br>    Value: !GetAtt OperatorAccessKey.SecretAccessKey</pre><p>Save this file as cloudformation.yaml and then create the cloudformation stack by using the following command:</p><pre>aws cloudformation deploy \<br>--stack-name kms-example \<br>--template-file ./cloudformation.yaml \<br>--capabilities CAPABILITY_NAMED_IAM \<br>--region us-east-1</pre><p>This command will create a cloudformation stack named kms-examplethat have a user which username is kms.userand its credentials will be written in the OperatorAccessKeyOutput and secret access key in the OperatorSecretKeyOutput. You can view the credentials using AWS console or by using the following command:</p><p>aws cloudformation describe-stack kms-example</p><p>With our user in place, its time to create the KMS Key that will be used to encrypt and decrypt our secrets files.</p><h4>Creating the KMS Key</h4><p>Now we going to reuse the same cloudformation.yaml file, by adding the following declaration under the ‘Resources’ Key (right after <em>OperatorAccessKey </em>and before ‘Outputs’ Key):</p><pre>KMSKey:<br>      Type: AWS::KMS::Key<br>      Properties:<br>        KeyPolicy:<br>          Version: 2012-10-17<br>          Statement:<br>            - Sid: Enable Root IAM User Permissions<br>              Effect: Allow<br>              Principal:<br>                AWS: !Join<br>                  - &#39;&#39;<br>                  - - &#39;arn:aws:iam::&#39;<br>                    - !Ref &#39;AWS::AccountId&#39;<br>                    - &#39;:root&#39;<br>              Action: &#39;kms:*&#39;<br>              Resource: &#39;*&#39;<br>            - Sid: Enable User Permissions<br>              Effect: Allow<br>              Principal:<br>                AWS: !Join<br>                  - &#39;&#39;<br>                  - - &#39;arn:aws:iam::&#39;<br>                    - !Ref &#39;AWS::AccountId&#39;<br>                    - &#39;:user/kms.user&#39;<br>              Action:<br>                - &quot;kms:DescribeKey&quot;<br>                - &quot;kms:Encrypt&quot;<br>                - &quot;kms:Decrypt&quot;<br>                - &quot;kms:ReEncrypt*&quot;<br>                - &quot;kms:GenerateDataKey&quot;<br>                - &quot;kms:GenerateDataKeyWithoutPlaintext&quot;<br>              Resource: &#39;*&#39;</pre><p>In the Ouput Section add the following values:</p><pre><br>  KMSArn:<br>    Value:<br>      !GetAtt KMSKey.Arn</pre><p>In the snippet above we’re describing the following actions:</p><ol><li>Create a KMS Key where the root account can do everything and the account kms.user can use the it en|de|re|crypt values.</li><li>Output the Arn of KMS Key, which will be used by sops to find the Key during the operations.</li></ol><p>Update the kms-example stack by using the same command:</p><pre>aws cloudformation deploy \<br>--stack-name kms-example \<br>--template-file ./cloudformation.yaml \<br>--capabilities CAPABILITY_NAMED_IAM \<br>--region us-east-1</pre><p>Check the KMSArn value by using the command:</p><p>aws cloudformation describe-stack kms-example</p><p>This is the reference to the kms key expected by Mozilla SOPS in its operations.</p><h4>Mozilla SOPS</h4><p>Now that we have a User and a KMS Key created, its time to create and decrypt/encrypt some values. Lets start with a yaml file:</p><pre>sops --kms &lt;ARN&gt; credentials.yaml</pre><p>If all steps were executed correctly, the following text will appear (using your default editor):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/1*h0wOM8aY8xzHFmh8d6JZfg.png" /></figure><p>Change the content to the following and save it:</p><pre>---<br>secrets:<br>  db:<br>    user: username1<br>    password: password1</pre><p>Now, if you open the file using an text editor, you going to see something like this output (plus some metadata used by sops):</p><pre>secrets:<br>    db:<br>        user: ENC[AES256_GCM,data:LLTdXvf5Q60E,iv:qxUdc93ZvFlCieMrGJ+q4R9YJrr017mEdSKalp1FO1I=,tag:0n9DjDb2ikPquTm2mWI/iQ==,type:str]<br>        password: ENC[AES256_GCM,data:btCtnukKSXyN,iv:FjO3I0umDFRiVtLsWQJ2FUtIMimWOtjaycQ6TM89uDc=,tag:ZpFQz/s/KdVABO9fL5tmkg==,type:str]</pre><p>If you change your mind and want to edit the file, you can use the sops command to open the editor again:</p><pre>sops credentials.yaml</pre><p>This time you won’t need to use the kms &lt;ARN&gt; parameter because its was added to the metadata in the file. Nice isn’t?</p><p>If you want to generate a decrypted version of the file, you can use the command:</p><pre>sops decrypt credentials.yaml</pre><p>Which outputs the content to stdout or you can redirect to another file.</p><p>With this, you can commit the file to your VCS or even send the variables to Hashicorp Vault/S3/GCS by using the SOPS options!</p><h4>What else?</h4><p>Using SOPS is one way on how you can leverage your VCS and workflow around it, keeping the secrets with the source code.</p><p>Mozilla SOPS can send the credentials for a hashicorp vault or even be used to export the variables to environment.</p><p>There are other solutions like Bitnami KubeSeal (to work with Kubernetes) for example, but for now, that’s all Folks!</p><p>Edit: some reddit people assert that secret as a code would have a different meaning that intended in some scenarios, so I updated the title to match the correct meaning.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d069c45ae1b9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mercos-engineering/secrets-as-a-code-with-mozilla-sops-and-aws-kms-d069c45ae1b9">Secrets as a code with Mozilla SOPS and AWS KMS</a> was originally published in <a href="https://medium.com/mercos-engineering">mercos-engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>