<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Daniello on Medium]]></title>
        <description><![CDATA[Stories by Daniello on Medium]]></description>
        <link>https://medium.com/@dast04?source=rss-6374c4b54e6f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Stories by Daniello on Medium</title>
            <link>https://medium.com/@dast04?source=rss-6374c4b54e6f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 04:58:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dast04/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Writing Dockerfiles with Best Practices]]></title>
            <link>https://medium.com/@dast04/writing-dockerfiles-with-best-practices-c9aaaba53745?source=rss-6374c4b54e6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/c9aaaba53745</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[best-practices]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[dockerfiles]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Daniello]]></dc:creator>
            <pubDate>Sun, 19 Jan 2025 11:32:41 GMT</pubDate>
            <atom:updated>2025-01-19T11:32:41.595Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*MLFEYsEAmOBq4ztmdtzfKA.png" /><figcaption>Image from <a href="https://www.docker.com/blog/intro-guide-to-dockerfile-best-practices/">https://www.docker.com/blog/intro-guide-to-dockerfile-best-practices/</a></figcaption></figure><p>Docker has revolutionized the way we build, ship, and run applications. At the heart of Docker’s simplicity is the Dockerfile — a text file that contains instructions to assemble an image. While creating a basic Dockerfile is straightforward, adhering to best practices can greatly improve your image’s performance, security, and maintainability.</p><h3>Why it matters?</h3><p>A well-crafted Dockerfile reduces image size, prevents vulnerabilities, and ensures your application runs predictably across environments. Poorly written Dockerfiles, on the other hand, can lead to bloated images, security risks, and debugging nightmares.</p><h3>Best Practices</h3><p>To enable easy update and support contribution from the community, the list has been implemented in the GitHub below.</p><p><strong>Contributions are welcomed, and if you learned something make sure to leave a star!</strong>⭐</p><p><a href="https://github.com/danielstankw/Dockerfile-best-practices">GitHub - danielstankw/Dockerfile-best-practices: Compilation of Dockerfile best practices/ optimizations</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c9aaaba53745" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automated Parametrized Jenkins Builds with BitBucket Webhooks]]></title>
            <link>https://medium.com/@dast04/automated-parametrized-jenkins-builds-with-bitbucket-webhooks-46d6156d1d2d?source=rss-6374c4b54e6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/46d6156d1d2d</guid>
            <category><![CDATA[bitbucket]]></category>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[cicd]]></category>
            <category><![CDATA[webhooks]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Daniello]]></dc:creator>
            <pubDate>Sat, 19 Oct 2024 19:34:11 GMT</pubDate>
            <atom:updated>2024-10-19T19:34:11.639Z</atom:updated>
            <content:encoded><![CDATA[<p>In this guide, you’ll learn how to automate parameterized Jenkins builds triggered by changes in a Bitbucket repository.</p><p>Whether you’re working with a Jenkinsfile or not, you’ll discover how to set up a Jenkins job that passes key parameters from Bitbucket — streamlining your development workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eYTen9d1GCl6cF7ztXVfYg.png" /><figcaption>Parametrized Build — Pipeline</figcaption></figure><p>By the end of this article, you’ll know how to:</p><ul><li><strong>Trigger Jenkins jobs on branch changes:</strong> Automatically trigger a Jenkins build when a push or merge is made to a specific branch (the default setup will trigger on changes to any branch if you’re using the native Bitbucket plugin).</li><li><strong>Pass repository details via webhooks:</strong> Learn how to send critical information like branch name, repository name, and user details from Bitbucket to Jenkins using webhooks.</li><li><strong>Use one Jenkins job for multiple repositories:</strong> Set up a single Jenkins job to handle builds for multiple repositories in the same project by utilizing parameterized builds.</li><li><strong>Set up the Generic Webhook Trigger plugin:</strong> Understand why this plugin is a better option compared to the standard Bitbucket webhook, and how to use it effectively.</li></ul><p>By following these steps, you’ll improve the efficiency of your CI/CD pipeline and gain better control over how and when Jenkins jobs are triggered based on changes in your Bitbucket repositories.</p><h4>Prerequisites</h4><ul><li>Jenkins Server</li><li>BitBucket Repository</li></ul><blockquote>You can follow similar steps with other Git repositories but this article is specific to BitBucket.</blockquote><h4>Why You Should Use the Generic Webhook Trigger Plugin for Jenkins Instead of the Native Bitbucket Plugin</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/869/0*YEevftFIfpvhMKc8.png" /></figure><p>When exploring options for automating Jenkins builds from Bitbucket, one of the first solutions you may come across is the <a href="https://plugins.jenkins.io/bitbucket/">Bitbucket Plugin for Jenkins</a>. This plugin offers a straightforward way to trigger Jenkins jobs on push or merge events with minimal setup.</p><p>At first glance, its simplicity might seem appealing. However, as I discovered, this simplicity comes at the cost of flexibility. The native Bitbucket plugin lacks the configurability needed for more complex workflows. Here’s why:</p><h4><strong>Limitations of the Native Bitbucket Plugin</strong></h4><ul><li><strong>Limited control over triggers:</strong> While you can set up triggers for push or merge events, you cannot configure them to run only for specific branches. This becomes problematic if you only want builds to trigger under certain conditions, such as when code is merged into the develop branch.</li><li><strong>Branch-specific actions are not possible:</strong> If your team uses prototyping branches to experiment or fix bugs, you may not want Jenkins builds to be triggered every time code is pushed to those branches. Unfortunately, with the Bitbucket plugin, you cannot define such branch-specific behavior.</li><li><strong>Lack of parameter support:</strong> If your Jenkins pipeline needs to perform different actions based on the branch name, repository name, or other variables, the native plugin falls short. For complex scenarios where passing parameters is essential, it’s not the best tool.</li></ul><h4>Why the Generic Webhook Trigger Plugin is Superior</h4><p>For advanced use cases like these, the <a href="https://plugins.jenkins.io/generic-webhook-trigger/"><strong>Generic Webhook Trigger Plugin</strong></a> offers much more flexibility. It allows you to pass parameters such as the branch name, repository name, and user information directly to your Jenkins job. This makes it possible to:</p><ul><li>Trigger jobs only for specific branches.</li><li>Customize the behavior of your Jenkins pipeline based on repository data.</li><li>Set up one Jenkins job to handle multiple repositories and branches efficiently.</li></ul><h3>Configuring BitBucket Webhook</h3><ol><li>Go to your repository settings.</li><li>Click on <em>Webhooks</em></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/185/1*xGs6Iw4yNs2T2p5ueRzqxg.png" /><figcaption>Webhooks — BitBucket</figcaption></figure><p>3. Click <em>Create Webhook</em><br>This will result in a Webhooks configuration page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*gAjInPtHwWrZlL7LzB8N3Q.png" /><figcaption>BitBucket Webhook page</figcaption></figure><ul><li>Name: Name of your choice for the Webhook</li><li>URL : URL with the following format</li></ul><p><a href="https://%7BJENKINS_USER%7D:%7BJENKINS_PASS%7D@{jenkins_url}/generic-webhook-trigger/invoke">https://{JENKINS_USER}:{JENKINS_PASS}@{JENKINS_URL}/generic-webhook-trigger/invoke</a></p><blockquote>HTTP(S): modify the http/https depending on your setup.</blockquote><ul><li>JENKINS_USER: username used to log into Jenkins</li><li>JENKINS_PASS: password used to log into Jenkins / access token (<em>recommended</em>)</li><li>JENKINS_URL: <em>ex. my-little-jenkins-server.com</em></li></ul><p>It is a best practice<strong> not to</strong> use your account password for Webhooks, instead log into Jenkins and click on your username &gt; security and generate an <strong>access token</strong>, which will replace the password in the webbook.</p><blockquote>You can additionally add token parameter to link Webhook to the specific Jenkins job, but we will discuss it later.</blockquote><p>Finally select what events you wish to monitor.</p><blockquote>Note: Note: We cannot specify which specific branches to monitor from BitBucket directly. Instead, this must be configured on the Jenkins side. The BitBucket webhook can monitor every push event (if push events are selected), but whether Jenkins triggers a job based on that is determined by our configuration within Jenkins.</blockquote><p>To test the webhook, make change to the repository and push it to the BitBucket. <br>If you haven&#39;t set up jenkins job jet the following will be shown:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gypYgShIssnQhr5HwCghTA.png" /></figure><p>Click on view details. You will be able to see how many times the webhook was triggered and whether it succeeded or failed. Lets view details.</p><p>Click on last failure to view the request body. Don&#39;t worry for now about 404 error, as we haven&#39;t set a Jenkins side of webhook yet:</p><p>Go to the request body:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/267/1*CEVNfg1YMlEmLv0T0PPN-A.png" /><figcaption>Webhook event details</figcaption></figure><p>Copy the content to the IDE (<em>I used Visual Studio</em>) and enable json formatting to display a clear structure of the file.</p><p>Now we need to extract the JSONPath of element of our interest example: repo name, project name or branch name.</p><p>In my case I was interested in two variables, which you can see below together with their respective JSONPaths:</p><ul><li>Branch Name</li></ul><pre>$.changes[0].ref.displayId</pre><ul><li>Repository Name</li></ul><pre>$.repository.name</pre><blockquote>The structure of your JSONPath as well as your Request Body can be different, so you got two options here: either figure out based on the file how to access the element or use chat gpt!</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/804/1*4wHxC1dudwlXRIiBAiL3Pw.png" /><figcaption>Example prompt to obtain JsonPath</figcaption></figure><blockquote>To validate the expression we received use the following: <a href="https://jsonpath.curiousconcept.com/">JsonPath Expression Teste</a>r</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*t52grsaMkNxA5U2j.png" /><figcaption>Example of JsonPath expression tester and obtained result</figcaption></figure><p>At this point <strong>save the JsonPath expression</strong> as you will need it in Jenkins job settings.</p><h3>Jenkins plugins</h3><p>You will need the following plugins installed in your Jenkins.</p><ul><li><a href="https://plugins.jenkins.io/generic-webhook-trigger/">Generic Webhook Trigger</a></li><li><a href="https://plugins.jenkins.io/workflow-aggregator/">Pipelinie</a></li><li><a href="https://plugins.jenkins.io/bitbucket/">BitBucket</a></li></ul><h3>Jenkins Job Configuration</h3><p>With the obtained JSONPath expression of the element of your interest, webhook configured and plugins downloaded you can proceed to the final step — configuring your Jenkins job.</p><p><strong>In the Jenkins:</strong></p><ol><li>New Item</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/187/1*AoqcsKRAcBlNwnGdJeT5dQ.png" /></figure><p>2. Select item type: in this example I am using <strong>Pipeline</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*bU4E77DRTzBnbcRPMjDDvw.png" /></figure><p>3. In the settings in <strong>Build Triggers</strong> select <strong>Generic Webhook Trigger</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/397/1*31cQT7nGXm1yxqywUureEA.png" /><figcaption>Generic Webhook Trigger</figcaption></figure><p>4. Configuring the plugin:</p><p>Click on Post content parameters and then Add</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*-PwH6hJl5fQMzFedt-dYZA.png" /><figcaption>Generic Webhook Trigger — Configuration</figcaption></figure><ul><li>Variable: Is the environment variable name that you will use in your pipeline to refer to the element passed via webhook, in this example a branch name: BRANCH_NAME</li><li>Expression: The JsonPath expression we got from earlier steps used for retrieving the branch name from the webhook.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b3hFclo5D6zTjJVx02fsBQ.png" /><figcaption>Generic Webhook Trigger — Variable</figcaption></figure><p><strong><em>(Optional)</em></strong><em> </em><br>Set up a token <em>(make sure to modify webhook URL in BitBucket if you defined Token here) </em>if token is specfied the webhook URL will have the following form:</p><p><a href="https://%7BJENKINS_USER%7D:%7BJENKINS_PASS%7D@{jenkins_url}/generic-webhook-trigger/invoke">https://{JENKINS_USER}:{JENKINS_PASS}@{JENKINS_URL}/generic-webhook-trigger/invoke</a>?token=TOKEN_HERE</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E7Dw1I5xfzatieJWZG1Mww.png" /><figcaption>Generic Webhook Trigger — Token Configuration</figcaption></figure><p>For testing you can enable the following settings, which will print the passed variables in your logs, making it easier to debug .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/274/1*EbBGtYLwQyUX2xJw3ZoDBA.png" /><figcaption>Generic Webhook Trigger — Print Options</figcaption></figure><p>And finally, we got to answer the question:</p><blockquote>Question: How to trigger Jenkins job on event in specific branch (ex. develop) ?</blockquote><blockquote>Answer: <strong>Filters!</strong></blockquote><p>In the screenshot below we set a condition, in which we test whether the BRANCH_NAME variable is equal to develop, only then the job is triggered.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CKWohGxVBl_Vcp6UPQ2p_A.png" /><figcaption>Generic Webhook Trigger — Filters</figcaption></figure><p>In my case I also want to pass the repository name as I will use it in the pipeline, so I added another variable with:</p><ul><li>Variable: COMPONENT</li><li>Expression: $.repository.name</li></ul><h3>Jenkins Pipeline</h3><p>In this step you can go in two directions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/277/1*IruMG4P-PInfvQoguoh5Ew.png" /><figcaption>Option for Jenkinsfile based builds</figcaption></figure><p>In the pipeline section you can choose one of the two options:</p><ul><li><strong>Pipeline script from SCM — </strong>Uses Jenkinsfile stored in the Git repository as a source of the Job build. This is the recommended approach as it provides versioning.</li><li><strong>Pipeline script — </strong>Defines the Job directly in UI.</li></ul><p>For the sake of simplicity we will use the second approach, the second approach is very intuitive as well — if you got any questions feel free to contact me.</p><p>For this example we will run a simple job that will simply print the defined variables: branch name and repository name:</p><pre>node(&#39;agent&#39;) {    <br> stage(&#39;Print Hello&#39;) {<br>  echo &quot;Triggered from branch: ${env.BRANCH_NAME} and repo: ${env.COMPONENT}&quot; <br>    }<br>}</pre><blockquote>Note: I run Jenkins in kubernetes as an agent so the setup is slightly different — note the “agent” section.</blockquote><h3>Finally …</h3><p>Push a change to the branch you defined a watch the job being triggered !</p><p>If your job run was successful you can check logs and see the variables, thanks to print contributed variables section we selected earlier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/232/1*ZCW_qyuPHtGaV8jLypMw3A.png" /><figcaption>Printed variables</figcaption></figure><h4>Troubleshooting</h4><p>If your webhook doesn&#39;t rigger the job try:</p><ul><li>Run the Jenkins job manually for the first time, after which try again using webhook.</li><li>Define a plugin token and modify webhook URL — see reminder below:</li></ul><p><a href="https://%7BJENKINS_USER%7D:%7BJENKINS_PASS%7D@{jenkins_url}/generic-webhook-trigger/invoke">https://{JENKINS_USER}:{JENKINS_PASS}@{JENKINS_URL}/generic-webhook-trigger/invoke</a>?token=TOKEN_HERE</p><h3>You made it!</h3><p>If you like the style of the article and found the information useful, please consider clicking the 👏 emoji.</p><p><strong><em>Got any questions? Want to connect? — Feel free to add me on </em></strong><a href="https://www.linkedin.com/in/danielstankowski/"><strong><em>LinkedIn</em></strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/0*jz8hjCVUJoOYZTbH.jpg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=46d6156d1d2d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setting up ArgoCD LDAP Authentication & RBAC [2024]]]></title>
            <link>https://medium.com/@dast04/setting-up-argocd-ldap-authentication-rbac-2024-50c0d8135713?source=rss-6374c4b54e6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/50c0d8135713</guid>
            <category><![CDATA[authentication]]></category>
            <category><![CDATA[ldap]]></category>
            <category><![CDATA[cicd]]></category>
            <category><![CDATA[argo-cd]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Daniello]]></dc:creator>
            <pubDate>Mon, 16 Sep 2024 12:15:06 GMT</pubDate>
            <atom:updated>2024-09-16T12:15:06.812Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/356/1*FZ-kB6zsVCNZU42aPsS6XA.png" /><figcaption>Logging to ArgoCD with LDAP</figcaption></figure><p>In this article I will explain how to set up ArgoCD authentication with LDAP. There are other articles on this topic, but I found them incomplete. <em>The list of all resources can be found at the end of this article.</em></p><h4>1. Introduction</h4><p>For the installation follow the below steps, also described in <a href="https://argo-cd.readthedocs.io/en/stable/getting_started/">docs</a>. I will not focus on the installation part as there are a few approaches and there are many tutorials out there about the process. In my case I installed it using the following commands.</p><pre>kubectl create namespace argocd<br>kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml</pre><p>After successful installation with the above method you will get the following pods</p><pre>NAME                                                READY   STATUS <br>argocd-application-controller-0                     1/1     Running <br>argocd-applicationset-controller-xxxxxx             1/1     Running <br>argocd-dex-server-xxxxxx                            1/1     Running<br>argocd-notifications-controller-xxxxxx              1/1     Running<br>argocd-redis-xxxxxx                                 1/1     Running<br>argocd-repo-server-xxxxxx                           1/1     Running<br>argocd-server-xxxxxx                                1/1     Running</pre><p>Additionally you will also deploy various ConfigMaps. The important for us are the following</p><pre>argocd-cm                  <br>argocd-rbac-cm            </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*O2ztdg-40xzmfPZu.png" /><figcaption>ArgoCD Image</figcaption></figure><h4>2. LDAP Configuration</h4><p>Argo CD embeds and bundles <a href="https://github.com/dexidp/dex">Dex</a> as part of its installation, for the purpose of delegating authentication to an external identity provider. Multiple types of identity providers are supported (OIDC, SAML, LDAP, GitHub, etc…). SSO configuration of Argo CD requires editing the argocd-cm ConfigMap with Dex connector settings.</p><p>Our first step will be editing ConfigMap argocd-cm and adding dex.config to it, let’s look how we can do that.</p><blockquote>Note: For the full explanation of all the dex fields for LDAP check out this <a href="https://dexidp.io/docs/connectors/ldap/">documentation</a>.</blockquote><p>To configure authentication through Dex with an LDAP (Lightweight Directory Access Protocol) connector. Let’s create a file patch-dex.yaml with the following content:</p><pre>apiVersion: v1<br>data:<br>  # Leave as true untill LDAP has been successfuly set up - enables manual login<br>  admin.enabled: &quot;true&quot;<br>  dex.config: |<br>    connectors:<br>    - type: ldap<br>      name: LDAP<br>      id: ldap<br>      config:<br>        # Ldap server address (choice of port depends on your set-up, see docs)<br>        host: &quot;ldap.example.com:389&quot;<br>        insecureNoSSL: true<br>        insecureSkipVerify: true<br>        startTLS: false<br>        # Variable name stores ldap bindDN in argocd-secret<br>        bindDN: &quot;$dex.ldap.bindDN&quot;<br>        # Variable name stores ldap bind password in argocd-secret<br>        bindPW: &quot;$dex.ldap.bindPW&quot;<br>        usernamePrompt: username <br>        # Ldap user search attributes<br>        userSearch:<br>          baseDN: &quot;cn=users,dc=example,dc=com&quot;<br>          filter: &quot;(objectClass=person)&quot;<br>          username: uid<br>          idAttr: uid<br>          emailAttr: mail<br>          nameAttr: name<br>        groupSearch:<br>          baseDN: &quot;cn=users,dc=example,dc=com&quot;<br>          filter: &quot;(objectClass=group)&quot;<br>          userMatchers:<br>          - userAttr: uid<br>            groupAttr: member<br>          # Represents group name.<br>          nameAttr: name<br>  # This will prevent ldap login to redirect to itself.<br>  url: &lt;your-argocd-url&gt;</pre><p><strong>Let’s break down key elements (for more info check out </strong><a href="https://dexidp.io/docs/connectors/ldap/"><strong>docs</strong></a><strong>):</strong></p><p>Let’s assume you are using Active Directory and you have the following credentials: LDAP_URL, LDAP_USER, LDAP_PSW</p><ul><li>host: The LDAP_URL with port of 389 or 636 (depending on your setup)</li><li>bindDN: The LDAP_USER</li><li>bindPW : The LDAP_PSW</li><li>usernamePrompt : The text displayed in your username prompt in LDAP</li><li>baseDN : BaseDN to start search from, it will translate to query: &quot;(&amp;(objectClass=person)(uid=&lt;username&gt;))&quot;.</li></ul><h4>Kubernetes Secrets</h4><p>As you probably noted, sensitive information was provided using kubernetes secrets:</p><p>Create <em>Secret </em>manifest.</p><pre>apiVersion: v1<br>kind: Secret<br>metadata:<br>  name: argocd-ldap-secrets<br>  namespace: argocd<br>type: Opaque<br>data:<br>  bindDN: &lt;base64-encoded-bind-dn&gt;<br>  bindPW: &lt;base64-encoded-bind-pw&gt;</pre><p>Replace &lt;base64-encoded-bind-dn&gt; and &lt;base64-encoded-bind-pw&gt; with the base64-encoded values of your LDAP bind DN and password, respectively.</p><pre>echo -n &#39;your-bind-dn&#39; | base64<br>echo -n &#39;your-bind-pw&#39; | base64</pre><p>Save the file and apply it to your cluster</p><pre>kubectl apply -f path-to-your-secret-file.yaml</pre><p>The next few fields represent the attributes used in your AD, you need to check how are they defined in your organization — the names may be different.</p><pre># username attribute used for comparing user entries. This will be translated<br># and combined with the other filter as &quot;(&lt;attr&gt;=&lt;username&gt;)&quot;.<br>username: uid<br># String representation of the user.<br>idAttr: uid<br># Required. Attribute to map to Email.<br>emailAttr: mail<br># Maps to display name of users. No default value.<br>nameAttr: name</pre><blockquote><strong>Important!</strong><br>Depending on whether you wish to use uid, mail, name as an username the field of username will vary.</blockquote><p><em>Example: </em><em>username:uid means that the </em><em>idAttr:uid will obtain the value of user id, and then will use it as a username. You can change it to login by email address, or name if you wish to.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wdBG92Nl-PxvXSV61rxs-g.png" /></figure><p>The groupSearch settings follow similar approach.</p><pre>userMatchers:<br>- userAttr: uid<br>  groupAttr: member<br># Represents group name.<br>nameAttr: name</pre><p>Similarly to the userSearch settings the values need to be adjuisted depending on your Active Directory set-up, in most cases the userAttr will vary.</p><blockquote><strong>Important!</strong><br>For userAttr try using a value of DN such that: userAttr:DN</blockquote><p>Once the dex-patch.yaml is configured we proceed with updating the argocd-cm ConfigMap.</p><pre>kubectl -n argocd patch configmaps argocd-cm --patch &quot;$(cat patch-dex.yaml)&quot;</pre><p>After updating ConfigMap, restart dex and server pods and wait till they are ready:</p><pre>NAME                                                READY   STATUS <br>argocd-application-controller-0                     1/1     Running <br>argocd-applicationset-controller-xxxxxx             1/1     Running <br>argocd-dex-server-xxxxxx                            1/1     Running<br>argocd-notifications-controller-xxxxxx              1/1     Running<br>argocd-redis-xxxxxx                                 1/1     Running<br>argocd-repo-server-xxxxxx                           1/1     Running<br>argocd-server-xxxxxx                                1/1     Running</pre><pre>kubectl -n argocd delete pod argocd-dex-server-xxxxxx<br>kubectl -n argocd delete pod argocd-server-xxxxxx</pre><p>Access ArgoCD URL:</p><blockquote>If you wish to disable manual login modify in patch-dex.yaml the admin.enabled value and set it to &quot;false&quot;</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iSjcSB85VYZB8pgS0wO-kA.png" /><figcaption>LDAP + credential login page</figcaption></figure><p>At this point you should be able to log in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*zDPVZKnIj6PwGp8v.jpeg" /><figcaption>:-)</figcaption></figure><h4>3. RBAC (Role-based access control)</h4><p>We can define RBAC roles, and then SSO groups can be mapped to those roles, an example RBAC config can be found at <a href="https://argo-cd.readthedocs.io/en/stable/operator-manual/argocd-rbac-cm-yaml/">argo-rbac-cm.yaml</a></p><p>policy.csv is an file containing user-defined RBAC policies and role definitions (<em>optional</em>).</p><pre> policy.csv: |<br>    # Grant all members of the group &#39;my-org:team-alpha; the ability to sync apps in &#39;my-project&#39;<br>    p, my-org:team-alpha, applications, sync, my-project/*, allow<br>    # Grant all members of &#39;my-org:team-beta&#39; admins<br>    g, my-org:team-beta, role:admin<br><br>  policy.default: role:readonly<br><br>  # scopes controls which OIDC scopes to examine during rbac enforcement (in addition to `sub` scope).<br>  # If omitted, defaults to: &#39;[groups]&#39;. The scope value can be a string, or a list of strings.<br>  scopes: &#39;[cognito:groups, email]&#39;</pre><p>ArgoCD come with some built in roles <a href="https://github.com/argoproj/argo-cd/blob/master/assets/builtin-policy.csv">https://github.com/argoproj/argo-cd/blob/master/assets/builtin-policy.csv</a></p><ul><li><strong>role:readonly</strong> — grants read-only permissions</li><li><strong>role:admin</strong> — grants admin permissions</li></ul><p>When a user is authenticated in Argo CD, it will be granted the role specified in policy.default.</p><blockquote><strong>All authenticated users get <em>at least</em> the permissions granted by the default policies. This access cannot be blocked by a </strong><strong>deny rule</strong></blockquote><p>The above is important to remember, as that might not be the desired behavior for your setup. My recommendation is to change the default RBAC to the one below:</p><pre>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: argocd-rbac-cm<br>  namespace: argocd<br>data:<br>  policy.default: role:none<br>  policy.csv: |<br>    p, role:none, *, *, */*, deny<br>  scopes: &#39;[groups, email]&#39;</pre><p>This will assign by default a none policy to a user with no access to any of the resources, therefore user will be able to log in into ArgoCD but wont be able to view any applications or settings.</p><p>To deploy changes:</p><pre>kubectl edit configmap argocd-rbac-cm -n argocd <br>kubectl rollout restart deployment argocd-server -n argocd</pre><h4>4. Creating Groups, Roles and Policies</h4><p>Using RBAC we can create policies and assign them to specific users/ groups, let’s have a look on how this can be done:</p><ol><li><strong>Assigning users/ group to an internal role</strong></li></ol><p>g, &lt;user/group&gt;, &lt;role&gt;</p><ul><li>&lt;user/group&gt;: The entity to whom the role will be assigned. It can be a local user or a user authenticated with SSO or a group. When SSO is used, the user will be based on the sub claims, while the group is one of the values returned by the scopes configuration.</li><li>&lt;role&gt;: The internal role to which the entity will be assigned.</li></ul><p><strong>2. Assigning policy to a role/user/group.</strong></p><p>p, &lt;role/user/group&gt;, &lt;resource&gt;, &lt;action&gt;, &lt;object&gt;, &lt;effect&gt;</p><ul><li>&lt;role/user/group&gt;: The entity to whom the policy will be assigned to.</li><li>&lt;resource&gt;: The type of resource on which the action is performed.</li><li>&lt;action&gt;: The operation that is being performed on the resource.</li><li>&lt;object&gt;: The object identifier representing the resource on which the action is performed. Depending on the resource, the object&#39;s format will vary.</li><li>&lt;effect&gt;: Whether this policy should grant or restrict the operation on the target object. One of allow or deny.</li></ul><h4>5. Using SSO Users/ Groups</h4><p>Let’s recall from previous section:</p><blockquote>g, &lt;user/group&gt;, &lt;role&gt;</blockquote><blockquote>&lt;user/group&gt;: The entity to whom the role will be assigned. It can be a local user or a user authenticated with SSO or a group. When SSO is used, the user will be based on the sub claims, while the group is one of the values returned by the scopes configuration.</blockquote><p>When using SSO:</p><ul><li>user is based on the sub claims.</li><li>group is based on the scopes configuration.</li></ul><h4>ArgoCD CLI</h4><p>To login into the ArgoCD CLI we have two options. Either you can log in with the master user and password (unless you disabled it) or you can log in using LDAP</p><pre># admin user and password (will be disabled)<br>argocd login &lt;argocd_url&gt; --grpc-web<br><br># with LDAP<br>argocd login &lt;argocd_url&gt; --sso</pre><p>After logging in you will be able to view clusters, applications, add and delete them.</p><p><strong>Have fun and explore!</strong></p><h3>You made it!</h3><p>If you like the style of the article and found the information useful, please consider clicking the 👏 emoji.</p><p><strong><em>Got any questions? Want to connect? — Feel free to add me on </em></strong><a href="https://www.linkedin.com/in/danielstankowski/"><strong><em>LinkedIn</em></strong></a></p><p>Good luck!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/0*dvVFOoI-9Y3RT4Ty.jpg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=50c0d8135713" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Host your own Helm Chart Repository in Kubernetes [ChartMuseum]]]></title>
            <link>https://medium.com/@dast04/host-your-own-helm-chart-repository-in-kubernetes-chartmuseum-98e808fba17e?source=rss-6374c4b54e6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/98e808fba17e</guid>
            <dc:creator><![CDATA[Daniello]]></dc:creator>
            <pubDate>Thu, 22 Aug 2024 14:43:28 GMT</pubDate>
            <atom:updated>2024-09-16T19:00:46.994Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://chartmuseum.com/docs/#"><strong>ChartMuseum</strong></a> is an open-source Helm Chart repository server that simplifies the process of managing Helm charts for Kubernetes. It provides a simple way to host your own chart repository, enabling users to easily share, store, and access Helm charts.</p><p>It is a simple tool, so I will keep it simple in this article and will show you how to set it up in Kubernetes as well as how to upload/delete/update your charts with it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*ZEUeyaof1dX4UfFm.png" /><figcaption>ChartMuseum logo</figcaption></figure><h3>1. Installing ChartMuseum in K8s.</h3><p>To install it, we will use <a href="https://github.com/chartmuseum/charts/tree/main/src/chartmuseum">helm chart</a>.</p><blockquote><strong>Assumption</strong>: <br>You have a working Kubernetes cluster with Helm installed.</blockquote><p>Add ChartMuseum Helm repository to local Helm installation.</p><pre>helm repo add chartmuseum https://chartmuseum.github.io/charts</pre><p>After adding the chart to the local repo we can install charts from this repository and check their version.</p><pre>helm search repo chartmuseum/chartmuseum --versions</pre><p>Before installing chartmuseum we need to modify the configuration values file. Lets obtain it and save the default values to values.yaml file.</p><pre>helm show values chartmuseum/chartmuseum &gt; values.yaml</pre><p>You can check out the latest version of values.yaml in <a href="https://github.com/chartmuseum/charts/blob/main/src/chartmuseum/values.yaml">here</a>.</p><h4>2. Modifying values file</h4><p>Fortunately for us the helm chart is very simple. Lets look at the key values that you might want to adjust depending on your setup.</p><p>If you want to store your charts in AWS bucket you need to set appropriate values in the Amazon-related section of the chart.</p><pre>env:<br>  open:<br>    # s3 bucket to store charts for amazon storage backend<br>    STORAGE_AMAZON_BUCKET:<br>    # prefix to store charts for amazon storage backend<br>    STORAGE_AMAZON_PREFIX:<br>    # region of s3 bucket to store charts<br>    STORAGE_AMAZON_REGION:<br>    # alternative s3 endpoint<br>    STORAGE_AMAZON_ENDPOINT:<br>    # server side encryption algorithm<br>    STORAGE_AMAZON_SSE:</pre><p>In my setup I have set DISABLE_API: true.This will allow me to view all the charts in the <a href="https://mychartmuseum.com/api/charts">https://&lt;your_chartmuseum_url&gt;/api/charts</a> URL.</p><blockquote><strong>Note</strong>: Anyone with this URL will be able to modify the charts in the repository! <br>Unfortunately ChartMuseum doesn’t provide in-built authentication mechanism so you will need to implement it externally.</blockquote><pre>    # disable all routes prefixed with /api<br>    DISABLE_API: false</pre><p>If you decide to store your charts in a filesystem such as AWS EFS you can define a persistence settings below. In my case to keep values file sweet and simple I proceeded to create PVC and PV <strong>externally </strong>from the helm chart.</p><pre>persistence:<br>  enabled: true<br>  accessMode: ReadWriteMany<br>  size: 2Gi<br>  labels: {}<br>    # name: value<br>  path: /storage<br>  ## A manually managed Persistent Volume and Claim<br>  ## Requires persistence.enabled: true<br>  ## If defined, PVC must be created manually before volume will be bound<br>  existingClaim: chartmuseum-pvc<br><br>  pv:<br>    enabled: false</pre><p>Once your values.yamlfile is ready you can go ahead and install it.</p><pre>helm install &lt;release_name&gt; chartmuseum/chartmuseum --namespace &lt;namespace&gt; --values values.yaml</pre><p>Assuming you configured service and persistence right you should be good to go and the ChartMuseum should be accessible.</p><blockquote><strong>Important</strong>!<br>Make sure to set the Object Store policies enable write and read.<br>If you are using PV for example EFS, you need to make sure that a correct permissions are set. If you will face issues, you can use sudo chmod 777 -R &lt;folder_name&gt; to give r-w-x permissions to the folder where the EFS is mounted.</blockquote><h4>2. Accessing chart repository</h4><p>I have configured my service as a AWS LoadBalancer and then mapped it to a custom DNS name.</p><p>A properly configured ChartMuseum when accessed should result in:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/629/1*IQvrDSoMibeCdkANEEt5_A.png" /></figure><h3>2. Adding / Deleting / Updating the helm chart in ChartMuseum</h3><p>Now lets look into how to add, delete and update helm chart in Chartmuseum. In this case we will assume we have created a helm chart called mychart and all the relevant files are located in the folder with the same name.</p><p>The standard helm chart has the following strcture:</p><pre>mychart/<br>  ├── Chart.yaml          # Required: Chart metadata<br>  ├── values.yaml         # Required: Default values for templates<br>  ├── charts/             # Optional: Dependency charts<br>  ├── templates/          # Required: Kubernetes resource templates<br>  │   ├── deployment.yaml # Example template for a Deployment<br>  │   ├── service.yaml    # Example template for a Service<br>  │   ├── _helpers.tpl    # Optional: Helper templates<br>  │   ├── ingress.yaml    # Example template for an Ingress<br>  │   ├── hpa.yaml        # Example template for HorizontalPodAutoscaler<br>  │   ├── NOTES.txt       # Optional: Post-installation information<br>  ├── crds/               # Optional: Custom Resource Definitions<br>  ├── templates/tests/    # Optional: Helm test hooks<br>  │   └── test-connection.yaml # Example test template<br>  └── .helmignore         # Optional: Patterns to ignore when packaging the chart</pre><p>Let’s access the folder where all the files are located.</p><pre>cd mychart/</pre><p>Before we package the chart and upload it to the ChartMuseum let’s assure our Chart is free of errors:</p><h4>Checking validity of a chart</h4><p><strong>1.Using template command:</strong></p><pre>helm template /path/to/your/chart -f values.yaml</pre><p><strong>Purpose: </strong>Renders the template. Checks for potential issues like invalid YAML or resource conflicts.</p><p><strong>2. Dry installation</strong></p><pre>helm install --dry-run --debug /path/to/your/chart</pre><p><strong>Purpose: </strong>Simulates the installation process without actually deploying resources. It helps ensure that templates are rendered correctly and that there are no runtime issues.</p><p><strong>3. Helm lint</strong></p><pre>helm lint</pre><p><strong>Purpose:</strong> This checks the syntax and structure of the chart. It will catch common issues like missing fields or incorrect values.</p><h4>Uploading the chart to the repository</h4><p>First step requires us to package our helm chart as tgz file, while in the mychart directory, execute:</p><pre>helm package .</pre><p>This will generate a mychart-0.1.0.tgz file.</p><p>Next, we will <strong>upload </strong>it to our chartmuseum</p><pre>curl --data-binary &quot;@mychart-0.1.0.tgz&quot; https://&lt;your_chartmuseum_url&gt;/api/charts</pre><p>To verify that the chart was uploaded</p><pre>curl -X GET https://&lt;your_chartmuseum_url&gt;/api/charts/mychart</pre><p>In case we wish to delete a chart</p><pre>curl -X DELETE https://&lt;your_chartmuseum_url&gt;/api/charts/mychart/mychart/0.1.0</pre><h4>Updating helm chart after modifications</h4><ol><li>Open the Chart.yaml file.</li><li>Update the version field to a new version number. For example, if the current version is 1.0.0, you might update it to 1.0.1.</li><li>Package the chart: helm package .</li><li>curl — data-binary “<a href="http://twitter.com/mychart">@mychart</a>-0.1.0.tgz” https://&lt;your_chartmuseum_url&gt;/api/charts</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*cPN3uJiloA6l48sj.jpeg" /></figure><h3>3. Using helm charts from helm repository</h3><p>Add a new Helm chart repository to your local Helm client.</p><pre>helm repo add chartmuseum &lt;your_chart_museum_url&gt;</pre><p>Update your local Helm repository cache with the latest charts and metadata from all the repositories</p><pre>helm repo update</pre><p>Search for charts in the chartmuseum repository.</p><pre>helm search repo chartmuseum/</pre><p>Lets say we have a chart called cookie</p><pre>helm show values chartmuseum/cookie&gt; values.yaml</pre><p>Now that we have the values.yaml we can configure it.<br>The last step is to install the helm chart.</p><pre>helm install &lt;release_name&gt; chartmuseum/&lt;chart_name&gt; --values values.yaml</pre><h3>You made it!</h3><p>If you like the style of the article and found the information useful, please consider clicking the 👏 emoji.</p><p><strong><em>Got any questions? Want to connect? — Feel free to add me on </em></strong><a href="https://www.linkedin.com/in/danielstankowski/"><strong><em>LinkedIn</em></strong></a></p><p>Good luck!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/0*5ym2EPN3xDm-lqDC.jpg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=98e808fba17e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Theory — Monitoring with Prometheus & Thanos [Part 1/2]]]></title>
            <link>https://medium.com/@dast04/theory-monitoring-with-prometheus-thanos-part-1-2-8bc500a9c6a8?source=rss-6374c4b54e6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/8bc500a9c6a8</guid>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[thanos]]></category>
            <category><![CDATA[monitoring]]></category>
            <dc:creator><![CDATA[Daniello]]></dc:creator>
            <pubDate>Sun, 30 Jun 2024 09:29:59 GMT</pubDate>
            <atom:updated>2024-07-02T16:36:19.670Z</atom:updated>
            <content:encoded><![CDATA[<h3>Theory — Monitoring with Prometheus &amp; Thanos [Part 1/2]</h3><p>This series of articles will dive into a full monitoring solution for <strong>multi-cluster</strong> setup using <strong>Prometheus, Thanos, and Grafana.</strong> <br>In Part 1, we will build the necessary theoretical fundamentals, making our lives easier before reaching Part 2 — <a href="https://medium.com/@dast04/full-installation-monitoring-with-prometheus-thanos-part-2-2-fe98fcdbe448">the deployment (<em>practical</em>) section.</a></p><h4>Overview of articles</h4><ul><li><strong>Part 1</strong>: Theory — overview and deeper dive into Thanos.</li><li><strong>Part 2</strong>: <a href="https://medium.com/@dast04/full-installation-monitoring-with-prometheus-thanos-part-2-2-fe98fcdbe448">Practice— installation process.</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*THbU0dwegmcXKy7_82Oe4Q.jpeg" /><figcaption>Prometheus and Thanos by Leonardo.ai</figcaption></figure><h3>Why Thanos? — when Prometheus is not good enough</h3><blockquote><strong>Monitoring </strong>is a crucial aspect of DevOps, offering valuable insights into a range of system metrics. These metrics encompass system performance (such as CPU and GPU usage, and memory consumption), application performance (including response times and custom metrics), infrastructure health, and more.</blockquote><p>You were asked to deploy a <strong>monitoring</strong> solution for your company clusters.</p><p>You read that <a href="https://prometheus.io/"><strong>Prometheus</strong></a> is the tool for the task, and in combination with Grafana, you can create some insightful visualizations (<em>you can deploy them together with other useful components using </em><a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack"><em>kube-prometheus-stack</em></a><em>, more on that later).</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9Eg1WlnT_WkmT-t1.png" /><figcaption>Example of Grafana dashboard</figcaption></figure><p>You successfully deployed Prometheus which scraped metrics of your interest and created a dashboard in Grafana. Soon you find out that a new cluster has been created and the team wants to visualize the metrics from the <strong>new and an existing </strong>cluster on the <strong>same</strong> dashboard.</p><p>After some Google-ing, you will realize that using <a href="https://prometheus.io/docs/prometheus/latest/federation/">prometheus federation</a> is one approach you can use, but it does come with <strong>many limitations.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*-EKcUjqJ7W0wm3psTX5M0Q.png" /><figcaption>Prometheus Federation Limitations</figcaption></figure><p>As you can see, some of the limitations are game-breaking, especially if the goal is to create a <strong>production-grade multi-cluster</strong> monitoring setup, but don’t worry not all of the hope is lost…</p><h3>Thanos to the rescue!</h3><blockquote><a href="https://thanos.io/tip/thanos/getting-started.md/"><em>Thanos documentation </em></a>— see more on that in <a href="https://medium.com/@dast04/full-installation-monitoring-with-prometheus-thanos-part-2-2-fe98fcdbe448">Part 2.</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pxjok2ACCc7PJOCM.jpg" /><figcaption>Thanos — generated by Leonardo.ai</figcaption></figure><p>Prometheus is an excellent tool for monitoring and alerting in single-cluster setups. However, you can significantly enhance your functionality by integrating Prometheus with Thanos and its various microservices. Here are some benefits of using <strong>Prometheus with Thanos:</strong></p><ol><li><strong>High Availability (HA):</strong> <br>Thanos allows for the deployment of multiple Prometheus replicas, ensuring high availability. This redundancy addresses the issue of a single point of failure, so if one Prometheus instance crashes, another can take over seamlessly.</li><li><strong>Long-term Storage:<br></strong>Thanos integrates with various object storage systems, such as AWS S3 bucket, providing a cost-effective solution for long-term storage of Prometheus metrics. It also supports down-sampling, which reduces the resolution of older data to save storage space while maintaining the ability to query historical data effectively.</li><li><strong>Querying Across Clusters</strong>: <br>While Prometheus Federation<em> (as discussed earlier) </em>can offer some cross-cluster querying capabilities, Thanos takes it further by enabling global querying of metrics across multiple clusters.</li><li><strong>Global-View</strong>: <br>Thanos provides a unified global view of all Prometheus instances, enabling querying metrics from multiple clusters or data centers as if they were a single entity. This simplifies the monitoring of distributed environments and enhances your ability to diagnose issues across the entire infrastructure.</li><li><strong>Flexibility: </strong><br>Thanos components can be implemented independently, offering great flexibility. After installing the sidecar and querier, you can choose to implement additional features such as long-term storage, data compaction with the compactor, and rule evaluations with the ruler. This modular approach allows you to tailor the setup to your specific needs and scale as required.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I8ZgOoCUorv3Wi2GxX5TLg.png" /><figcaption>Example of Global View using Thanos</figcaption></figure><h3>Architecture</h3><p>In this section, we will describe (one of many possible) setups for our clusters, and give a brief overview of components and their functionality.</p><h4>Our Setup</h4><p>We have two clusters (can be scaled up), the <em>master, </em>and the <em>slave.</em></p><p><strong>Master cluster</strong></p><ul><li>The “brain” of monitoring, responsible for providing a global view of all the clusters. <br>Comes with <a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack"><em>kube-prometheus-stack</em></a><em>, </em>which includes <a href="https://github.com/prometheus-operator/prometheus-operator">Prometheus operator</a>, <a href="https://prometheus.io/docs/visualization/grafana/">Grafana</a>, <a href="https://prometheus.io/docs/alerting/latest/alertmanager/">Alertmanager</a> as well as node <a href="https://prometheus.io/docs/instrumenting/exporters/">exporters</a>. In addition, it contains Thanos micro-services: side-car, querier, store, and compactor.</li></ul><p><strong>Slave cluster(s)</strong></p><ul><li>Runs applications/services we wish to monitor. <br>Comes with a Prometheus operator, Alertmanager, node exporters, and a Thanos sidecar.</li></ul><blockquote>In the <strong>slave</strong> cluster we <strong>don’t need</strong> to deploy:<br>Grafana: we only require it on the master cluster, to provide a centralized view.<br>Querier, store, and compactor: the reason why we don’t need them will become apparent, continue reading.</blockquote><h4>Architecture — explained</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rc6sUmOc8aof0_XiaruOVw.png" /><figcaption>Example architecture — by Author</figcaption></figure><ul><li><strong>Prometheus</strong> feeds real-time metrics into the <strong>Thanos Sidecar</strong>. It also persists the data in Persistent Volume for a minimum of 6 hours.</li><li>The <strong>Thanos Sidecar</strong> uploads metrics to <strong>Object Storage </strong>every 2 hours (<em>default) </em>and allows <strong>Thanos Query</strong> to access real-time data.</li><li><strong>Thanos Store</strong> fetches historical data from <strong>Object Storage</strong> for querying.</li><li><strong>Thanos Query</strong> aggregates data from <strong>Thanos Sidecar</strong>, <strong>Thanos Store</strong>, and potentially other query nodes to provide a unified querying interface.</li><li><strong>Thanos Compactor</strong> optimizes data in <strong>Object Storage</strong> to improve query performance.</li></ul><p><strong><em>Q: Do I need to enable persistence on the Prometheus if the data anyway is saved in the Object Store?</em></strong><br><em>A: It’s highly recommended to do so. If Prometheus crashes within the 2h time difference it takes for it to save data to the object store, the short-term metrics will be lost.</em></p><p><strong><em>Q: What should the data retention period be for the Persistent Volume? </em></strong><br><em>A: By default, it is 15 days, but with Thanos, you can decrease this time to 6 hours (3 times x 2h = 6h) which provides enough of a safety net in case of connectivity issues.</em></p><h4>A deeper dive into Thanos components</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*orCvlkHQjL7f_itc" /><figcaption>Thanos infrastructure from <a href="https://thanos.io/tip/thanos/quick-tutorial.md/">https://thanos.io/tip/thanos/quick-tutorial.md/</a></figcaption></figure><p>All the information below is based on the <a href="https://thanos.io/tip/thanos/getting-started.md/">Thanos Docs</a>. This article provides the most important information presented in a digestible way.</p><ol><li><a href="https://thanos.io/tip/components/sidecar.md/"><strong>Thanos Sidecar</strong></a> — runs as a sidecar process in the same pod as Prometheus server.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/611/1*rA2aZKq93yVjKniP1oNciA.png" /><figcaption><a href="https://learn.microsoft.com/en-us/azure/architecture/patterns/sidecar#solution">https://learn.microsoft.com/en-us/azure/architecture/patterns/sidecar#solution</a></figcaption></figure><p>It has two roles:</p><ul><li>Back up Prometheus data into an object storage every 2 hours (<em>for example AWS S3 bucket</em>).</li><li>Provides other Thanos components with an access to the Prometheus metrics via gRPC API (<em>for example Thanos Query)</em></li></ul><p>In the thanosService configuration section in the kube-prometheus-stack values.yaml we can find:</p><pre>prometheus:<br>  thanosService:<br>     portName: grpc<br>     port: 10901<br>     targetPort: &quot;grpc&quot;<br>  <br>     httpPortName: http<br>     httpPort: 10902<br>     targetHttpPort: &quot;http&quot;</pre><p>As shown in the diagram and in the code snippet above, sidecar uses HTTP port 10902 for uploading the metrics to Object Store, while gRPC port 10901 is used by Thanos Query to query the metrics from the sidecar (metrics not older than 2 hours).</p><p>Once configured, the sidecar container can be found within the Prometheus pod which contains 3 containers: <em>prometheus</em>, <em>config-reloader</em> and<em> thanos-sidecar.</em></p><pre>prometheus-monitoring-kube-prometheus-prometheus-0       3/3</pre><p>If you choose to use the sidecar to also upload data to object storage:</p><ul><li>Must specify object storage (--objstore.* flags)</li><li>It only uploads uncompacted Prometheus blocks. For compacted blocks, a Thanos Compactor is required.</li><li>Because Sidecar uploads data to object storage every 2h this allows Prometheus servers to be run with relatively low retention while their historic data is made durable and queryable via object storage.</li><li>The <strong>retention of Prometheus</strong> is recommended to not be lower than three times of the min block duration, so <strong>6 hours</strong>. This achieves resilience in the face of connectivity issues to the object storage since all local data will remain available within the Thanos cluster. If connectivity gets restored the backlog of blocks gets uploaded to the object storage.</li></ul><blockquote>NOTE: This still does NOT mean that Prometheus can be fully stateless, because if it crashes and restarts you will lose ~2 hours of metrics, so persistent disk for Prometheus is highly recommended.</blockquote><p>Below you can find examples of the container arguments for prometheus &amp; thanos-sidecar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*Yaca69Fp-NkngCTqxvbpIQ.png" /><figcaption>Prometheus container</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*uytrcBcB_E8tzS0etFN6xA.png" /><figcaption>Thanos-sidecar container</figcaption></figure><p>For the list of the flags with their explanations <a href="https://thanos.io/tip/components/sidecar.md/#flags">see the docs.</a></p><p><a href="https://thanos.io/tip/components/query.md/">2. <strong>Thanos Quer</strong></a><strong>y </strong>— aka “Querier” gathers the data, evaluates the query and returns the result. It essentially allows to aggregate and optionally deduplicate multiple metrics backends under single Prometheus Query endpoint.</p><p><strong>Querier is used to:</strong></p><p>a) Evaluate PromQL queries against all instances at once.</p><p>b) Automatically detect which Prometheus server needs to be contacted for given query (based on <a href="https://thanos.io/tip/thanos/quick-tutorial.md/#external-labels"><em>external labels</em></a>) and deduplicates data collected from Prometheus HA pairs.</p><p>c) Aggregate data from any number of the different storages like: sidecar, object store (S3), another querier or ruler — anything that implements gRPC StoreAPI.</p><p>The key manifest for deployment of the Querier is thanos-query-deployment.yaml which can be found <a href="https://github.com/thanos-io/kube-thanos/blob/main/manifests/thanos-query-deployment.yaml">here</a>.</p><p>Below I present to you the most important args in the manifest file.</p><pre>      containers:<br>      - args:<br>        - query<br>        - --grpc-address=0.0.0.0:10901<br>        - --http-address=0.0.0.0:9090<br>        - --log.level=info<br>        - --log.format=logfmt<br>        - --query.replica-label=prometheus_replica<br>        - --query.replica-label=rule_replica<br>        - --endpoint=dnssrv+_grpc._tcp.thanos-store.thanos.svc.cluster.local:10901<br>        - --endpoint=dnssrv+_grpc._tcp.thanos-receive-ingestor-default.thanos.svc.cluster.local:10901<br>        - --query.auto-downsampling</pre><ul><li>--query.replica-label=&lt;YOUR_UNIQUE_LABEL_NAME&gt;: Labels to treat as a replica indicator along which data is deduplicated. We can define the field multiple times. The default ones are prometheus_replica andrule_replica</li><li>If we want to access from the side cluster we can define<br>--endpoint=dnssrv+_grpc._tcp.monitoring-kube-prometheus-thanos-discovery.monitoring.svc.cluster.local:10901.<br>To scrape metrics from a sidecar located in another cluster, we first need to set up the service to expose it.</li></ul><p><strong>3. Store Gateway— aka Thanos Store</strong></p><p>The Store Gateway serves as an API gateway for historical data stored in object storage buckets. It implements the Store API, allowing Thanos Query nodes to access and retrieve historical data efficiently. This makes it an essential part of any setup where long-term data storage and retrieval are required</p><p>Thanos Sidecar, is responsible for storing Prometheus data in the object store. However, querying this historical data directly from S3 can be cumbersome and inefficient. This is where the Store Gateway comes in:</p><ol><li><strong>Implements gRPC Data API</strong>: The Store Gateway leverages gRPC to implement the same data API as the Thanos Sidecar, ensuring compatibility and ease of integration.</li><li><strong>Acts as an API Gateway</strong>: By serving as an intermediary, it allows query nodes to access historical data seamlessly without direct interaction with the object storage.</li></ol><p>One of the standout features of the Store Gateway is its minimal local disk space requirement. It uses a small amount of disk space for caching metadata about the data stored in the object storage bucket. This cache:</p><ul><li><strong>Improves Restart Times</strong>: The cached metadata helps in speeding up the startup process after a restart.</li><li><strong>Minimal Disk Usage</strong>: The cache rarely exceeds a few gigabytes, making it lightweight and easy to manage.</li></ul><p><a href="https://thanos.io/tip/components/compact.md/"><strong>4. Thanos Compactor</strong></a></p><p>Scans the object storage bucket and performs necessary compaction. It is also responsible for creating downsampled copies of data to speed up queries. This dual function ensures that the data remains optimized for efficient querying and storage.</p><p><strong>Key Features:</strong></p><ol><li><strong>Compaction:</strong> Merges smaller blocks of time-series data into larger ones, reducing the overhead during query execution.</li><li><strong>Downsampling:</strong> Generates lower-resolution data, which is quicker to query for longer time ranges.</li></ol><blockquote>The Compactor is not in the critical path of querying or data backup, meaning it can be run as a periodic batch job or be left running to always compact data as soon as possible. For effective data processing, it is recommended to provide 100–300GB of local disk space.</blockquote><p>Among Thanos components, only the Compactor, Sidecar, Receive, and Ruler should have write access to the object storage. Of these, only the Compactor is authorized to delete data.</p><p><strong>5. Other components</strong></p><p>There are a few other elements which I will not cover in this post, you can find the list of all of the components <a href="https://thanos.io/tip/components/">here</a>.</p><ul><li><a href="https://thanos.io/tip/components/rule.md/">Thanos Ruler</a></li><li><a href="https://thanos.io/tip/components/receive.md/#receiver">Thanos Receiver</a></li><li><a href="https://thanos.io/tip/components/tools.md/">Thanos Tools</a></li></ul><p><strong>Congratulations </strong><br>That will be it for this article. Make sure to check out <a href="https://medium.com/@dast04/full-installation-monitoring-with-prometheus-thanos-part-2-2-fe98fcdbe448">Part 2</a> for the full installation process.<br>Congratulations on making it all the way here. Click 👏to let me know you found this article useful.</p><p><strong>Want to learn more?</strong></p><ul><li><a href="https://medium.com/@dast04"><strong>Explore</strong></a><strong> </strong>additional articles I’ve written</li><li><a href="https://www.linkedin.com/in/danielstankowski/"><strong>Subscribe</strong></a><strong> </strong>let’s connect on LinkedIn!</li></ul><p>That will be it for this article. Make sure to check out <a href="https://medium.com/@dast04/full-installation-monitoring-with-prometheus-thanos-part-2-2-fe98fcdbe448">Part 2</a> for the full installation process.<br>Congratulations on making it all the way here. Click</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*gxuHnJgPgh_qlFGE.jpg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8bc500a9c6a8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Full Installation — Monitoring with Prometheus & Thanos [Part 2/2]]]></title>
            <link>https://medium.com/@dast04/full-installation-monitoring-with-prometheus-thanos-part-2-2-fe98fcdbe448?source=rss-6374c4b54e6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe98fcdbe448</guid>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[thanos]]></category>
            <dc:creator><![CDATA[Daniello]]></dc:creator>
            <pubDate>Sun, 30 Jun 2024 09:28:28 GMT</pubDate>
            <atom:updated>2025-02-02T15:09:06.666Z</atom:updated>
            <content:encoded><![CDATA[<h3>Deployment — Monitoring with Prometheus &amp; Thanos [Part 2/2]</h3><p>If you have read <a href="https://medium.com/@dast04/theory-monitoring-with-prometheus-thanos-part-1-2-8bc500a9c6a8">Part 1</a>, you are ready for the installation process, if you haven’t yet, I recommend you do so, in this section I will not be spending time on explanations on theory but focus on the practical aspect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rc6sUmOc8aof0_XiaruOVw.png" /><figcaption>Example architecture — by Author</figcaption></figure><h4>Prerequisites &amp; Assumptions</h4><p>The prerequisites for this article are as follows:</p><ul><li>Kubernetes cluster (using EKS)</li><li><a href="https://helm.sh/docs/helm/helm_install/">Helm</a></li><li>monitoring namespace — to create new namespace kubectl create ns monitoring</li><li><em>I am going to be using AWS (S3, LB, EKS and EFS) but you can adopt the code to your cloud platform.</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*PJ4Dp6EW1TktZeAJxUDf_g.png" /><figcaption>Checklist for the whole process.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*smKvsgfSQoFJL0Ht.jpg" /><figcaption>:)</figcaption></figure><h4>1. Data persistence for prometheus and alertmanager</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/605/1*pdHyg5rrNIf8WbuYdpKxdg.png" /></figure><p>First we will create StorageClass to enable persistence for Alertmanager and Prometheus.<br>In my setup I will be storing the data in AWS EFS, but you can chose provisioner that fits your needs and modify the manifest accordingly.</p><p><strong>StorageClass</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/40e2cd631b426018e9de423253c72524/href">https://medium.com/media/40e2cd631b426018e9de423253c72524/href</a></iframe><p>Creating the storage classes in the monitoring namespace.</p><pre>kubectl apply -f prometheus-storage-class.yaml -n monitoring</pre><p>With the created storage class, we can move on to create Persistent Volumes. Because I am using an EFS I first need to create it in AWS console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g_JIHw6ZOahD-Xt1GgUeAg.png" /><figcaption>Amazon EFS</figcaption></figure><p>After creating the EFS, make sure to mount it, you can find instruction on how to do that after you create EFS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/258/1*W6vWXLwbHkcsiKdURE_8Ag.png" /><figcaption>Click <strong>Attach</strong> to view available mount options.</figcaption></figure><p>Now let’s create Persistent Volume.</p><p><strong>PersistentVolume</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e64a611ff9f6512246e2667a708066e2/href">https://medium.com/media/e64a611ff9f6512246e2667a708066e2/href</a></iframe><pre>k apply -f pv-prometheus.yaml -f pv-alertmanager.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/599/1*NNt9Nx9t4qBaHT0QeiBJ-w.png" /><figcaption>We are done with first step!</figcaption></figure><h4>2. Create an S3 bucket</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/201/0*DPo2I25GlRh9N4_3" /></figure><p>We first create an S3 bucket in AWS Region local to the Prometheus environment (we do not want to introduce unnecessary) latency issues.</p><p>Next, we create IAM policy, where &lt;bucket&gt; is your S3 bucket name.</p><blockquote>If you will face issues refer to the following resources: <a href="https://thanos.io/tip/thanos/storage.md/#object-storage--data-format">Thanos docs </a>, <a href="https://aws.amazon.com/blogs/opensource/improving-ha-and-long-term-storage-for-prometheus-using-thanos-on-eks-with-s3/">blog </a>post by Raj Bagwe or <a href="https://medium.com/@yakuphanbilgic3/thanos-for-multi-cluster-monitoring-in-aws-eks-clusters-using-s3-storage-09c684e05e2b">blog</a> post by Yakuphan.</blockquote><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Sid&quot;: &quot;Statement&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;s3:ListBucket&quot;,<br>                &quot;s3:GetObject&quot;,<br>                &quot;s3:DeleteObject&quot;,<br>                &quot;s3:PutObject&quot;<br>            ],<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:s3:::&lt;bucket&gt;/*&quot;,<br>                &quot;arn:aws:s3:::&lt;bucket&gt;&quot;<br>            ]<br>        }<br>    ]<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/441/1*73b4_XR1t_jo-tkTPndIXg.png" /></figure><h4>3. Deployment of <a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack">kube-prometheus-stack</a> (with Thanos sidecar)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Wq9dHUNlR0-CmIAr.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/1*mUSBL-ujjV_EuGDSlzdgdw.png" /></figure><p>The first step will involve installing Prometheus stack on both <em>master</em> and <em>slave</em> clusters using<em> </em><a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack"><em>helm chart.</em></a></p><pre>helm repo add prometheus-community https://prometheus-community.github.io/helm-charts<br>helm repo update</pre><p>We can start with downloading the values.yaml file, which we will edit to our needs, it is best practice to “freeze” the version.</p><blockquote><strong>Note!</strong> Change the chart version to fit your needs, at the time of this article the latest version is “60.2.0&quot;, for all the releases <a href="https://github.com/prometheus-community/helm-charts/releases">click here</a>.</blockquote><pre>helm show values prometheus-community/kube-prometheus-stack --version &quot;60.2.0&quot; &gt; values.yaml</pre><p>Let’s create a shell script to make installation faster called install_prom.sh , change execution permissions chmod +x install_prom.sh , now we will have an easy way to deploy the helm chart by executing sh install_prom.sh .</p><pre>NAMESPACE=monitoring<br>RELEASE_NAME=monitoring<br><br>helm upgrade --install $RELEASE_NAME prometheus-community/kube-prometheus-stack \<br>    --namespace $NAMESPACE \<br>    --values values.yaml \<br>    --version &quot;60.2.0&quot;</pre><p>The default values.yaml file as well as one with necessarily changes you can find in the GitHub repo for this article.</p><p><a href="https://github.com/danielstankw/Thanos_prometheus/tree/main">GitHub - danielstankw/Thanos_prometheus</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/535/1*ySfnwwa1BUulzfrcRcEQSw.png" /></figure><p>Below I present to you sections of the values.yaml file that were modified.</p><p><strong>Modification of values file for kube-prometheus-stack</strong></p><p><strong>Step 1</strong><br>Change the port type from <em>ClusterIP </em>to <em>NodePort </em>on Alertmanager service to make it accessible via http://&lt;POD_IP&gt;:&lt;SERVICE_PORT&gt;. <br>Additionally enable persistence by creating Persistent Volume Claim (PVC) (<em>PV was created in previous section</em>)</p><pre>alertmanager:<br>  service:<br>    type: NodePort<br>  alertmanagerSpec:<br>    storage:<br>      volumeClaimTemplate:<br>        metadata:<br>          name: alertmanager-pvc<br>        spec:<br>          storageClassName: alertmanager<br>          accessModes: [&quot;ReadWriteOnce&quot;]<br>          resources:<br>            requests:<br>              storage: 50Gi</pre><p><strong>Step 2<br></strong>Change the Grafana dashboard password — it is recommended to use secrets instead of hardcoding the values.<br><strong>For the main cluster only</strong>! — enable multi-cluster support.</p><pre>grafana:<br># user: admin<br># default password: admin<br>  adminPassword: &lt;YOUR_PSW_HERE&gt;<br>  sidecar:<br>    dashboards:<br>      multicluster:<br>        global:<br>          enabled: true</pre><p><strong>Step 3 </strong><br>To enable cross cluster metric querying the service needs to be able to handle traffic from outside of the cluster. We can chose here to set up an ingress, but in this case I will be using AWS <strong>network</strong> load balancer.</p><p>If you decide to go with an Ingress option you can either define it in the values.yaml</p><pre>  # Ingress exposes thanos sidecar outside the cluster<br>  thanosIngress:<br>    enabled: false</pre><p>or you can expose the Thanos sidecar service as <em>NodePort </em>type and then define Ingress as separate file, to route the traffic.</p><blockquote><strong><em>AWS LoadBalancer</em></strong><br>List of available <a href="https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/guide/service/annotations/">annotations</a> and a general guide.</blockquote><pre>prometheus:<br>  # Service for thanos service discovery on sidecar<br>  # Enable this can make Thanos Query can use<br>  # `--store=dnssrv+_grpc._tcp.${kube-prometheus-stack.fullname}-thanos-discovery.${namespace}.svc.cluster.local` to discovery<br>  # Thanos sidecar on prometheus nodes<br>  thanosService:<br>    annotations:<br>      service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: &lt;TAGS&gt;<br>      service.beta.kubernetes.io/aws-load-balancer-internal: &quot;true&quot;<br>      service.beta.kubernetes.io/aws-load-balancer-subnets: &lt;subnet-1&gt;, &lt;subnet-2&gt;<br>      service.beta.kubernetes.io/aws-load-balancer-type: nlb<br>      &lt;KEY&gt; : &lt;VALUE&gt;<br>      &lt;KEY&gt; : &lt;VALUE&gt;<br>      &lt;KEY&gt; : &lt;VALUE&gt;<br>    type: LoadBalancer<br>    clusterIP: &quot;&quot;<br>  <br>  # ServiceMonitor to scrape Sidecar metrics<br>  # Needs thanosService to be enabled as well<br>  thanosServiceMonitor:<br>    enabled: true</pre><p><strong>Step 4<br></strong>By default the Thanos sidecar sends data to the Object Store <strong>every 2 hours.<br></strong>Therefore the<strong> </strong>retention of Prometheus is recommended to not be lower than three times the min block duration, <strong>so 6 hours. <br></strong>Disabling compaction is necessarily if user wishes to store long term metrics in the Object Store.</p><pre>prometheus:<br>  service:<br>    type: NodePort<br>  <br>  serviceMonitor: <br>  # If true, pass --storage.tsdb.max-block-duration=2h to prometheus. <br>  # This is already done if using Thanos<br>    disableCompaction: true</pre><pre>prometheus:<br>  prometheusSpec:<br>    # External labels to add to any time series or <br>    # alerts when communicating with external systems<br>     externalLabels:<br>       cluster: cluster_1<br>       env: dev    <br>    <br>    # Prometheus metrics retention time in PV<br>     retention: 6h<br>    # High availability<br>     replicas: 2 <br><br>    resources:<br>      requests:<br>        cpu: 512m<br>        memory: 3072Mi<br>      limits:<br>        cpu: 512m<br>        memory: 4096Mi<br><br>    storageSpec:<br>      volumeClaimTemplate:<br>        metadata: <br>          name: prometheus-pvc<br>        spec:<br>          storageClassName: prometheus<br>          accessModes: [&quot;ReadWriteOnce&quot;]<br>          resources:<br>            requests:<br>              storage: 50Gi</pre><p><strong>Step 5</strong><br>Enable Thanos sidecar and creation of config secret. <br>With it we can write data to the Object Store (S3 in our case), to enable long-term metrics persistence.</p><pre>prometheus:<br>  prometheusSpec:<br>    thanos:<br>      baseImage: quay.io/thanos/thanos<br>      version: v0.35.1<br>      objectStorageConfig:<br>        existingSecret:<br>          key: thanos-storage-config.yaml<br>          name: thanos-storage-config </pre><p>Where the secret has the following structure:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/557c8753bfb960f22a22629037542b92/href">https://medium.com/media/557c8753bfb960f22a22629037542b92/href</a></iframe><p>We can create it in monitoring namespace via:<br>To verify secret was created: kubectl get secrets</p><pre>kubectl -n monitoring create secret generic thanos-storage-config --from-file=thanos-storage-config.yaml=thanos-storage-config.yaml</pre><p>With the modified values.yaml we can deploy our helm chart.</p><pre>sh install_prom.sh<br>helm list</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pr7lW47_GJlIHwakwlG-pA.png" /><figcaption>helm list</figcaption></figure><p>We can see that the helm deployment is active and has name of monitoring</p><p>We see that all pods are running properly!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*yyxi6vyiyMzhExJTTPjW5w.png" /><figcaption>kubectl get pods -n monitoring</figcaption></figure><p>We can also see that the PVC was created and PV was bound.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U9hzwzrypyuU3pQNAE7dXg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/516/1*aAEstaspnSoLdfOv3YzhFg.png" /></figure><h4>Investigation of the Prometheus pod!</h4><pre>k describe pod prometheus-monitoring-kube-prometheus-prometheus-0</pre><p>As we can see the pod contains 3 containers: prometheus, thanos-sidecar and config-reloader. We will focus on the first two.</p><blockquote><strong>NOTE: You don’t need to modify anything, the below examples are for the purpose of expanding ones understanding.</strong></blockquote><p><strong>prometheus</strong></p><ul><li>Thanos can watch changes in Prometheus configuration and refresh Prometheus configuration if --web.enable-lifecycle enabled.</li><li>The --storage.tsdb.min-block-duration and --storage.tsdb.max-block-duration must be set to equal values to disable local compaction in order to use Thanos sidecar upload, otherwise leave local compaction on if sidecar just exposes StoreAPI and your retention is normal.</li><li>Prometheus metrics retention time --storage.tsdb.retention.time=6h</li></ul><pre>Args:<br>      --web.console.templates=/etc/prometheus/consoles<br>      --web.console.libraries=/etc/prometheus/console_libraries<br>      --config.file=/etc/prometheus/config_out/prometheus.env.yaml<br>      --web.enable-lifecycle<br>      --web.external-url=http://monitoring-kube-prometheus-prometheus.monitoring:9090<br>      --web.route-prefix=/<br>      --storage.tsdb.retention.time=6h<br>      --storage.tsdb.path=/prometheus<br>      --storage.tsdb.wal-compression<br>      --web.config.file=/etc/prometheus/web_config/web-config.yaml<br>      --storage.tsdb.max-block-duration=2h<br>      --storage.tsdb.min-block-duration=2h</pre><p><strong>thanos-sidecar</strong></p><pre>Args:<br>      sidecar<br>      --prometheus.url=http://127.0.0.1:9090/<br>      --grpc-address=:10901<br>      --http-address=:10902<br>      --objstore.config=$(OBJSTORE_CONFIG)<br>      --tsdb.path=/prometheus<br>      --log.level=info<br>      --log.format=logfmt<br>      --prometheus.http-client={&quot;tls_config&quot;: {&quot;insecure_skip_verify&quot;:true}}</pre><p>Checking out the services.</p><ul><li>I have created externally Application Load Balancers for Grafana and Querier, which is why they appear here. (<em>will deploy querier in the next steps)</em></li><li>Network Load Balancer called monitoring-kube-prometheus-thanos-discovery was created successfully! — this is a <strong>crucial component </strong>for successful implementation of multi-cluster monitoring setup.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EIpZ5G2IEbSc65Xuz8SlgQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/1*wkPaoaYC4ER1DC5b-BWGuw.png" /></figure><h4>4. Deployment of Thanos components</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/1*O1a38yefuN7abbAPVzlVtw.png" /></figure><p>Depending on your setup you might need different components. In my case I will be using <em>Thanos Store, Thanos Compactor </em>and <em>Thanos Query — </em>if you are unfamiliar with the purpose of each of the components, check out Part 1 and Part 2.</p><p>There are various methods for deploying Thanos components, I will go with the “messier” one using <a href="https://github.com/thanos-io/kube-thanos/tree/main/manifests">kube-thanos</a> manifest files. Unlike when using helm chart, here each file needs to be applied by hand making the process more cumbersome but also more controllable — in my opinion.</p><p><a href="https://github.com/thanos-io/kube-thanos/tree/main/manifests">kube-thanos/manifests at main · thanos-io/kube-thanos</a></p><p><strong>4.1. Thanos Store</strong></p><p>You can find all the files for this section in here: &lt;LINK&gt;</p><p>Besides changes in the namespace there are some few important fields we will look into.</p><p>thanos-store-statefulSet.yaml</p><pre>      - args:<br>        - store<br>        - --log.level=info<br>        - --log.format=logfmt<br>        - --data-dir=/var/thanos/store<br>        - --grpc-address=0.0.0.0:10901<br>        - --http-address=0.0.0.0:10902<br>        - --objstore.config=$(OBJSTORE_CONFIG)<br>        - --ignore-deletion-marks-delay=24h<br>        env:<br>        - name: OBJSTORE_CONFIG<br>          valueFrom:<br>            secretKeyRef:<br>              key: thanos-storage-config.yaml<br>              name: thanos-storage-config</pre><p>Once we download all the files, we deploy them using</p><pre>kubectl apply -f thanos-store-statefulSet.yaml -f thanos-store-service.yaml -f thanos-store-serviceMonitor.yaml -f thanos-store-serviceAccount.yaml</pre><p><strong>4.2. Thanos Compactor</strong></p><p>Similary to the store we need to define in the:</p><p>thanos-compact-statefulSet.yaml</p><pre>        env:<br>        - name: OBJSTORE_CONFIG<br>          valueFrom:<br>            secretKeyRef:<br>              key: thanos-storage-config.yaml<br>              name: thanos-storage-config</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/381/1*Or4YKRuFY4e_dBsapjPdeg.png" /></figure><p><strong>4.3. Thanos Query</strong></p><p><strong>This is the ‘brain’ behind succesful multi-cluster setup! </strong>— if something won’t work with this setup then its most likely wrong configuration on the querier or improper exposure of the services.</p><p>The default includes:</p><pre>        - query<br>        - --grpc-address=0.0.0.0:10901<br>        - --http-address=0.0.0.0:9090<br>        - --log.level=info<br>        - --log.format=logfmt<br>        - --query.replica-label=prometheus_replica<br>        - --query.replica-label=rule_replica<br>        - --endpoint=dnssrv+_grpc._tcp.thanos-store.thanos.svc.cluster.local:10901<br>        - --endpoint=dnssrv+_grpc._tcp.thanos-receive-ingestor-default.thanos.svc.cluster.local:10901<br>        - --query.auto-downsampling</pre><p>We will make a few changes as well as explain the important fields:</p><p><strong>Stays the same</strong></p><ul><li>--endpoint=dnssrv+_grpc._tcp.thanos-store.thanos.svc.cluster.local:10901 This configuration instructs Thanos to discover all endpoints within the thanos-service service in the monitoring namespace, and use the declared port named grpc</li><li>--endpoint=dnssrv+_grpc._tcp.thanos-receive-ingestor-default.thanos.svc.cluster.local:10901 parameter tells Thanos Query to discover and connect to the Thanos Receive service using DNS SRV records, specifically looking for a gRPC service running on TCP protocol within the specified Kubernetes namespace and cluster domain, at port 10901.</li></ul><p><strong>Changes</strong></p><ul><li>--query.replica-label=&lt;YOUR_UNIQUE_LABEL_NAME&gt;: Labels to treat as a replica indicator along which data is deduplicated. We can define the field multiple times if needed:</li><li><strong>Sidecar Discovery: <br></strong>If we want to access from the side cluster we can define<br>--endpoint=dnssrv+_grpc._tcp.monitoring-kube-prometheus-thanos-discovery.monitoring.svc.cluster.local:10901 , which we can obtain via kubectl get svc <br>On the other hand if we wish to access metrics from a sidecar located in another cluster, we first need to set up the service — I used Network Load Balancer (see previous section) and then add the Load Balancer external IP in here. Which is why we added: (<em>annonymized for security)</em></li><li>--endpoint=dns+aXXXXXXX-XXXXXXX.elb.us-east-2.amazonaws.com:10901</li><li>--endpoint=dns+aYYYYYYY-YYYYYYY.elb.us-east-2.amazonaws.com:10901</li></ul><pre>      containers:<br>      - args:<br>        - query<br>        - --grpc-address=0.0.0.0:10901<br>        - --http-address=0.0.0.0:9090<br>        - --log.level=info<br>        - --log.format=logfmt<br>        - --query.replica-label=cluster <br>        - --endpoint=dnssrv+_grpc._tcp.thanos-store.monitoring.svc.cluster.local:10901<br>        - --endpoint=dnssrv+_grpc._tcp.thanos-receive-ingestor-default.monitoring.svc.cluster.local:10901<br>        # sidecar discovery on cluster_1 (master)<br>        # - --endpoint=dnssrv+_grpc._tcp.monitoring-kube-prometheus-thanos-discovery.monitoring.svc.cluster.local:10901<br>        - --endpoint=dns+aYYYYYYY-YYYYYYY.elb.us-east-2.amazonaws.com:10901<br>        # sidecar discovery on cluster_2 (slave)<br>        - --endpoint=dns+aXXXXXXX-XXXXXXX.elb.us-east-2.amazonaws.com:10901<br>        - --query.auto-downsampling<br>        env:<br>        - name: HOST_IP_ADDRESS<br>          valueFrom:<br>            fieldRef:<br>              fieldPath: status.hostIP<br>        image: quay.io/thanos/thanos:v0.35.1<br>        imagePullPolicy: IfNotPresent</pre><p>After applying we can see</p><p>NOTE: In my case cluster_1 label is called ‘toolset’ and cluster_2 vllmmodel, which is why you can view them here.</p><p>Disclaimer: I have no clue why there are two endpoints per cluster — if you do please let me know!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/1*rVF7lrf5ozMK6FEPNahvzw.png" /></figure><h4>5. Verification</h4><p>For both Thanos Query and Grafana services I have define AWS Load balancer to expose them outside of the cluster.</p><blockquote><strong>Note: </strong>To get necessarily selectors describe the default services that were created.</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8bbbf37c5693d0cac4cc07be2313c054/href">https://medium.com/media/8bbbf37c5693d0cac4cc07be2313c054/href</a></iframe><p><strong>5.1. Thanos Query</strong></p><p><strong>5.2. Grafana Dashboard</strong></p><p>After accessing the grafana, head out to the Data sources section visible on the left side.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*dAvIO3Tdcx-xYrEeBiWJYw.png" /><figcaption>Grafana &gt; Data sources</figcaption></figure><p>In the UI below we modify few parameters:</p><ul><li>Name: <strong>Thanos</strong> (+ toggle to set as default)</li><li>Connection — Prometheus server URL: <a href="http://thanos-query:9090/"><strong>http://thanos-query:9090</strong></a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cISESpUh2k15F5XkHyDPag.png" /><figcaption><em>Data Sources &gt; Add new data source &gt; choose Prometheus</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/1*o3mTS_R67rxJ7vXgNiUV4g.png" /><figcaption>Change the Performance section</figcaption></figure><p>With that we configured Thanos data source, now we can proceed with building dashboard.</p><p>Now when building grafana dashboard define new variable cluster</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/938/1*mmSuHOFPl2G-emA7iMnkLQ.png" /></figure><p>And you will be able to build dashboard with cluster</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*YJ6fg1xktQ-6k1TYW18WDg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/504/1*wIoNS_UtmRzfMWnt5EjJbA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/209/1*glJvcqjnyukuHof7-eHzrg.png" /></figure><p><strong>Congratulations </strong><br>Congratulations on making it all the way here. Click 👏to let me know you found this article useful.</p><p><strong>Want to learn more?</strong></p><ul><li><a href="https://medium.com/@dast04"><strong>Explore</strong></a><strong> </strong>additional articles I’ve written</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*gxuHnJgPgh_qlFGE.jpg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe98fcdbe448" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Writing Custom Prometheus Exporters (in Python) — Kubernetes]]></title>
            <link>https://medium.com/@dast04/writing-custom-prometheus-exporters-in-python-kubernetes-73626b66d78c?source=rss-6374c4b54e6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/73626b66d78c</guid>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[exporters]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Daniello]]></dc:creator>
            <pubDate>Mon, 22 Apr 2024 15:02:52 GMT</pubDate>
            <atom:updated>2024-08-22T14:43:12.312Z</atom:updated>
            <content:encoded><![CDATA[<h3>Writing Custom Prometheus Exporters (in Python) — Kubernetes</h3><p>In this article, I demonstrate the general steps needed in creating a custom Prometheus exporter, creating a starting block for anyone attempting to develop their exporters, and will do so via a simple example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*fXFsSdZKm2u3Ai2h.png" /><figcaption>Prometheus</figcaption></figure><p>Prerequisites:</p><ul><li>Kubernetes cluster</li><li>Installed Helm</li><li>Deployed Prometheus and Grafana (I recommend<a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack"> kube-prometheus-stack</a>)</li></ul><h3>Prometheus Exporter</h3><p>In this article, the exporter generates a random number between 1 and 10 and exposes it to Prometheus via /metrics endpoint.</p><p>Before we explore how to write <strong>custom</strong> exporters check if what you are looking for <a href="https://prometheus.io/docs/instrumenting/exporters/">exists already.</a></p><h4>What is an exporter?</h4><p>Exporter is an abstraction layer between the application and Prometheus, which takes application-formatted metrics and converts them to Prometheus metrics for consumption. Because Prometheus is an HTTP pull model, the exporter typically provides an endpoint where the Prometheus metrics can be scraped.</p><p>The relationship between Prometheus, the exporter, and the application in a Kubernetes environment can be visualized in the following way:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/0*fdaYES9GSqh3LzYf.png" /><figcaption>Image from : <a href="https://trstringer.com/quick-and-easy-prometheus-exporter/">https://trstringer.com/quick-and-easy-prometheus-exporter/</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*bldXpj0Vt8XTdtYn.jpg" /><figcaption>Let’s get practical …</figcaption></figure><h4>1. Create an exporter (Prometheus Client)</h4><p>The <a href="https://prometheus.github.io/client_python/getting-started/three-step-demo/">Prometheus client</a> is a Python client for writing exporters. Below you can find a simple example main.py which generates a random number between 1 and 10, every 30 seconds. Additionally, it starts a Prometheus HTTP server on port 8000.</p><blockquote>Note: Remember to createrequirements.txt file with necessary Python dependencies, in this case: prometheus-client</blockquote><p>main.py</p><pre>import random<br>import time<br>from prometheus_client import start_http_server, Gauge<br><br># Create a Prometheus gauge metric<br>random_number_metric = Gauge(&#39;random_number&#39;, &#39;Random number generated every 30 sec&#39;)<br><br>def generate_random_number():<br>    # Generate a random number between 1 and 10<br>    return random.randint(1, 10)<br><br>if __name__ == &#39;__main__&#39;:<br>    # Start the Prometheus HTTP server on port 8000<br>    start_http_server(8000)<br>    <br>    while True:<br>        # Generate a random number<br>        random_number = generate_random_number()<br>        print(&#39;Random number is: &#39;, random_number)<br>        # Set the value of the Prometheus metric<br>        random_number_metric.set(random_number)<br>        <br>        # Sleep for 30 sec<br>        time.sleep(30)</pre><p>In the Gauge the random_number is the name of the metric, while random number generated every 30 sec is a a help string or description for the metric. Metric types other than Gauge can be used, and their use depends on the metric you intend to measure for example:</p><ul><li>Counter: for metrics that can only be incremented and reset at process restart</li><li>Gauge: can go up and down</li><li>Summary: tracks the size and number of events</li><li>Histogram</li></ul><p>and more… (<em>for additional information check </em><a href="https://prometheus.github.io/client_python/instrumenting/"><em>LINK</em></a><em>)</em></p><h4>2. Verify the metrics</h4><p>We can verify that the exporter is working as intended</p><pre>curl -iL 127.0.0.1:8000/metrics</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/1*ubHUyY8_G_BBRhQesSPOYQ.png" /><figcaption>Output of the curl command.</figcaption></figure><h4>3. Containerizing an application</h4><p>Now is the time to dockerize our application. We will start with creating a Dockerfile for our application.</p><p>Dockerfile</p><pre>FROM python:3.11-alpine<br># This prevents Python from writing out pyc files<br>ENV PYTHONDONTWRITEBYTECODE 1<br># This keeps Python from buffering stdin/stdout<br>ENV PYTHONUNBUFFERED 1<br><br>WORKDIR /app<br><br>COPY requirements.txt /app/requirements.txt<br><br>RUN pip install -r requirements.txt<br><br>COPY main.py /app/main.py<br><br>EXPOSE 8000<br><br>ENTRYPOINT [&quot;python3&quot;, &quot;main.py&quot;]</pre><h4>5. Build the image and push it to the remote repository.</h4><p>This step will depend on your personal preference. I have used AWS ECR for that purpose.</p><p>We have written the exporter itself using prometheus-client, dockerized the application, and pushed it to the remote repository. <br>What is left to be done is deploying our application in Kubernetes, we will handle that in the steps below…</p><h4>6. Create manifest files (Deployment, Service and ServiceMonitor)</h4><p>We first create deployment.yaml and service.yaml files.</p><p>deployment.yaml</p><pre>kind: Deployment<br>apiVersion: apps/v1<br>metadata:<br>  name: random-num<br>spec:<br>  replicas: 1<br>  # match the pods controlled by deployment<br>  selector:<br>    matchLabels:<br>      app: random-num<br>  # pod template used by deploy to create a new pod.<br>  template:<br>    metadata:<br>      labels:<br>        app: random-num<br>    spec:<br>      containers:<br>        - name: random-num<br>          image: &lt;IMAGE_URL&gt;<br>          imagePullPolicy: Always<br>          ports:<br>            - containerPort: 8000</pre><p>service.yaml</p><p>We set the port to the number we wish, for example, 9102 -port on which the service will listen for incoming traffic inside the cluster. When traffic comes to port 9102 on the Service, it will be forwarded to port 8000 on the Pods that match the selector.</p><p>For development, we will leave the service type as ClusterIP<em> </em>(default) — for production, you would want to change it, for example to: LoadBalancer. You can also use NodePort service type, which doesn’t require port forwarding.</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: random-num<br>  labels:<br>    app: random-num<br>spec:<br>  selector:<br>    app: random-num<br>  ports:<br>    - name: metrics<br>      # Sets port on which the Service will listen for incoming traffic within the cluster.<br>      port: 9102<br>      # Specifies the port on the Pods to which the traffic should be forwarded. <br>      targetPort: 8000<br>      protocol: TCP</pre><p>After defining the above elements we will look into a key component that will link the Prometheus to our exporter, allowing for scraping the appropriate metrics — ServiceMonitor.</p><h4>ServiceMonitor</h4><p>A ServiceMonitor in Prometheus is a custom resource provided by the Prometheus Operator, which is used to automatically discover and monitor services running on Kubernetes. It allows Prometheus to dynamically discover endpoints of your applications and scrape their metrics.</p><blockquote><strong><em>ServiceMonitor </em></strong>acts as a bridge between your Kubernetes services/Pods and Prometheus, enabling automatic discovery and monitoring of your applications’ metrics without the need for manual configuration.</blockquote><p>I assume you have already installed Prometheus and Grafana in your K8s cluster. If you didn’t I recommend you use<a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack"> kube-prometheus-stack </a>which is the easiest and fastest way to deploy the stack needed for monitoring.</p><p>We check the existing service-monitors:</p><pre># Define alias for convenience<br>alias k=&quot;kubectl&quot;<br><br># Display existing service monitors<br>k get servicemonitor</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/401/1*b1Gfcmyf7NkGfw9eE1zlGw.png" /><figcaption>result of: kubectl get servicemonitor</figcaption></figure><p>By default when installing Prometheus with <em>kube-prometheus-stack </em>you will get access to node-exporter which scrapes information about your Kubernetes cluster and exposes the information to Prometheus/ grafana. Let’s view its manifest file.</p><pre>k get servicemonitor monitoring-prometheus-node-exporter -o yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/415/1*_LRNwFFeAkT-h3X2LTzy8A.png" /><figcaption>Investigating service-monitor manifest</figcaption></figure><p>For Prometheus to <strong>discover our service-monitor it does so through a release label</strong>. In our configuration we defined the label to be monitoring, this is crucial to check as we will need this release label value in our service monitor!</p><p>servicemonitor.yaml</p><blockquote>Note: The label release is set to monitoring though you need to check it for yourself as it can be different for you. The scraping interval for metrics from the service monitored by the ServiceMonitor is every 10 seconds.</blockquote><pre>apiVersion: monitoring.coreos.com/v1<br>kind: ServiceMonitor<br>metadata:<br>  name: random-num-servicemonitor<br>  namespace: prometheus<br>  labels:<br>    release: monitoring<br>spec:<br>  selector:<br>    matchLabels:<br>      app: random-num<br>  endpoints:<br>    - port: metrics<br>      interval: 10s</pre><p>Now that we have all of them in place we can deploy them</p><pre>k apply -f service.yaml<br>k apply -f servicemonitor.yaml<br>k apply -f deployment.yaml</pre><p>To enable external access to the/metrics endpoint (port on the container) we will perform port forwarding.</p><pre>k port-forward service/&lt;svc-name&gt; &lt;host-port&gt;</pre><p>In our case:</p><pre>k port-forward service/random-num 9102</pre><p>Accessing the content of the /metrics endpoint.</p><pre># 127.0.0.1 or localhost - &quot;loopback&quot; address<br>curl -iL 127.0.0.1:9102</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/1*ubHUyY8_G_BBRhQesSPOYQ.png" /><figcaption>Sample output</figcaption></figure><h4>Visualizing the scrapped metrics in Prometheus and Grafana</h4><p>To access the Prometheus URL, follow these steps:</p><pre># retrieve the Node Port IP associated with <br># prometheus-monitoring-kube-prometheus-prometheus-0 ex. 10.137.XXX.XXX<br>k get pods -o wide<br><br># obtain the port mapping ex. 9090:30000/TCP<br>k get svc </pre><p>We then access using the URL: http://10.137.XXX.XXX:30000</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ARpxGhsn779QRz7bsXyVFg.png" /></figure><p>Recall that the name of the metrics random_number</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WzN4t0-vHAfN09bVBeSD3w.png" /></figure><p>Great! — we can see that Prometheus is successfully discovering the metrics we wish to scrape. Now let’s proceed to Grafana to visualize the same results.</p><p>We can similarly access Grafana as we accessed Prometheus i.e. using IP and port. After logging in we can go and create a new Dashboard, while choosing as source <em>Prometheus.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RPj9_0v_uuoVKkkuZa7Jbw.png" /><figcaption>New Dashboard</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jHmMkHJWFix_210pxmLkjQ.png" /><figcaption>Choosing “random_numer” metrics</figcaption></figure><p>After selecting random_number as our metric we click: “Run queries”. We can then manipulate the visualization in a way that fits our needs. <br>Below I present a table view.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2I29XuHAkRBEC6xhpeKLwg.png" /></figure><p>To validate match between Grafana results and atual results we can display the logs of our random_num pod</p><pre>k logs -f pod &lt;random_num_pod_name&gt;</pre><p>The numbers should match.</p><h3>Conclusion</h3><p>This will be it for today. I hope this short guide will help you out in creating your custom exporters and will lay a foundation for some of the more complex exporters.</p><h3>You made it!</h3><p>If you like the style of the article and found the information useful, please consider clicking the 👏 emoji.</p><p><strong><em>Got any questions? Want to connect? — Feel free to add me on </em></strong><a href="https://www.linkedin.com/in/danielstankowski/"><strong><em>LinkedIn</em></strong></a></p><p>Good luck!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/0*oBbpLB8clKPqtdvd.jpg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=73626b66d78c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes — Introduction & Architecture☸️- part 1]]></title>
            <link>https://medium.com/@dast04/kubernetes-introduction-architecture-%EF%B8%8F-part-1-33fbd2ab02d4?source=rss-6374c4b54e6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/33fbd2ab02d4</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[orchestration]]></category>
            <category><![CDATA[k8s]]></category>
            <dc:creator><![CDATA[Daniello]]></dc:creator>
            <pubDate>Sun, 19 Nov 2023 19:36:15 GMT</pubDate>
            <atom:updated>2023-11-19T21:18:27.106Z</atom:updated>
            <content:encoded><![CDATA[<h3>Kubernetes — Introduction &amp; Architecture☸️- part 1</h3><p>Kubernetes also known as K8s, is an open-source system for automating deployment, scaling, and management of containerized applications. <br>In this article, we will dive into the basics of Kubernetes, and outline the most important functionality of Kubernetes as well as its components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*MpUHCyZLvTpXMi5g.jpeg" /><figcaption>:)</figcaption></figure><blockquote><strong><em>What is K8s?</em></strong> — Kubernetes is often denoted as K8s. By taking the first and last letter of the word Kubernetes we are left with <strong>K_s</strong>. There are 8 letters between the first and last letter which results in <strong>K</strong>ubernete<strong>s → K</strong>8<strong>s.</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h3cJr34prc95-4Gl3baw9g.png" /></figure><h3><strong>Fundamentals</strong></h3><p>To understand Kubernetes let’s answer some basic questions first.</p><p><strong>Q: Why use K8s? <br>Q: What problem does it solve? How are those problems solved?</strong></p><p>Imagine you are running applications in containers, for example, Docker. <em>Q: What will you do if the container goes down? </em><br>A: The simplest solution is to restart it manually but that is far from an optimal approach and can lead to significant downtime.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*qdE77L04YA98B4cG.jpg" /></figure><p>This is especially the case when we have multiple containers across various virtual machines, in large-scale applications. <br>Kubernetes automatically checks the health of each pod, and restarts or replaces it.</p><p>Besides orchestrating containers, K8s can do much much more. Let’s look in detail at some of the capabilities of Kubernetes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/1*gtXebloPAjQyhs9J7guloA.png" /><figcaption>Kubernetes …</figcaption></figure><ol><li><strong>Container Orchestration</strong>: Management and coordination of containers, ensuring that applications are deployed and run consistently across various VMs.<br><em>Kubernetes defines the desired state of applications through YAML files. It then works to maintain this state (actual → desired) by managing containers. It schedules containers based on the available resources.</em></li><li><strong>Scalability</strong>: Automatic scaling of application deployments, making it easier to handle varying workloads.<br><em>Automatically adjusts the number of Pod replicas (aka horizontal scaling) based on the available resources ex. CPU usage. It then decreases or increases the number of replicas to meet the demand.</em></li><li><strong>Self-Healing</strong>: Ensuring applications are always running by restarting failed containers, replacing and rescheduling containers when nodes die, and killing containers that don’t respond to user-defined health checks.</li><li><strong>Rollout:</strong> Kubernetes updates applications using rolling updates. When new versions of an application or configuration are deployed, K8s gradually replaces instances of the old version with the new one, without downtime.<br><strong>Rollbacks: </strong>Enables easy rollback to previous versions if something goes wrong.</li><li><strong>Load Balancing:</strong> Automatically distributing network traffic. Kubernetes provides IPs and DNS names to containers, helping to balance loads and make the application accessible.<br><em>Kubernetes automatically distributes incoming network traffic across Pods.</em></li></ol><h3>Kubernetes Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/971/1*9-mMrpRVpEdLIT_ztfWfUA.png" /><figcaption>Kubernetes Architecture — Image by the Author</figcaption></figure><h4>Control Plane (aka Master Node)</h4><p>Manages worker nodes, and runs <strong>system</strong> Pods only. <br>Its main components are:</p><ul><li><strong><em>API Server</em>: </strong>Acts as a gateway to the cluster and as an interaction point for users and Kubernetes components through various interfaces including UI and CLI. It serves as an endpoint for the Kubernetes CLI (kubectl), facilitating all operations within a K8s cluster.</li><li><strong><em>Scheduler</em></strong>: Assigns new Pods, to nodes within the cluster. It plans and distributes workloads efficiently across the cluster. In collaboration with the API Server, the Scheduler determines the optimal nodes for deploying applications or workloads, taking into account both the resources available on the nodes and those required by the workloads. While it decides where to place Pods for optimal performance, the actual execution of these scheduling decisions is carried out by the kubelet.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/867/1*dytocKG9B8-JDnXequEgkQ.png" /><figcaption>Kubernetes Cluster — Image by the Author</figcaption></figure><ul><li><strong><em>Controller manager:</em></strong> Operates multiple controller processes. These controllers constantly monitor the state of the cluster, striving to match the actual state with the desired state (<em>maintaining for example 5 replicas of the Pod at any given time). </em>This includes managing various aspects like nodes, replications, endpoints, and service accounts.</li><li><strong><em>Cloud Controller Manager</em></strong>: This component manages controllers specific to cloud providers, handling resources external to the K8s cluster. It is active only if the Kubernetes cluster is cloud-based. For local or non-cloud clusters, this controller is not in operation. Its primary role is to interface with cloud providers for managing nodes, load balancers, and routing paths.</li><li><strong><em>Etcd</em></strong>: A key-value store of the cluster state. It maintains the state of the K8s cluster, including all API objects, ensuring data persistence and reliability across the cluster.</li></ul><h4>Worker Node</h4><p>The worker node is responsible for running containerized applications in a K8s cluster.</p><ul><li><strong><em>Pods</em></strong>: The smallest deployable units in K8s, representing an abstraction over one or more containers. Typically, a Pod contains a single container, but it can also encompass multiple containers if there are closely related sub-applications or services that need to function together. Containers within a Pod share resources and network space, enabling them to interact closely as needed.</li><li><strong><em>Kubelet</em></strong>: An agent running on each Worker Node, tasked with ensuring that the Pods on the node are running as expected. It maintains regular communication with the API Server on the master node and is a crucial component for managing the overall cluster through commands issued via kubectl.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/787/1*XyQoP2LGT1zwoc2kBiC4eQ.png" /><figcaption>Worker Node architecture — Image by the Author</figcaption></figure><ul><li><strong><em>Kube-proxy</em></strong>: Handles network communications within a Kubernetes cluster. It ensures that the necessary network rules are in place, allowing for efficient and secure communication between different components, particularly for services and Pods.</li><li><strong><em>Container runtime:</em></strong> Responsible for running containers on each node. Kubernetes supports several container runtimes, with Docker being the most widely used. Key responsibilities of the container runtime include pulling container images from a registry, starting and stopping containers, and managing other container-level operations.</li></ul><h3>Kubernetes Flow</h3><p>We have already covered the structure of the K8s cluster, to solidify our understanding below I present a typical workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/971/1*9-mMrpRVpEdLIT_ztfWfUA.png" /><figcaption>Kubernetes Architecture — Image by the Author</figcaption></figure><ol><li>Admin submits a deployment specification in YAMLorJSONformat. The specification details everything ranging from container images, number of replicas, network settings, volumes, and more. Next, the admin uses UI or kubectl to submit this deployment specification to the Kubernetes cluster.</li><li>API Server receives the request and stores the desired state in <em>etcd</em>.</li><li>The <em>Scheduler </em>watches for new Pods and assigns them to the nodes. Based on the internal algorithm the scheduler decides which node is best suited for each Pod. Next, the <em>scheduler </em>updates Pods information in the <em>API Server</em>, assigning it to the node.</li><li>Kubelet on the assigned nodes starts the container using the specified container runtime. When a kubelet on the node notices a Pod has been assigned to its node, it instructs the container runtime (ex. docker) to pull (download) the necessary container image (if not yet present) and start the container. The kubelet then continuously monitors the state of the Pods and reports back to the API server to keep the system informed.</li><li>Kube-proxy configures the network so the new Pods can communicate with other Pods and external services.</li></ol><p>Through these steps, Kubernetes automates the deployment and management of containerized applications. It ensures that the actual state of the system matches the desired state specified by the admin. If a Pod fails, the system recognizes the state divergence and starts a new Pod to maintain the desired state.</p><p>The Controller Manager’s activities run in parallel with the steps previously outlined, and its controllers are constantly interacting with the API Server to watch the current state of the cluster and make changes where necessary.</p><h3>K8s Components —example</h3><p>Let’s picture the following. <br>We have a (worker) node which is a physical or virtual machine. Two Pods are created inside the node, one for an application and the other for a database.</p><p>Upon creation, Pods are assigned an internal IP address, this address however is ephemeral — it doesn’t persist. This means that in a case Pod crashes, the IP address assigned to it will get deleted i.e. won’t persist. As you can imagine that is not great news, if we wish to communicate with a Pod via internal IP, the IP address will change often. Kubernetes offers a solution — <em>Service</em>. <br>Service sets a permanent IP address attached to the Pod, which doesn’t change with the “death” of the Pod.</p><p>Shifting our focus to the database Pod. By default there is no storage management done by K8s, instead, it is in user’s responsibility to attach storage to the Pod (the storage is outside of the cluster and can be either a path to a specific location on the local host or remote cloud storage).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*CBR_wZL_bin8mfpq4dWJGg.png" /></figure><p>Working with databases requires defining values such as database URL, USERNAME, and PASSWORD. It would be convenient not to have them hardcoded into the image of the database application so that they can be easily modified as needed. This is where the <em>ConfigMap </em>and <em>Secret </em>configuration files come into play.</p><p><em>ConfigMap</em> is a file where the user can configure properties such as database URL. For more sensitive data such as database username and password, the variables can be stored in a file called <em>Secret </em>which unlike <em>ConfigMap</em> is base64 encoded for enhanced security. <br>Once those configuration files are populated the variables can be shared with Pods for example via environmental variables.</p><p>As mentioned before in order to persist the IP address a service is used. However, if the user desires to connect externally via the service to the Pod the URL will look as follows</p><blockquote><em>http:app-service-ip:port → http:192.10.5.01:8282</em></blockquote><p>This is a sufficient solution for development setup but not for production. For production, we probably would prefer something of this type (which is not only more convenient but also more secure, as it is secure protocol).</p><blockquote>https:my-app.com</blockquote><p>This is where Ingress comes into play. Ingress is a “middleman” between the request and <em>Service</em>, it defines routing rules that forward requests to internal services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/1*A0wsDyKZiH0li8nmFbs_Wg.png" /><figcaption>Service — Ingress: Image by the Author</figcaption></figure><p><strong><em>What happens when the Pod goes down? </em><br></strong>Kubernetes offers functionality to replicate our Pods. Instead of using Pods directly when working with clusters, we use <em>Deployments</em> which are abstractions on top of Pods and are a form of a blueprint, that works with multiple Pods at the same time. By using deployments that manage multiple replicas of the pods, in case one of the pods crashes, the application will continue running on the replica, while K8s will restart the non-working pod, resulting in no downtime in the application.</p><p><strong><em>Can we use deployments for the pods that have databases? <br></em></strong>No! — Deployments are stateless i.e. they do not preserve any information. Databases on the other hand are stateful — they preserve the state that can be used in the future. When dealing with stateful applications in Kubernetes the way in which replicas are created and managed is slightly more involved and I will cover it in the next articles, but what is important to remember is that for databases <em>StatefulSets</em> are used instead of <em>Deployments.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/517/1*6qG-Py7HycBNAxtEcuM33w.png" /><figcaption>Illustration of 2 Nodes, with replicas of application and database Pods.</figcaption></figure><h3>You made it!</h3><p>If you like the style of the article and found the information useful, please consider clicking the 👏 emoji.</p><p><strong><em>Got any questions? Want to connect? — Feel free to add me on </em></strong><a href="https://www.linkedin.com/in/danielstankowski/"><strong><em>LinkedIn</em></strong></a></p><p>Good luck!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*Oinh6RvDPGhwMxfV.jpg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=33fbd2ab02d4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running Airflow and Mlflow with docker-compose [2023]]]></title>
            <link>https://medium.com/@dast04/running-airflow-with-docker-compose-2023-for-machine-learning-a78eeadc00cd?source=rss-6374c4b54e6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/a78eeadc00cd</guid>
            <category><![CDATA[mlflow]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Daniello]]></dc:creator>
            <pubDate>Tue, 10 Oct 2023 09:11:25 GMT</pubDate>
            <atom:updated>2023-10-13T07:05:18.328Z</atom:updated>
            <content:encoded><![CDATA[<p>Welcome to the guide on running Apache Airflow and Mlflow with Docker! Before we dive into the details feel free to check my other articles: <a href="https://medium.com/@dast04/git-version-control-how-to-get-started-9493dcbd3360">Tutorial on Git</a> &amp; <a href="https://medium.com/@dast04/mastering-anaconda-intuition-and-practice-for-beginners-6db55994060c">Tutorial on Anaconda and virtual environment.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/925/1*EcD__eYHnd6Gi7H9eaY83w.png" /><figcaption>Docker + Airflow</figcaption></figure><h3>Table of Content</h3><p>After reading this article you will:</p><ul><li>Learn how to run Apache Airflow in Docker with docker-compose.</li><li>Understand the airflow configurations and how to modify it for your needs.</li><li>Learn how to extend an airflow image using Dockerfile allowing you for example to add PyPI dependencies (<em>if your Dags require them</em>).</li><li>Track ML experiments using Mlflow and run Mlflow UI server inside Docker.</li></ul><blockquote>Note: In this article I will not dive deep into basics of Docker and Airflow. <br>If you need to learn more about them, check out my other articles.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*V0253skTlvcBQYm4.jpg" /></figure><h3>Step 1: Running Airflow in Docker + modifying docker-compose.</h3><blockquote>For official documentation check out the <a href="https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html">link</a>.</blockquote><p>In the rest of this article I will assume that you have both docker and docker-compose installed, if that is not the case please do so (there are many videos on YT about how to do it) . To check if you have everything installed, check the versions by running docker --version and docker-compose --version.</p><p>Let’s get going with process of running airflow in Docker:</p><ol><li>Create a folder called <em>airflow</em>, and execute in terminal (in the airflow directory):</li></ol><pre>curl -LfO &#39;https://airflow.apache.org/docs/apache-airflow/2.7.1/docker-compose.yaml&#39;</pre><p>This will download the docker-compose.yaml file to the current directory.</p><blockquote>For the link used with the curl command, make sure to get the <a href="https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html#fetching-docker-compose-yaml">official link</a> for the latest version!</blockquote><p>2. By default the created docker-compose.yamlfile is quite long, we will make some modifications to make it simpler and fit our needs. I will first explain <strong>why </strong>we will delete and then I will explain <strong>what</strong>:</p><h4>The why … 🤷‍♂️</h4><p>For the purpose of this article, we will run tasks on <strong>single machine, with a small/medium sized workload.</strong> Because of this we won’t need a default CeleryExecutor and instead will use LocalExecutor . The choice of executor is crucial as it allows us to simplify the docker-compose file significantly, more about it below.</p><h4>Local vs Celery Executor — when to use which?</h4><p>The LocalExecutor should be sufficient if you want to run a small number of concurrent tasks on a single server. You should think of moving to CeleryExecutor when you feel that you need to distribute the jobs across multiple servers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*qyTrt1Sow7V_NJByw91HLg.png" /><figcaption>Comparison between Local and Celery Executor (credit: Author + ChatGTP4)</figcaption></figure><h4>Redis</h4><p>Redis is often used as a message broker when you’re using the CeleryExecutor, it helps to manage communication between the Airflow scheduler and worker. If you are using the LocalExecutorand running Airflow on a single machine, you don’t need Redis.</p><h4>Airflow Worker</h4><p>In Airflow, a worker is a component that executes the tasks given to it by the scheduler. When using the CeleryExecutor, workers allow Airflow to execute tasks on different machines.</p><p>You should use airflow-worker when you need to distribute task execution across multiple machines, especially in production environments with high workloads. In this article we won’t use it, as we are working with simpler setups and single-machine deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/0*1Gd6xZq3Fd6LTWQK.gif" /></figure><h4><strong>What we will change (Summary)</strong></h4><blockquote>For the fully changed docker-compose.yaml file please check the <a href="https://github.com/danielstankw/airflow-docker-compose/blob/main/docker-compose.yaml">GitHub repo</a>.</blockquote><p>We will make the following changes to the docker-compose.yaml file. Change the executor to LocalExecutor , comment/ delete redis service, airflow worker and flower services. Comment celery configuration as well as adjust the dependency section depends_on.</p><ul><li>Line 57: change CeleryExecutor→ LocalExecutor</li><li>Line 61, 62: comment out.</li><li>Line 65: change to False(we don’t want to have any example dags)</li><li>Line 83, 84: comment out.</li><li>Line 104–114: comment out.</li><li>Line 148–169: comment out.</li><li>Line 280–297: comment out.</li></ul><p><strong>The final version of the </strong><strong>docker-compose.yaml file can be found in </strong><a href="https://github.com/danielstankw/airflow-docker-compose/tree/main"><strong>this repo</strong></a><strong>.</strong></p><p>With the modified docker-compose.yaml file we can continue.</p><p>3. Create folders for dags/ logs and plugins</p><pre>mkdir -p ./dags ./logs ./plugins</pre><p>4. Generate .env file using:</p><pre>echo -e &quot;AIRFLOW_UID=$(id -u)&quot; &gt; .env</pre><p>5. Initialize the airflow database.</p><pre>docker-compose up airflow-init</pre><p>This will set all the images and login information for the Airflow.</p><blockquote>User: <strong>airflow, </strong>password: <strong>airflow</strong></blockquote><p>After initialization is complete, you should see a message like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/1*3iWCtuxx9XfpcVTm24iPUQ.png" /><figcaption>Image from official Airflow documentation.</figcaption></figure><p>6. Running container in detached mode:</p><pre>docker-compose up -d</pre><p>Now go to the <a href="http://localhost:8080/">http://localhost:8080/</a> and log in using user <em>airflow</em> and password <em>airflow. </em>Use docker ps to make sure the container is running.</p><blockquote>NOTE: It can take a few minutes for webserver to get online!</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pOoVCAjf5yInPti97CBjTQ.png" /><figcaption>Airflow webserver UI after logging in with the credentials.</figcaption></figure><h3>2. Extending an image … why?</h3><blockquote>For official documentation check out the <a href="https://airflow.apache.org/docs/docker-stack/build.html">link</a>.</blockquote><p>When running Airflow locally, you might want to use an extended image containing some additional dependencies — for example, adding new <strong>python packages </strong>or upgrading<strong> airflow providers</strong> to a later version. We will add those later but for now, lets create a simple Dockerfile.</p><p>When running Airflow DAGs we will need some python dependencies. Because we are running Airflow in Docker, we need to install dependencies <strong>inside </strong>Docker. We can do that by extending an Airflow image.</p><ol><li>Create a Dockerfile.airflow in the same the same directory where the docker-compose file is located.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*fiPhp-H8BM1vffPwOeHo3A.png" /><figcaption>Image by the Author</figcaption></figure><p>The image base i.e. <em>apache/airflow:2.7.1</em> is taken from the docker-compose file.</p><p>2. Next we will modify docker-compose.yaml file further. As noted by the comment we will comment out the <em>image</em> filed and uncomment <em>build </em>filed. We specify the context and dockerfile fields.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G6pe3Vrw66m3gk7FXojkbw.png" /><figcaption>Image by the Author</figcaption></figure><p>3. Last but not least create requirements.txt file where we will store dependencies needed for running our Dags.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*djiQ7aDDK7mH5BmZXfJVlQ.png" /><figcaption>Image by the Author</figcaption></figure><h3>3. Adding Mlflow</h3><p>So far we have successfully run an extended image of Airflow using Docker. We have copied the requirements.txt file into Docker and installed PyPI dependencies we defined.</p><p>In the following section I will show how to run Mlflow tracking server with the following configuration:</p><ul><li><em>Tracking: </em>local</li><li><em>Artifact Store: </em>local</li><li><em>Backend: </em>sqlite (local)</li></ul><blockquote>Note: In the articles to come, I will provide more information how run Mlflow on a cloud: with tracking on EC2, Artifact store using S3 bucket and Backend using PostgreSQL.</blockquote><ol><li>Similarly to previous section we will start with creating Dockerfile.mlflow file.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*pj3MjBBK9gkbH32rn-Hrgw.png" /></figure><p>2. Next we will modify docker-composefile.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*l61K_4dqYKP9Obqmk5vW7w.png" /></figure><h3>4. Thats it!</h3><p>Now that we defined everything we can build an image using docker-compose build . After which we can start the containers via docker-compose up .</p><p>With the running containers you can access:</p><ul><li>airflow webserver: <a href="http://localhost:8080/">http://localhost:8080</a></li><li>mlflow ui: <a href="http://localhost:5000/">http://localhost:5000</a></li></ul><h3>You made it!</h3><p>If you like the style of the article and found the information useful, please consider clicking the 👏 emoji.</p><p><strong><em>Got any questions? Want to connect? — Feel free to add me on </em></strong><a href="https://www.linkedin.com/in/danielstankowski/"><strong><em>LinkedIn</em></strong></a></p><p>Good luck!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*Ll-21Dp6eL7nYu5i.jpg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a78eeadc00cd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Git (Version Control) — How to get started?]]></title>
            <link>https://medium.com/@dast04/git-version-control-how-to-get-started-9493dcbd3360?source=rss-6374c4b54e6f------2</link>
            <guid isPermaLink="false">https://medium.com/p/9493dcbd3360</guid>
            <category><![CDATA[github]]></category>
            <category><![CDATA[version-control]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[git]]></category>
            <dc:creator><![CDATA[Daniello]]></dc:creator>
            <pubDate>Fri, 29 Sep 2023 06:47:01 GMT</pubDate>
            <atom:updated>2023-10-13T07:05:26.423Z</atom:updated>
            <content:encoded><![CDATA[<h3>Git (Version Control) — How to get started?</h3><p>In this article, we will discuss the usefulness of Git as a version control platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*thx7JBXPHeJ5Mv31v9KL5w.png" /><figcaption>Image by the Author</figcaption></figure><h3>The general workflow</h3><p>When we start out with our version control, we first want to define the <strong>Working Directory. </strong>This<strong> i</strong>s where we will work on our project and modify the files.</p><p>If we consider the file in the working directory, it can be in <strong>3</strong> possible stages:</p><ol><li><em>Modified:</em> This means the files with the updated changes are not yet stored in the local repository.</li><li><em>Staged:</em> Files with updated changes are marked to be committed, but aren’t yet committed.</li><li><em>Committed:</em> This means that the changes you made to your file are safely stored in the local repository.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q1ZGwoZh36bEOUDmXhrNKQ.png" /><figcaption>Git workflow scheme with commands (made by the Author)</figcaption></figure><blockquote>⚠️<strong>Terminology Alert: </strong>Repository aka Repo<em>: </em>a storage space where all the files, folders, and information related to a specific project are kept. It’s used in software development to store the code, documentation, images, and other items needed for a project.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*47Ss1Q5E7gMdwGgpquY8pA.png" /><figcaption>Types of Repositories (Image by the Author)</figcaption></figure><p><strong>Getting started with Git</strong></p><ol><li>Install Git: <a href="https://git-scm.com/book/en/v2/Getting-Started-Installing-Git"><strong>LINK</strong></a></li><li>Once installed we can check the version of Git with the following command:</li></ol><pre>git --version</pre><p>3. In order to set up Git and set up the local repository we first need to update our credentials. Those will be displayed when we make changes to the files, and they will identify us as the author of those changes.</p><pre># set the name of the user (any name you want to go by)<br>git config --global user.name &quot;YOUR_USERNAME&quot;<br><br># set the email of the user (any email you want to use so that you can be contacted)<br>git config --global user.email &quot;your_email@gmail.com&quot;<br><br># To check the info you just provided<br>git config --global --list </pre><p>Before we proceed we want to create an SSH key allowing us to save files and changes to those files not only in our local repository but also on remote such as GitHub.</p><h4>Setting an SSH key</h4><blockquote>⚠️<strong>Terminology Alert: </strong><em>SSH: </em>An SSH key is a pair of cryptographic keys that can be used to authenticate to an SSH server as an alternative to password-based logins. It is a part of the SSH (Secure Shell) protocol, which is used to securely connect to remote servers.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/0*xhO9WPU_rs3vat3x.png" /></figure><p>I highly recommend following the tutorial in this <a href="https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent"><strong>link</strong></a><strong> </strong>for details on setting up an SSH key for GitHub.</p><p>You can also find a brief version of the tutorial below — for the Linux OS.</p><ol><li>In the terminal execute:</li></ol><pre>ssh-keygen -t ed25519 -C &quot;your_email@example.com&quot;</pre><p>Click Enter to accept the default location where the ssh key will be saved which will be in the ~/.ssh directory. Next, you will prompted to define a local password, which will be used for authentication on your local machine.</p><p>2. Adding SSH to SSH agent</p><blockquote>Adding your SSH key to the SSH agent is like adding your house key to a keychain that you always carry with you. It makes using the key more convenient and secure. It’s a program that runs in the background and holds your private keys in memory, so you don’t have to enter the passphrase every time you use the key.</blockquote><pre>eval “$(ssh-agent -s)”<br>ssh-add ~/.ssh/id_ed25519</pre><p>3. Add SSH key to GitHub account</p><p>We first need to copy the SSH key. To find the key we first execute the command below to see the content of the SSH key.</p><pre>cat ~/.ssh/id_ed25519.pub</pre><p>4. I will assume you have a GitHub account. If not, go ahead and create one before proceeding. Once inside the <a href="https://github.com/">GitHub</a>:</p><p>Setting → Access → SSH and GCP Keys</p><p>5. Add a new key and paste the SSH key into the field.</p><h4>You are now authorized to make changes to the Remote Repositories!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/0*JnBicTge1Hd775kj.jpg" /></figure><h3>Hands-on example (Git commands) 👨‍💻</h3><p>Visual Studio allows for much easier interaction with Git thanks to its integration tools. To keep this article agnostic to the code interpreter I will use the terminal to showcase the functionality.</p><p>Let’s start off by creating a folder called <strong>test</strong>, and inside of it a python file <strong>scripy.py </strong>with the following content. (The functionality of the code doesn’t matter, we are here to learn Git not how to code).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PGFsDLF0RVgkBjEtHZCwOg.png" /></figure><ol><li>Let’s initialize a Working Directory using a terminal.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/328/1*-ttpIIrEJv5Bi2MLH9upgw.png" /></figure><pre>git init</pre><p>This will return: <em>Initialized empty Git repository in /home/user/test/.git/</em></p><blockquote>For you, the path will be different, but as long as you get the message, you are on the right track.</blockquote><blockquote>If you wish to delete the local git repo you can use: rm -ft .git command</blockquote><p>2. Now create two files: <em>README.md</em> and <em>.gitignore</em>.</p><pre>touch README.md<br>touch .gitignore</pre><p>The first one is a file, where we will describe what the code is all about and add some info for people who will view our project. The second is a file that is used to define files we do not wish to push (update) to the repository. Sometimes we have test files, or junk that we simply do not need, we can include them in this file and they will not be added.</p><p>3. At this stage we have 3 files. Let’s recall the diagram we drew earlier:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WSoFjo3u6x_18VZosvWfjQ.png" /><figcaption>Image by the Author</figcaption></figure><p>Now we wish to add all of the files in the working directory to the staging index by executing: git add . the . means that we will add all the files in the work directory. Nex to see the status of the staging index we use git status .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/1*ayP9E9wQZ892UuXhr9bAMA.png" /></figure><p>As we can see there are 3 changes marked with green to be committed. Let’s make our first commit git commit -m &quot;First Commit&quot;</p><p>The -m flag allows us to add messages. Make sure to add descriptive messages, as they will help you track changes and find a version that worked the best in case you need to revert to the previous version of the code.</p><p><strong>Let’s recap the commands:</strong></p><pre>git add .<br>git status<br>git commit -m &quot;First Commit&quot;</pre><p>Now if we check using git status</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/359/1*a9X6dU-MLZOR6VlNXtJ8qA.png" /></figure><p>Let’s modify our <strong>scripy.py</strong> file by adding an extra function</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yTOnFHxOwTZxT_5jA8eqkA.png" /></figure><p>To commit changes we can perform the exact same steps as before. If you wish you can use git add scripy.py instead of git add . but the outcome will be the same. Let’s modify the commit message to git commit -m “Second commit”`</p><h4>Recap</h4><p>So far we have created 3 files, which we focused on script.py. We first created a function <em>welcome</em> and committed it to the local repo, next, we added a <em>bye</em> function and committed it again.</p><p>Let’s check the commit history, you should see your information as well as the time and date, and the name of the commit that was executed.</p><pre>git log</pre><h3>Let’s introduce the concept of branches</h3><p>Alright! Imagine you are writing a book. You have written a few chapters, and now you want to try out different ideas for the next chapters, but you also want to keep the original chapters as they are. So, you make a copy of your book and make changes in the copy while keeping the original intact. In Git, this concept is similar to creating branches.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/861/1*MQytXrIyB87Ac0Ydbbc6fw.png" /><figcaption>Image by the Author</figcaption></figure><ul><li>Main Branch (usually master or main): This is the original book or the main line of development. It usually contains stable and tested code.</li><li>Branch: This is like a copy of your book where you can try out new ideas. It’s a divergent path where you can make changes without affecting the main branch.</li><li>Feature Branch: This is a type of branch where you develop a new feature in isolation.</li><li>Merge: Once you are happy with the changes in your branch, you can combine (or “merge”) them back into the main branch</li></ul><h3>How to work with branches?</h3><p>The workflow we will follow is: [Check branch list] -&gt; [Create dev branch] -&gt; [Make changes to the dev branch] -&gt; [Merge dev branch with the main branch]</p><p>Let’s go back to our example, and execute the following command to view the list of available Git branches:</p><pre>git branch</pre><p>The output will be *master which means you are working currently in the main aka master branch.</p><p>As mentioned before let’s imagine we want to try out new ideas, instead of messing around with the main (stable) branch, we can create a branch and experiment on it. Once we finish experimentation, and test that everything works well, we can then decide to merge/ combine the new idea with the main code.</p><ol><li>Let’s create a branch called <em>dev</em> branch</li></ol><pre>git branch dev</pre><p>We can now check available branches using git branch command as we did before</p><pre>dev<br>*master</pre><p>As you can see the * appears next to the master branch, to work on dev branch we need to switch to it.</p><p>2. Switch aka <strong><em>checkout </em></strong>to the dev branch so we can work on it:</p><pre>git checkout dev</pre><p>3. Now that we are on the dev branch we can edit code, make changes, and experiment :) for example, we can edit the python function to print <em>Bye Bye — </em>Complex! (I know)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HtOo7BZQ3Zq_zxPl.jpeg" /></figure><p>The edited function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xKEShFPb7_LEIDQpUTrHIw.png" /></figure><p>Those are all the changes we wish to make. We tested the new code and want to combine it with the main code. How can we do it?</p><h4>Combining/ Merging Dev branch with Master</h4><p>If we wish to merge <em>dev</em> branch to <em>main</em> we first need to go back to our main branch, only from main branch we can merge the dev branch.</p><pre>git checkout main<br>git merge dev</pre><p>Now that we merged the branch we can delete the Dev branch as we no longer need it:</p><pre>git branch -d dev</pre><blockquote>NOTE: Merging is where most of the issues happen. When using a code editor like Visual Studio code, it is recommended to install GitLens plugin which adds various useful functionality. In case of conflicts during merging a manual acceptance/ edit of code is needed.</blockquote><h3>Congrats!</h3><p>So far you have been able to understand how to work with the repo locally. Sooner or later you will want to work with co-workers on the project, for that purpose, we need to learn how to use<em> a remote repository.</em></p><h3>Remote repository</h3><ol><li>Go to GitHub and create a new repo</li><li>After creating repo called <em>test</em> you will see a similar page to one shown before:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DXRlc6-AuQFOmS2tPYs_-w.png" /></figure><p>3. From the page choose the SSH option and copy the link provided starting with<strong> <em>git@github.com:user/test.git</em></strong></p><pre><br># Saving the repo address<br>git remote add origin git@github.com:user/test.git<br><br># List the remote connection we have to other repos<br>git remote -v<br><br># Send the code to the &quot;origin&quot; <br>git push -u origin master</pre><blockquote>If you wish to change the already set repo address you can do so via: git remote set-url origin [new-url]</blockquote><p>Let’s break down the commands a bit:</p><ul><li>git remote add: This command is used to connect your local repository to a remote repository.</li><li>origin: This is a name that represents the URL of the remote repository. It’s a shorthand, and &quot;origin&quot; is a conventional name used to refer to the main remote repository.</li><li>git@github.com:user/test.git: This is the SSH URL of the remote repository on GitHub where &quot;user&quot; is the username and &quot;test&quot; is the repository name.</li><li>git remote -v: This command is used to list all the remote connections you have to other repositories, with “-v” standing for “verbose”, which means it will show more detailed information.</li><li>git push: This command is used to upload your local repository content to a remote repository.</li><li>-u: This option sets up an upstream tracking branch, meaning Git will remember the remote and branch you are pushing to, allowing you to use git push and git pull in the future without specifying the remote and branch.</li><li>origin: This represents the remote repository you are pushing to, which was added in the first step.</li><li>master: This is the branch you are pushing.</li></ul><p>4. Refresh GitHub and viola! You can see the files we added and modified on GitHub, as well as our commit messages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1009/1*dJZuby0AyLKidgFtdvEVTw.png" /></figure><h3>Useful tips</h3><ul><li>To un-stage changes use git reset .</li><li>Make small commits — often!</li><li>Ifyou have multiple commits that haven’t been pushed to the remote repo, before pushing using git merge &lt;branch_name&gt; — —squash this will squash all the commits together so that when displayed on GitHub they will be displayed in a more organized way.</li></ul><h3>You made it!</h3><p>If you like the style of the article and found the information useful, please consider clicking the 👏 emoji.</p><p><strong><em>Got any questions? Want to connect? — Feel free to add me on </em></strong><a href="https://www.linkedin.com/in/danielstankowski/"><strong><em>LinkedIn</em></strong></a></p><p>Good luck!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*eSyFc5ylk3ROdoTP.jpg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9493dcbd3360" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>