<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mojabi Rafi (Albraftitan) on Medium]]></title>
        <description><![CDATA[Stories by Mojabi Rafi (Albraftitan) on Medium]]></description>
        <link>https://medium.com/@mojabi.rafi?source=rss-79e312c30db8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*4nNO6py06zA_JRUb</url>
            <title>Stories by Mojabi Rafi (Albraftitan) on Medium</title>
            <link>https://medium.com/@mojabi.rafi?source=rss-79e312c30db8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 09:23:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mojabi.rafi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[ArgoCD and Vault Integration made easy [Part 2]]]></title>
            <link>https://medium.com/@mojabi.rafi/argocd-and-vault-integration-made-easy-part-2-7cbebe00412b?source=rss-79e312c30db8------2</link>
            <guid isPermaLink="false">https://medium.com/p/7cbebe00412b</guid>
            <category><![CDATA[argocd-vault]]></category>
            <category><![CDATA[argocd-secret-management]]></category>
            <category><![CDATA[kubernetes-argocd]]></category>
            <category><![CDATA[vault]]></category>
            <category><![CDATA[argo-cd]]></category>
            <dc:creator><![CDATA[Mojabi Rafi (Albraftitan)]]></dc:creator>
            <pubDate>Thu, 24 Apr 2025 14:29:59 GMT</pubDate>
            <atom:updated>2025-04-24T14:29:59.340Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kCs7zmHj6ZvkAPonVsy2NA.jpeg" /></figure><p>In previous story we managed to make ArgoCD and Vault ready for integration. having these components ready, its time deep dive in how we can make this integration and understand the authentication mechanism.</p><blockquote>Quick Recall: Remember we have ArgoCD and Vault deployed on separated Kubernetes clusters, which is very similar to the situation you can face with in production environments.</blockquote><p>Honestly, Integration between these components is NOT simple concept :) At least for me it took a bit of time to understand what exactly is going on under the hood. Good news is, I will try my best to make it as simple as possible for you.</p><blockquote>NOTE: All the files you need to replicate steps are shared in the following Git Repository<br><a href="https://github.com/rafimojabi/devopszen/tree/master/argocd-vault-integration">https://github.com/rafimojabi/devopszen/tree/master/argocd-vault-integration</a></blockquote><p><strong>Step 0: Creating a sample secret and policy in Vault</strong></p><p>Before anything else, We need to have a secret created in Vault. If you already have it, you can use it, but as we deployed a fresh instance of Vault, we need to create it manually.</p><p>To create a secret in Vault, follow the following path</p><pre>Secrets Engine &gt; Enable new Engine &gt; KV &gt; Enable Engine &gt; create secret </pre><p>Create the Secret with path “<em>development/sample-secret</em>”. Inside the path, simply create a key “<em>lab-secret</em>” with your desired value.</p><p>Having the Secret in place, its time to create a <strong>Policy </strong>that define which actions (read, write, delete) is permitted for this secret using the created policy. To create new policy</p><pre>Policies &gt; Create ACL policy</pre><p>To define the permitted actions, set the following ACL for the created policy</p><pre># Manage auth methods broadly across Vault<br>path &quot;development/*&quot;<br>{<br>  capabilities = [ &quot;read&quot;, &quot;list&quot;]<br>}</pre><blockquote><strong>Note 1</strong>: Any component that tries to access the value of a secret should use a defined policy in Vault. Vault then evaluate if the policy introduced by calling component has enough permission to do the action on the secret</blockquote><blockquote><strong>Note 2</strong>: Using <strong>*</strong> we basically allowed read capability for all secret paths created under <strong><em>development</em></strong><em>. </em>That includes our created secret path as well</blockquote><p><strong>Step 1- Setup Kubernetes Authentication Method in Vault</strong></p><p>Lets think about where requests for accessing the created service account are coming from. The answer is <strong>Kubernetes. </strong>ArgoCD is deployed in Kubernetes which means any request for accessing secrets eventually comes from a Kubernetes Cluster. So we need to add <strong><em>Kubernetes Authentication Method </em></strong>in Vault. This way, Vault knows requests are coming from a Kubernetes cluster, and based on configuration, can evaluate if the requester entity is eligible to fetch the secret!</p><p>Create a Kubernetes Authentication method</p><pre>Access &gt; Enable new method &gt; Kubernetes</pre><p>You need to set a <strong><em>Path </em></strong>first. This Path needs to be referenced later by ArgoCD AVP plugin when we configure to do so. For the time being, Set it as “<strong><em>argocd-cluster</em></strong>”</p><p>Having the path created, its time for configure the authentication method.</p><p>The first item to configure in “<strong><em>Kubernetes host</em></strong>”. In simple words, it indicated requests that are being come from a Kubernetes and indicating the created path (argocd-cluster in our case) should be verified by which Kubernetes cluster. When those requests with created path are received by Vault, Vault will verify if they are allowed to ask secrets from Kubernetes cluster we address here. You should put ArgoCD Kubernetes API address here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bozT56Rk67erweapFnAx5A.png" /><figcaption>Kubernetes Authentication configuration page - Vault</figcaption></figure><p>Next you need to define Kubernetes CA certificate of target Kubernetes cluster. You can fetch this CA in several ways. As eventually it is <strong><em>argocd-repo-server </em></strong>that asks for secrets, lets extract it from this pod itself<strong><em>.</em></strong></p><pre>kubectx  kind-lab-argocd<br>kubectl exec argocd-repo-server-xxx-yyy -- cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IY3ayo6njeGhydooirdZNA.png" /></figure><p>Copy the CA and simply paste it as text in Kubernetes CA certificate section.</p><p>You need to <strong><em>“disable use of local CA and service account JWT”. </em></strong>Why? Because the request are coming from another Kubernetes cluster (not the same one Vault resides in). So if Vault uses its local CA and JWT token, authentication will fail.</p><p>Instead, in <strong>Kubernetes options </strong>section, You need to provide JWT Token for the target Cluster. Dont rush :) We will understand soon What is JWT token reviewer and how it works soon !</p><blockquote>NOTE: If you are using the same Git repository mentioned in this post, You have already the next extra step done and you can skip it. In case you are trying to setup argocd vault integration with the components you already have, next step is necessary</blockquote><p><strong>Extra Step: </strong>Not all service accounts in Kubernetes cluster are allowed to be used for authentication purposes. There is a specific <em>clueter role </em>in Kubernetes, namely `<em>system:auth-delegator</em>`. Tokens for all service accounts bound with this clusterRole can be used to verify its Kubernetes cluster. Having ArgoCD deployed, we already have service account <strong><em>“argocd-repo-server” </em></strong>created. This is the service account being mounted in <em>argocd-repo-server-xxx </em>pod. What we need to do is to create a <em>service-account-token </em>for this service account first. Then we need to create a <em>clusterRoleBinding </em>for this service account and <em>clusteRole </em>`<em>system:auth-delegator</em>`. From then on, the created token can be used for Kubernetes cluster authentication. Applying the following file will do the job mentioned above.</p><pre>  - apiVersion: v1<br>    kind: Secret<br>    metadata:<br>      name: &quot;argocd-repo-server-secret&quot;<br>      namespace: argocd<br>      annotations:<br>        kubernetes.io/service-account.name: argocd-repo-server<br>    type: kubernetes.io/service-account-token<br><br>  - apiVersion: rbac.authorization.k8s.io/v1<br>    kind: ClusterRoleBinding<br>    metadata:<br>      name: &quot;crb-auth-delegator&quot;<br>    subjects:<br>      - kind: ServiceAccount<br>        name: argocd-repo-server<br>        namespace: argocd<br>    roleRef:<br>      apiGroup: rbac.authorization.k8s.io<br>      kind: ClusterRole<br>      name: system:auth-delegator</pre><p><strong>Token Reviewer JWT</strong>: This is the token for a service account bound with system:auth-delegator. In our case, As the AVP containers are using argocd-repo-server serviceAccount, this is the token for this service account, which can be retrieved from secret created out of it.</p><pre>kubectl -n argocd get secret argocd-repo-server-secret -o jsonpath=&quot;{.data.token}&quot; | base64 -d</pre><p>Get the output and place in the corresponding field</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-Z9b445maNa1zd3sLO6Xvw.png" /></figure><p>We are done with the configuration for now.</p><p><strong>Step 2 - Configure Role for the created Authentication Method</strong></p><p>We need to define a role for this authentication method as well. All entities authenticated with this method, will get this role. By attaching this role to the previously created policy, authenticated entities now can also access the secret we created under development path.</p><p>Create role</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n2o-4N3ES49ndDIrHqN-xA.png" /></figure><ul><li>name: your desired name. We set it as argoCD</li><li>Bound Service Account names: name of the service account being used to connect to vaul. in our case it is argocd-repo-server</li><li>bound service account namespaces: the namespace for the service account. in our case it is argocd</li><li>Generated Token’s Policies: The policy that should be used. in our case development.</li></ul><p>Save and you have the role created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/774/1*6YgO4uM0Q2LfyIZtg0MIpA.png" /></figure><p>Having all the mentioned steps completed in Vault, We can switch back ArgoCD configuration for the rest. Vault side is set!</p><p><strong>Step 3 _ Install Argocd Vaul Plugin (AVP)</strong></p><p>ArgoCs should now start talking with Vault API to fetch secrets. We use AVP plugin for this purpose. Using the configuration we set for this plugin, it knows when and how to talk with Vault and replace place holders for Secrets.</p><p>To Install AVP, we rely on ArgoCD helm chart and Side-car implementation of Argocd plugin. You need to upgrade the helm chart using provided values file.</p><pre>repoServer:<br>  serviceAccount:<br>    name: argocd-repo-server<br>  volumes:  <br>    - configMap:<br>        name: cmp-plugin<br>      name: cmp-plugin<br>    - name: custom-tools<br>      emptyDir: {}<br>  initContainers:<br>    - name: download-tools<br>      image: registry.access.redhat.com/ubi8<br>      env:<br>        - name: AVP_VERSION<br>          value: 1.16.1<br>      command: [sh, -c]<br>      args:<br>        - &gt;-<br>          curl -L https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v$(AVP_VERSION)/argocd-vault-plugin_$(AVP_VERSION)_linux_amd64 -o argocd-vault-plugin &amp;&amp;<br>          chmod +x argocd-vault-plugin &amp;&amp;<br>          mv argocd-vault-plugin /custom-tools/<br>      volumeMounts:<br>        - mountPath: /custom-tools<br>          name: custom-tools<br>  extraContainers:<br>    - name: avp<br>      command: [/var/run/argocd/argocd-cmp-server]<br>      image: registry.access.redhat.com/ubi8<br>      env:<br>        - name: ARGOCD_ENV_VAULT_ADDR<br>          value: &quot;http://192.168.178.22:8200&quot;<br>        - name: ARGOCD_ENV_AVP_AUTH_TYPE<br>          value: &quot;k8s&quot;<br>        - name: ARGOCD_ENV_AVP_K8S_MOUNT_PATH<br>          value: auth/argocd-cluster<br>        - name: ARGOCD_ENV_AVP_K8S_ROLE<br>          value: argocd<br>        - name: ARGOCD_ENV_AVP_TYPE<br>          value : vault<br>      securityContext:<br>        runAsNonRoot: true<br>        runAsUser: 999<br>      volumeMounts:<br>        - mountPath: /var/run/argocd<br>          name: var-files<br>        - mountPath: /home/argocd/cmp-server/plugins<br>          name: plugins<br>        - mountPath: /tmp<br>          name: tmp<br>        # Register plugins into sidecar<br>        - mountPath: /home/argocd/cmp-server/config/plugin.yaml<br>          subPath: avp.yaml<br>          name: cmp-plugin<br>        # Important: Mount tools into $PATH<br>        - name: custom-tools<br>          subPath: argocd-vault-plugin<br>          mountPath: /usr/local/bin/argocd-vault-plugin<br>    - name: avp-helm<br>      command: [/var/run/argocd/argocd-cmp-server]<br>      image: quay.io/argoproj/argocd:v2.7.9<br>      env:<br>        - name: ARGOCD_ENV_VAULT_ADDR<br>          value: &quot;http://192.168.178.22:8200&quot;<br>        - name: ARGOCD_ENV_AVP_AUTH_TYPE<br>          value: &quot;k8s&quot;<br>        - name: ARGOCD_ENV_AVP_K8S_MOUNT_PATH<br>          value: auth/argocd-cluster<br>        - name: ARGOCD_ENV_AVP_K8S_ROLE<br>          value: argocd<br>        - name: ARGOCD_ENV_AVP_TYPE<br>          value : vault<br>      securityContext:<br>        runAsNonRoot: true<br>        runAsUser: 999<br>      volumeMounts:<br>        - mountPath: /var/run/argocd<br>          name: var-files<br>        - mountPath: /home/argocd/cmp-server/plugins<br>          name: plugins<br>        - mountPath: /tmp<br>          name: tmp<br><br>        # Register plugins into sidecar<br>        - mountPath: /home/argocd/cmp-server/config/plugin.yaml<br>          subPath: avp-helm.yaml<br>          name: cmp-plugin<br><br>        # Important: Mount tools into $PATH<br>        - name: custom-tools<br>          subPath: argocd-vault-plugin<br>          mountPath: /usr/local/bin/argocd-vault-plugin<br><br>extraObjects:<br>  <br>  - apiVersion: v1<br>    kind: ConfigMap<br>    metadata:<br>      name: cmp-plugin<br>      namespace: argocd<br>    data:<br>      avp.yaml: |<br>        apiVersion: argoproj.io/v1alpha1<br>        kind: ConfigManagementPlugin<br>        metadata:<br>          name: argocd-vault-plugin<br>          namespace: argocd<br>        spec:<br>          allowConcurrency: true<br>          discover:<br>            find:<br>              command:<br>                - sh<br>                - &quot;-c&quot;<br>                - &quot;find . -name &#39;*.yaml&#39; ! -name &#39;*values.yaml&#39; | xargs -I {} grep \&quot;&lt;path\\|avp\\.kubernetes\\.io\&quot; {} | grep .&quot;<br>          generate:<br>            command:<br>              - sh<br>              - &quot;-c&quot;<br>              - &quot;for file in $(find . -type f -name &#39;*.yaml&#39; ! -name &#39;values.yaml&#39; ! -path &#39;*/statics/*&#39;); do argocd-vault-plugin generate --verbose-sensitive-output \&quot;$file\&quot;; done&quot;<br>          lockRepo: false<br>      avp-helm.yaml: |<br>        apiVersion: argoproj.io/v1alpha1<br>        kind: ConfigManagementPlugin<br>        metadata:<br>          name: argocd-vault-plugin-helm<br>        spec:<br>          allowConcurrency: true<br>          discover:<br>            find:<br>              command:<br>                - sh<br>                - &quot;-c&quot;<br>                - &quot;find . -name &#39;Chart.yaml&#39; | grep -qz . &amp;&amp; find . -name &#39;values.yaml&#39;&quot;<br>          generate:<br>            command:<br>              - sh<br>              - &quot;-c&quot;<br>              - |<br>                helm template $ARGOCD_APP_NAME --include-crds . |<br>                argocd-vault-plugin generate -<br>          lockRepo: false <br><br><br><br></pre><p>Update the Helm chart with the updated values</p><pre>helm upgrade argocd argo/argo-cd -n argocd -f ./argo-values.yaml</pre><blockquote>NOTES:<br>Discovery part of the plugin engages in every sync process of ArgoCD. If the status of the discovery command is success, The plugin will be invoked and generate command will be run. In our case, it is clear that discovery command is a simple find which check all the files recursively, and check if it has the pattern of avp.kubernetes.io or path|. These two are being used if any file that supposed to be rendered by values from Vault.</blockquote><blockquote>The <em>generate</em> command will get list of those files, and run argocd-vault-plugin generate --verbose-sensitive-output \&quot;$file\&quot;; per each. argocd-vault-plugin generate will use the provided information for connecting to and being authenticated against Vault as environment variables.</blockquote><p>Followings are the env avp relies on</p><pre>ARGOCD_ENV_VAULT_ADDR<br>ARGOCD_ENV_AVP_AUTH_TYPE<br>ARGOCD_ENV_AVP_K8S_MOUNT_PATH<br>ARGOCD_ENV_AVP_K8S_ROLE<br>ARGOCD_ENV_AVP_TYPE</pre><blockquote>NOTE: You remember us creating the Role for Vault authentication. The same Role will also be referenced here. This is how Vault knows which request is related to which configuration and can call back Kubernetes cluster to verify the request.</blockquote><p><strong>Step 4 - Create a secret object to be deployed by ArgoCD</strong></p><p>Now that we have all the configurations set, its time to create a secret object with placeholders. AVP plugin will detect those object and based on place holders, will fetch corresponding secrets from Vault, Render them and them deploy them on the target cluster. If you are using the Git repository I shared earlier, You can find a sample secret under the path applications/sample-secret.</p><pre>apiVersion: v1<br>kind: Secret<br>metadata:<br>  name: avp-sample-secret<br>  namespace: default<br>  annotations:<br>    avp.kubernetes.io/path: &quot;development/data/sample-secret&quot;<br>type: Opaque<br>stringData:<br>  secret-value: &lt;lab-secret&gt;</pre><p>To grasp a better idea how this secret will invoke AVP plugin and the values stored in vault will be placed as values placeholder, note the followings:</p><ul><li>The main part that causes AVP plugin being invoked is the following annotation</li></ul><pre>annotations:<br>    avp.kubernetes.io/path: &quot;development/data/sample-secret&quot;</pre><ul><li>Full illustration of how ArgoCD plugins work and how is their lifecycle can be found in the video. In a nutshell, AVP plugin has the following part as its discovery</li></ul><pre>apiVersion: v1<br>   kind: ConfigMap<br>   metadata:<br>     name: cmp-plugin<br>     namespace: argocd<br>   data:<br>     avp.yaml: |<br>       apiVersion: argoproj.io/v1alpha1<br>       kind: ConfigManagementPlugin<br>       metadata:<br>         name: argocd-vault-plugin<br>         namespace: argocd<br>       spec:<br>         allowConcurrency: true<br>         discover:<br>           find:<br>             command:<br>               - sh<br>               - &quot;-c&quot;<br>               - &quot;find . -name &#39;*.yaml&#39; ! -name &#39;*values.yaml&#39; | xargs -I {} grep \&quot;&lt;path\\|avp\\.kubernetes\\.io\&quot; {} | grep .&quot;<br>         generate:<br>         .<br>         .<br>         .</pre><ul><li>Discovery part of the plugin engages in every sync process of ArgoCD. If the status of the discovery command is success, The plugin will be invoked and the command will be run. In our case, it is clear that discovery command is a simple find which check all the files recursively, and check if it has the pattern of avp.kubernetes.io or path|. These two are being used if any file that supposed to be rendered by values from Vault.</li><li>The <em>generate</em> command will get list of those files, and run argocd-vault-plugin generate --verbose-sensitive-output \&quot;$file\&quot;; per each. argocd-vault-plugin generate will use the provided information for connecting to and being authenticated against Vault as environment variables.</li></ul><p>If all the configs are set correctly, AVP will get the values stored in Vault under the path development/data/sample-secret, check the key lab-secret and fetch the value of it, and render the secret before it is being deployed in Cluster.</p><p>At the end, if you need a comprehensive video to see all the mentioned steps in action, please refer to my youtube video.</p><p><a href="https://www.youtube.com/watch?v=G71f3Uv7htk">https://www.youtube.com/watch?v=G71f3Uv7htk</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7cbebe00412b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ArgoCD and Vault Integration made easy [Part 1]]]></title>
            <link>https://medium.com/@mojabi.rafi/argocd-and-vault-integration-made-easy-part-1-bd01fdf9755d?source=rss-79e312c30db8------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd01fdf9755d</guid>
            <category><![CDATA[vault]]></category>
            <category><![CDATA[argo-cd]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[devops-practice]]></category>
            <dc:creator><![CDATA[Mojabi Rafi (Albraftitan)]]></dc:creator>
            <pubDate>Sun, 20 Apr 2025 06:53:45 GMT</pubDate>
            <atom:updated>2025-04-27T09:01:38.442Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*ygiRF6zDmee1uUxf7sg54A.png" /></figure><p>Using ArgoCD to manage your deployment in a Kubernetes cluster sounds like a great way to go, As the only thing you need is to have deployment definitions in a Git repository and configure ArgoCD to use the repo as source of truth. They you can sit back, enjoy your coffee and see how ArgoCD does its magic!</p><p>But wait! What if it comes to create sensitive data in form of Kubernetes Secrets or ConfigMaps?! Obviously, putting them in Git is a bit …! Then How one should manage its secret objects with ArgoCD?</p><p>Here is where a proper Secret Management Tool like Vault comes in handy. You need to offload process of keeping secret secrets to Vault, and Configure ArgoCD to fetch secrets from Vault instead of reading them from raw files in Git. In these 2 part story, I will show you how you can make it from scratch. Part 1 of this story is about bringing up everything from scratch. You need to have ArgoCD and Vault and Kubernetes clusters for each before starting integration of them two.</p><blockquote>If you already have K8s cluster, ArgoCD and Vault up and running, you can safely jump to Part 2 directly</blockquote><h3>[Part 1] — Components Provisioning</h3><p>Components provisioning will be covered following these steps:<br>- Step 1 — Bringing up Kubernetes clusters<br>- Step 2 — Deploy ArgoCD<br>- Step 3 — Deploy Vault Using ArgoCD</p><p>Going through these 3 steps, we will have the required infrastructure to make ArgoCD and Vault Integration.</p><blockquote>NOTE: All materials for both parts are already provided in my github repository at <a href="https://github.com/rafimojabi/devopszen/tree/master/argocd-vault-integration"><strong>https://github.com/rafimojabi/devopszen/tree/master/argocd-vault-integration</strong></a></blockquote><p>Also in case you prefer to watch all steps in action, feel free to give the video created out of this instruction <a href="https://youtu.be/Xbuq8scbyEg">here</a></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FXbuq8scbyEg%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DXbuq8scbyEg&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FXbuq8scbyEg%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/4f6685cb1b6162ce4e0d86a3273ea780/href">https://medium.com/media/4f6685cb1b6162ce4e0d86a3273ea780/href</a></iframe><p><strong>Step 1 — Bringing up Kubernetes clusters</strong></p><p>The Architecture we follow is almost as same as the one you see in production environments. We will have 2 separate Kubernetes clusters, one for ArgoCD and one for Vault. We considered Vault as separated component deployed in Kubernetes. However, You can have it installed in other ways as well.</p><p>We use KIND tool to bring up Kubernetes clusters. This is pretty much cool tool and provides you with a Kubernetes cluster as Docker containers in your Machine. Perfect for test purposes!</p><p>In the provided git repository, we have following structure</p><pre>.<br>├── applications<br>│   ├── sample-secret<br>│   │   └── secret.yaml<br>│   └── vault<br>│       ├── values.yaml<br>│       └── vault.yaml<br>├── argocd-values.yaml<br>├── infrastructure<br>│   ├── ArgoCD-cluster-config.yaml<br>│   └── Vault-cluster-config.yaml<br>└── README.md</pre><p>Kind configuration files required for bringing up Kubernetes clusters are placed under `<em>infrastructure</em>` directory. Use the following commands to bring up clusters, one after another</p><pre>cd infrastructure<br>kind create cluster --name lab-argocd --config ./ArgoCD-cluster-config.yaml<br>kind create cluster --name lab-vault --config ./Vault-cluster-config.yaml</pre><blockquote>NOTE: Do not forget to replace your <strong>interface IP address</strong> to config files</blockquote><p><strong>Step 2 — Deploy ArgoCD</strong></p><p>Now we need to deploy ArgoCD in its cluster. We will use ArgoCD Helm chart to do so.</p><pre>kubectx kind-lab-argocd<br>helm repo add argo https://argoproj.github.io/argo-helm<br>helm repo update<br>kubectl create ns argocd<br>helm install argocd argo/argo-cd -n argocd</pre><p>We here change the context to theKubernetes cluster of ArgoCD first. We add the helm repo of ArgoCD to our helm repositories and finally deploy ArgoCD in the namespace <em>argocd.</em></p><p>For the sake of simplicity, We did not configure and install any Ingress in Kubernetes cluster. Therefore, to access ArgoCD login page, we use <em>port-forward</em> functionality of kubectl tool.</p><pre>kubectl port-forward service/argocd-server -n argocd --address 0.0.0.0 8080:443</pre><p>You can access to ArgoCD login page in port 8080 of you localhost now.</p><p>NOTE: username by default is <strong><em>admin</em></strong> and password can be found using following command</p><pre>kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath=&quot;{.data.password}&quot; | base64 -d</pre><p>Using the mentioned credentials, you now are able to login to a fresh instance of ArgoCD. It is now to be configured and be ready to deploy Vault and other deployments if needed.</p><p><strong>Step 3 — Deploy Vault Using ArgoCD</strong></p><p>As mentioned, Now that we have ArgoCD up and running, its time to configure it and make it ready for Vault deployment in other cluster. But wait, ArgoCD yet is not aware of the other Kuerbetnes cluster dedicated to Vault. So we need first introduce the other Kubernetes cluster to ArgoCD, and provide it with enough of permissions to be able to deploy things there.</p><p>We shall leverage <em>argocd-cli </em>to add the other Kubernetes cluster. As you run it in you machine, and the configuration for accessing the other cluster is already added to your <em>kubeconfig </em>file, argocd-cli know exactly what are certificates and keys to access the other Kubernetes cluster (reading them from your kubeconfig file)</p><pre>argocd login 127.0.0.1:8080<br>argocd cluster add kind-lab-vault</pre><p>Having vault cluster added to ArgoCD known cluster, it can now take care of deployments in the other cluster as well.</p><p>Next we need add this git repository to ArgoCD attached repositories. We do that via argoCD UI. <em>All applications that are supposed to be deployed by ArgoCD, including Vault, will be placed under applications directory of this repository</em></p><p>Having the repository added to the ArgoCD repositories, Create an ArgoCD application. The target should be this repository, and path should be <em>argocd-vault-integration/applications</em>. Set the type of the application to <em>directory</em> and keep <em>recursive</em> enabled.</p><p>If all the configs set correctly, you will see the vault application being created and coming up.</p><blockquote>NOTE: This is important to note vault by default is coming up sealed. Meaning the pod will not be ready until you make vault unsealed. Following is how to</blockquote><pre>ot be ready until you make vault unsealed. Following is how to<br>kubectx kind-lab-vault<br>kubens vault<br>kubectl exec -it vault-0 sh<br>vault operator init</pre><p>This will provide you with Initial root Token along with 5 unseal keys. Keep this information somewhere as it will be required later. 3 out of 5 Unseal keys should be used to unseal the vault using the command vault operator unseal &lt;KEY&gt;. Lets say the following is the output</p><pre>Unseal Key 1: oKDDHikSmx0XwX8V5zgy0CCH/Mle/gz7synSn0SJ+/tl<br>Unseal Key 2: hXxYj575680eTSas1C0TQJOwxPKTCDBxB1xfvkySv97z<br>Unseal Key 3: 8LpgJGgrvJuXWRw17S6XF0c2GywD4+cUtdQ2KdeoQQYZ<br>Unseal Key 4: VHp14753wUwsjthkF9YJR7F+Ny5besCN0YdBVHuoI8UI<br>Unseal Key 5: qg0Yg2pPD/QQq5tR8znqiLh0Axz1Ayxak/WTqiUr8fTX</pre><pre>Initial Root Token: hvs.4IhgBifbmiFifYK7HC0PmJ0d</pre><p>The following command will make vault unsealed and accessible</p><pre>vault operator unseal hXxYj575680eTSas1C0TQJOwxPKTCDBxB1xfvkySv97z<br>vault operator unseal oKDDHikSmx0XwX8V5zgy0CCH/Mle/gz7synSn0SJ+/tl<br>vault operator unseal 8LpgJGgrvJuXWRw17S6XF0c2GywD4+cUtdQ2KdeoQQYZ</pre><p>Cool! You just managed to set Vault unsealed! :) It’s time to access Vault UI as well to verify it is unsealed and you are able to configure it</p><pre>kubectl port-forward service/vault --address 0.0.0.0   8200:8200</pre><p>That’s it for the first Part. Having our Infrastructure ready, we have more to do to configure ArgoCD, AVP and Vault so that they can successfully talk to each other and fetch/deliver secrets store in Vault. Feel free to put a comment in case you face any difficulties provisioning the mentioned infra. Will see you in the <a href="https://medium.com/@mojabi.rafi/argocd-and-vault-integration-made-easy-part-2-7cbebe00412b">Part2 story</a> :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd01fdf9755d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[This is how to create CICD pipeline from Zero to Hero (part 2)]]></title>
            <link>https://medium.com/@mojabi.rafi/this-is-how-to-create-cicd-pipeline-from-zero-to-hero-dee2e0035df8?source=rss-79e312c30db8------2</link>
            <guid isPermaLink="false">https://medium.com/p/dee2e0035df8</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[cicd]]></category>
            <dc:creator><![CDATA[Mojabi Rafi (Albraftitan)]]></dc:creator>
            <pubDate>Mon, 17 Apr 2023 09:38:11 GMT</pubDate>
            <atom:updated>2023-04-17T09:42:08.761Z</atom:updated>
            <content:encoded><![CDATA[<p>In the previous article, we discussed the importance of analyzing a system, identifying its problems, and providing solutions. To address the issue of time-consuming manual application testing and deployment processes, we recommend the following solutions:</p><ol><li>Avoid manual tasks</li><li>Deploy the system in a separate and dedicated Kubernetes cluster</li><li>If using Google Cloud Platform, create a separate project for the system</li><li>Ensure the system is scalable, maintainable, and reliable</li></ol><p>To implement these solutions, we will be using the following tools:</p><ol><li>Infrastructure: Google Cloud and Terraform</li><li>CI/CD tools: Jenkins, Helm, Kustomize, ArgoCD, and GitHub</li></ol><p>Now that we know what needs to be implemented, let’s dive into the process.</p><p>Preparing the Infrastructure</p><p>This step can be broken down into two sub-sections:</p><ol><li>Setting up the required configuration in Google Cloud by manually creating a new project</li><li>Developing IaC (Infrastructure as Code) with Terraform, which will connect to the created project and set up the necessary infrastructure, including VPC, networks, sub-networks, Kubernetes, etc.</li></ol><p><strong>Creating a New Project in Google Cloud:</strong></p><p>Login to your google cloud account, click on list of projects and click on “New Project” link. Set your desired project name. In the course of this toturial, we use “devops-tools”.</p><p>Note: if you are working with a business account, dont forget to select the organization and folders as well. In personal account no organization can be defined.</p><p>Having your project created, copy your project id as we need it to configure Terraform codes and lets dive into terraform codes.</p><p><strong>Develope required IaC code</strong></p><p>Please find all the codes for this series of toturials at this <a href="https://github.com/rafimojabi/DailyDevOpsDelimmas">Github Repository</a>.</p><p>We are creating new infrastructure. It contains both IaC and Deployments. We create a level of isolation between these two parts using the following directory structure</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/895/1*zJh2Ha20kAG4EFQz4zG1-Q.png" /><figcaption>Directory Structure For Our Git Repository</figcaption></figure><p>Full description of each terraform part can be found in my corresponding <a href="https://www.youtube.com/watch?v=2MI7ivRjE2s">video here</a> at 11:47. Therefore I provide only a recap of it here</p><ul><li>provider.tf: We need to address terraform google provider here. We also need to reference to the region and projectID of created project in previous step. The more important part is to set terraform backend, which should be used to keep terraform state file. This is a best practic to keep this file somewhere remote, safe and accessible for terraform and we use “bucket” to keep this file.</li><li>vpc.tf: We setup new VPC that should be used for the components working in the new project.</li><li>subnets.tf: We define our desired subnet for the created VPC here. We define out main cider range, and then a secondary cidr IP range that should be used in Kubernetes cluster</li><li>router.tf: We create and attach a router to our created network</li><li>nat.tf: We need to provide internet access for the pods available in kubernetes. For that purpose, we create a Nat and Router_nat and attach it to our network.</li><li>Kubernetes.tf: Configuration for our Kubernetes cluster including its network, region, logging features, number of nodes etc. should be defined here</li><li>node-pools.tf: We need to create new nodepool with our desired configuration including number of nodes, machine type, ouath scope etc. Here. This nodepool will later attached to our created kubernetes cluster.</li></ul><p>We then initiate terraform using</p><pre>terraform init</pre><p>Yet before apply changes, terraform should somehow be authenticated to access google cloud resources. We do this using following coomand</p><pre>gcloud auth login</pre><p>By Granting the permission, required env var for terraform will be set and we cat deploy our infrastructure using</p><pre>terraform apply</pre><p>It take around 20 minutes to create our infrastructure as we defined in our terraform codes.</p><p>Having The infrastructure up, we are ready to proceed to next step which is configure Jenkins and the rest on it. We will continue with in in the next tutorials. If you found it useful, you can access to the rest of materials before I posting it here using following series</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F2MI7ivRjE2s%3Flist%3DPLHrDWXA1Cuzc9J7wScMIC_5-hWHhf95Bd&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D2MI7ivRjE2s&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F2MI7ivRjE2s%2Fhqdefault.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/31999548bd6e53b292eda87eaae0df8c/href">https://medium.com/media/31999548bd6e53b292eda87eaae0df8c/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FfkaKhFUaL8M&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DfkaKhFUaL8M&amp;image=http%3A%2F%2Fi.ytimg.com%2Fvi%2FfkaKhFUaL8M%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/fac7cf1e46f1a3a219e8d9c23d484089/href">https://medium.com/media/fac7cf1e46f1a3a219e8d9c23d484089/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FTL_8VYEdDUE%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DTL_8VYEdDUE&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FTL_8VYEdDUE%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/8c95ce4d0edd4b83622b3a746349cba7/href">https://medium.com/media/8c95ce4d0edd4b83622b3a746349cba7/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fq8XEWsLrj10%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dq8XEWsLrj10&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fq8XEWsLrj10%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a40d80f978e1469a15b64a5a533ee7a7/href">https://medium.com/media/a40d80f978e1469a15b64a5a533ee7a7/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dee2e0035df8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[This is how you create CICD pipeline, from Zero to Hero (part1)]]></title>
            <link>https://medium.com/@mojabi.rafi/this-is-how-you-create-cicd-pipeline-from-zero-to-hero-part1-a52de4018bd0?source=rss-79e312c30db8------2</link>
            <guid isPermaLink="false">https://medium.com/p/a52de4018bd0</guid>
            <category><![CDATA[cicd]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Mojabi Rafi (Albraftitan)]]></dc:creator>
            <pubDate>Sun, 16 Apr 2023 22:11:30 GMT</pubDate>
            <atom:updated>2023-04-17T10:11:41.392Z</atom:updated>
            <content:encoded><![CDATA[<h3>This is how you create CICD pipeline, from Zeroto Hero (Intro)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aLfrL59TgKdMbCExNaFBlQ.jpeg" /></figure><p>As the DevOps engineer responsible for implementing processes related to DevOps in a startup company, the first step is to get familiar with the current system and identify opportunities for improvement or replacement. In the given scenario, the current system is a manual Legacy system where developers send their codes to a repository and then build the codes, create an image, and run a Docker container to test the application. This process is all manual, and there is a need to automate everything as much as possible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RaDPKI5RJsT8m2tGTDcI8w.png" /></figure><p>The suggested solution is to replace the Legacy system with a CI/CD pipeline triggered by events in the git repository. Developers will put their codes in the git repository, and any event, such as a push or merge, will trigger the pipeline. The pipeline will automate the manual jobs in the Legacy system, including building the code, creating the image, and running the container. Developers will no longer be responsible for these tasks, and they will have more time to focus on developing and improving code quality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WMpqW5HqutwrgL76fdJpcA.png" /></figure><p>The next step is to choose the infrastructure to install the CI/CD pipeline. Since there will be more than one CI/CD pipeline, and each developer may be working on different microservices, it is essential to have the ability to start building everything simultaneously without waiting for each other. The choice of infrastructure plays a critical role in the stability and scalability of the CI/CD system. The recommended solution is to set up the CI/CD pipeline inside a Kubernetes cluster. Kubernetes provides more flexibility related to scalability and can scale up or down the runners, providing cost savings.</p><p>Finally, the implementation of the infrastructure should be done using infrastructure as code. The recommended option is to use Google Cloud and infrastructure as code to ensure that the implementation is reproducible, scalable, and easy to maintain. Using infrastructure as code also provides version control, so changes made to the infrastructure can be tracked and rolled back if necessary.</p><p>Links to next parts:</p><ul><li><strong>This is how you create CICD pipeline, from Zeroto Hero (Part2): </strong><a href="https://medium.com/@mojabi.rafi/this-is-how-to-create-cicd-pipeline-from-zero-to-hero-dee2e0035df8">https://medium.com/@mojabi.rafi/this-is-how-to-create-cicd-pipeline-from-zero-to-hero-dee2e0035df8</a></li></ul><p>We will Cover each step in the following articles. Each article in this serries is a recap of its corresponding Youtube video in This Playlist</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F2MI7ivRjE2s%3Flist%3DPLHrDWXA1Cuzc9J7wScMIC_5-hWHhf95Bd&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D2MI7ivRjE2s&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F2MI7ivRjE2s%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/72a837dbe7225ab500509ccf52286dd4/href">https://medium.com/media/72a837dbe7225ab500509ccf52286dd4/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a52de4018bd0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setup Kubernetes Cluster on Virtual Box an automated way]]></title>
            <link>https://medium.com/@mojabi.rafi/setup-kubernetes-cluster-on-virtual-box-an-automated-way-7a62138ed36c?source=rss-79e312c30db8------2</link>
            <guid isPermaLink="false">https://medium.com/p/7a62138ed36c</guid>
            <category><![CDATA[kubernetes-cluster]]></category>
            <category><![CDATA[vagrant]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[virtualbox]]></category>
            <dc:creator><![CDATA[Mojabi Rafi (Albraftitan)]]></dc:creator>
            <pubDate>Thu, 15 Sep 2022 18:17:13 GMT</pubDate>
            <atom:updated>2022-09-28T17:14:37.978Z</atom:updated>
            <content:encoded><![CDATA[<p>If you have read the <a href="https://medium.com/@mojabi.rafi/create-a-kubernetes-cluster-using-virtualbox-and-without-vagrant-90a14d791617">previous tutorial</a> in this series, you already know how to setup a stable and fully customizable kubernetes cluster in your local machine. In this tutorial, We are making life much easier to setup the same cluster on virtual box in an automated way.</p><p>In order to make the provisioning process automated, we leverage <a href="https://www.vagrantup.com/">Vagrant</a> tool.</p><p><strong>Installing Vagrant</strong></p><p>I am using an Ubuntu machine and will describe how to install vagrant based on my machine. If you are using different machine (even windows), simply follow installation process described by Vagran on <a href="https://www.vagrantup.com/downloads">this page</a>.</p><p>On Ubuntu machine, simply do</p><pre>wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor | sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg</pre><pre>echo &quot;deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main&quot; | sudo tee /etc/apt/sources.list.d/hashicorp.list</pre><pre>sudo apt update &amp;&amp; sudo apt install vagrant</pre><p>You can simply check if vagrant is installed by running vagrant -v .</p><p><strong>Setup Kubernetes cluster</strong></p><p>I provided the required codes on my <a href="https://github.com/rafimojabi/DevOps-Play-Ground">DevOps-Play-Ground</a> github. simply clone the repository and run vagrant up command.</p><pre>git clone <a href="https://github.com/rafimojabi/DevOps-Play-Ground.git">https://github.com/rafimojabi/DevOps-Play-Ground.git</a><br>cd DevOps-Play-Ground/kubernetes/provision<br>vagrant up</pre><p>This command just fires the kubernetes cluster provisioning procedure, including Nodes creation, required package installation, network configuration etc. Based on you connection speed, it takes a while to download all required packages and control plane containers. When it ends, simply copy the output kubectl config and put it into you machine kubectl config file located in ~/.kube/config . You can find the mentioned output between</p><pre>---------COPY AND PASTE THE FOLLOWING IN ~/.kube/config OG MACHINE YOU WANT TO RUN KUBECTL COMMANDS-------</pre><pre>---------------------</pre><p>Having the config file copied on ~/.kube/config , you can simply run kubectl get node which shows the nodes and their status in your kubernetes cluster. If you want to see how this vagrant file works, you can continue reading this post I will explain everything. If not, Just Enjoy :) you have your cluster up and running.</p><p><strong>Note</strong>: <strong>According to previous article, we used HostOnly-adapter </strong><strong>vboxnet4 with IP range of </strong><strong>192.168.60.0/24 . In case you did not created it, simply change network to </strong><strong>vboxnet1 and IPs in Vagrant file and bootstrap.sh correspondingly.</strong></p><p>We will use this manner of creating Kubernetes cluster for the rest of tutorials.</p><p>Please do Follow to catch next tutorials, and give me a clap to encourage me :).</p><p><strong>Explaining How it works</strong></p><p>So what is Vagrant? Vagrant is one of the tools developed by <a href="https://www.hashicorp.com/">Hasicorp</a> to create development environment/infrastructure with easy workflow regardless of your role as a developer, operator, or designer. It leverages a declarative configuration file which describes all your software requirements, packages, operating system configuration, users, and more.</p><p><strong>Why Vagrant</strong></p><p>Vagrant has fully integrated with virtualbox and we are provisioning out infrastructure using virtualbox. Also we simply don’t have any better alternative like Terraform for this purpose (Terraform module for virtualbox modules is not rich enough and we cant do the things we can with Vagrant).</p><p><strong>Vagrant File</strong></p><p>Following is the content of our Vagrant file. Lets describe it</p><pre>ENV[&#39;VAGRANT_NO_PARALLEL&#39;] = &#39;yes&#39;</pre><pre>Vagrant.configure(2) do |config|</pre><pre>config.vm.define &quot;master01&quot; do |node|</pre><pre>node.vm.box               = &quot;generic/ubuntu1804&quot;</pre><pre>node.vm.box_check_update  = false</pre><pre>node.vm.hostname          = &quot;master01.kubernetes.cluster&quot;</pre><pre>node.vm.network &quot;private_network&quot;, ip: &quot;192.168.60.11&quot;, name: &quot;vboxnet4&quot;</pre><pre>node.vm.provider :virtualbox do |v|</pre><pre>v.name    = &quot;master01&quot;</pre><pre>v.memory  = 2048</pre><pre>v.cpus    = 2</pre><pre>end</pre><pre>node.vm.provider :libvirt do |v|</pre><pre>v.memory  = 2048</pre><pre>v.nested  = true</pre><pre>v.cpus    = 2</pre><pre>end</pre><pre>node.vm.provision &quot;shell&quot;, path: &quot;bootstrap.sh&quot;, args: &quot;master&quot;</pre><pre>end</pre><pre>(1..2).each do |i|</pre><pre>config.vm.define &quot;worker0#{i}&quot; do |node|</pre><pre>node.vm.box               = &quot;generic/ubuntu1804&quot;</pre><pre>node.vm.box_check_update  = false</pre><pre>node.vm.hostname          = &quot;worker0#{i}.kubernetes.cluster&quot;</pre><pre>node.vm.network &quot;private_network&quot;, ip: &quot;192.168.60.2#{i}&quot;</pre><pre>node.vm.provider :virtualbox do |v|</pre><pre>v.name    = &quot;worker0#{i}&quot;</pre><pre>v.memory  = 1024</pre><pre>v.cpus    = 1</pre><pre>end</pre><pre>node.vm.provider :libvirt do |v|</pre><pre>v.memory  = 1024</pre><pre>v.nested  = true</pre><pre>v.cpus    = 1</pre><pre>end</pre><pre>node.vm.provision &quot;shell&quot;, path: &quot;bootstrap.sh&quot;, args: &quot;worker&quot;</pre><pre>end</pre><pre>end</pre><pre>end</pre><p>We instructed Vagrant not to provision nodes in paralel. The reason is we need to first initialize master node so that other nodes (worker) can be join the cluster. As it can be seen, We used generic/ubuntu1804 Vagrant box. on vagrant up , Vagrant will download this ready to use box among <a href="https://app.vagrantup.com/boxes/search">vagrant boxes</a>. If you compare the file with our <a href="https://medium.com/@mojabi.rafi/create-a-kubernetes-cluster-using-virtualbox-and-without-vagrant-90a14d791617">previous tutorial</a>, The infra structure is just as what we described there. The vagrant file is quite easy to follow. The only trick is the bootstrap.sh file which is responsible to setup master and worker node, and create the cluster. I Will put its description in in continue.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a62138ed36c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Create a Kubernetes Cluster using Virtualbox — The Hard Way]]></title>
            <link>https://medium.com/@mojabi.rafi/create-a-kubernetes-cluster-using-virtualbox-and-without-vagrant-90a14d791617?source=rss-79e312c30db8------2</link>
            <guid isPermaLink="false">https://medium.com/p/90a14d791617</guid>
            <category><![CDATA[kubeadm]]></category>
            <category><![CDATA[kubernetes-cluster]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[virtualbox]]></category>
            <dc:creator><![CDATA[Mojabi Rafi (Albraftitan)]]></dc:creator>
            <pubDate>Sun, 11 Sep 2022 15:31:41 GMT</pubDate>
            <atom:updated>2022-09-25T09:37:08.263Z</atom:updated>
            <content:encoded><![CDATA[<p>Hi everyone! Have you ever thought about having a production like Kubernetes cluster privately so that you can play with Kubernetes with no fear and run various tests at ease?</p><p>To achieve this goal, we can go for following approaches:</p><ol><li>Run Kubernetes cluster on local machine using mini-kube</li><li>Run Kubernetes cluster on one of the cloud service like GoogleCloud, AWS etc.</li><li>Run Kubernetes cluster on your local machine using Virtualbox and virtual machines.</li></ol><p>The first solution has its own limitations. The major limitation of Minikube is that <strong>the local cluster has only one node</strong>, and therefore is not a great simulation of a production-level, multi-node Kubernetes cluster.</p><p>The second solution works, yet it is costly. As Learning Kubernetes and performing experiments on it is a time consuming process, non of the cloud services free trial would be enough and at the end it gonna charge us:)</p><p>The third solution is more complex that other two. Yet it provides us with significant flexibility. It is very like production level kubernetes clusters. We are able to add nodes, drain nodes, join master nodes, work with master’s etcd and use/experiment our desired Kubernetes network plugins. The other important benefit of setting up Kubernetes cluster with this approach is that as we have to setup everything from the scratch, we understand what is happening under-hood in a better way. Lets go for it then!</p><h3><strong>Setting up Kubernetes cluster in action</strong></h3><p>In the nutshell, we are going to implement the following architecture for setting up our Kubernetes cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*7udyNeQkQQQpsbJuYiV8Sg.png" /><figcaption>Figure 1-Cluster architecture</figcaption></figure><p>There are some key-points apparent in the figure 1.</p><ul><li>We start off by 1 master node</li><li>we have 3 worker nodes</li><li>Each VM has 2 network interfaces.</li><li>One network interface is attached to default NAT network in virtualbox. the purpose of this network is to provide the node internet access.</li><li>Another network interface is attached to a Host-Only network in virtualbox. All inside cluster communications would be through this network.</li></ul><p><strong>Prepare VirtualBox machines for deploying Kubernetes cluster</strong></p><p>According to figure-1, We need a private network that nodes can communicate with each other though. To create one go to File -&gt; Host Network Manager. Create new network there with no DHCP Enable option. Remember the network range created! we gonna use it soon. Mine is <em>192.168.60.1/24 .</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*9ZnFgOdCt8SkqWtXZffr-g.png" /></figure><p><strong>VM’s OS and configuration:</strong></p><p>I assume you are familiar with creating new virtual machine in virtual box. if not, no worries! just Do google it :). We gonna use following setup as of our VMs:<br>Master(s):<br> — Name: master# (number of master in the cluster)<br>— OS: ubuntu-server-18.04<br> — CPU: 2core<br> — Memory: 2GB<br>Workers:<br> — OS: ubuntu-server-18.04<br> — CPU: 1core<br> — Memory: 1GB</p><p>Please note each of the VMs should benefit from 2 networks. One default NAT network which provides internet access to our node, Another attached to Host Only Network created in previous step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*3Rjm2DAYcdNTjWiOP-2TTA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/1*Ptr-yb33HBmzJaB5I-KXbA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B9RjJbRyGzNBI_beiLumjw.gif" /></figure><p>After setting up the VM virtual hardware, it is time to install our operating system. There is no trick on it, except we have to assign our desired IP address in the range provided by Host Only adapter manually. In my case, I will assign <em>192.168.60.11 </em>to my first Master node. We will extend this lab in the next tutorials so lets reserve some IPs from <em>192.168.60.11</em> to <em>192.168.60.20. </em>Consequently, worker01 node will be 192.168.60.21 and so on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mhnGMc44Hnes37iWepXBEQ.gif" /></figure><p>Note: You should not set gateway 192.168.60.1 for any Node, Or you will have to change the default route (The route we need for Internet connection) to the NAT interface after OS install.</p><p>Note: Among All packages the installer ask you about, only select OpenSSH package to be installed.</p><p>Having OS installed and VM up and running, you should be able to ping the IP address of the VM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S43p0nbfq7Ji06g_YCwfaQ.gif" /></figure><p>We repeat the same steps for the workers too. At the end, the final node should represent the following table</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/304/1*nQ3gRRVKfbGV04iz-n-0fg.png" /></figure><p>After finishing the setup process, you should be able to ping all the targets from your host machine and from inside each of the nodes.</p><p><strong>Setup Kubernetes cluster using kubeadm tool</strong></p><p>So far we prepared the required infrastructure to deploy our Kubernetes cluster. Lets begin setting up the cluster :). There are several ways to setup Kubernetes cluster. We are going to leverage `<em>kubeadm` </em>tool. kubeadm enable us to create Kubernetes cluster that will pass <a href="https://kubernetes.io/blog/2017/10/software-conformance-certification/">Kubernetes Conformance tests</a>. kubeadm also support providing tokens, cluster upgrade etc.</p><p>Install required components</p><p>Each of the nodes need the following components available:</p><ul><li>kubelet: The kubelet is the primary “node agent” that runs on each node. It can register the node with the apiserver using one of: the hostname; a flag to override the hostname; or specific logic for a cloud provider.</li><li>kubectl: The Kubernetes command-line tool, <a href="https://kubernetes.io/docs/reference/kubectl/kubectl/">kubectl</a>, allows you to run commands against Kubernetes clusters. You can use kubectl to deploy applications, inspect and manage cluster resources, and view logs.</li><li>kubeadm: Using kubeadm, you can create a minimum viable Kubernetes cluster that conforms to best practices.</li></ul><p><strong>Step 1- Installing required components</strong></p><p>On each nodes, add the kubernetes repository</p><pre><br>sudo apt update &amp;&amp; sudo apt install -y curl<br>curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add</pre><pre>sudo apt-add-repository &quot;deb http://apt.kubernetes.io/ kubernetes-xenial main&quot;</pre><p>Having required repository added, install required components. At the time, the latest version of kubernets is 1.25.0. Yet as for more stability, we are going to use kubernetes version 1.22.10 for our cluster.<br>First, lets list what versions are available:</p><pre>apt list -a kubeadm</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*66wf57gAOa0825lgMRI70w.png" /></figure><pre><br>sudo apt install -y kubeadm=1.22.10-00 kubelet=1.22.10-00 kubectl=1.22.10-00</pre><p>Note: In case you copy the command above, please note the “-” inside the code. It can be copied as “ — ” which fail the command above.</p><p>It is recommended to hold the version on installed packages. We really don’t want to mess up with kubernetes auto updates :)</p><pre>sudo apt-mark hold kubeadm kubelet kubectl</pre><p>Check the version of installed components and it should be as version we just installed</p><p>kubead version</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1ZNDOOwVAcI64vXF9uX8aQ.png" /></figure><p>Next step is to install Docker on the nodes. The official document on installing docker on Ubunut can be found <a href="https://docs.docker.com/engine/install/ubuntu/">here</a>. Yer for the sake of simplicity, we install it using a simple command as follow:</p><pre>sudo apt install -y docker.io</pre><p>You can verify installation using</p><blockquote>docker version</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c0H-L7BcY68KdJzfBNgxWg.png" /></figure><p>Docker will not start on system startup. meanwhile it is a requisite for kubernetes cluster to be functional. Do Not forget to enable its service to make sure docker will get up and work on each node restart</p><pre>sudo systemctl enable docker</pre><p><strong>Changing Docker Cgroup Driver</strong></p><p>It is crucial to change docker cgroup Driver after install or you get error regarding “Docker group driver detected cgroupfs instead systemd” on cluster initialization. It is easy, and should be done on all nodes (master and workers).</p><pre>sudo cat &lt;&lt;EOF | sudo tee /etc/docker/daemon.json<br>{ &quot;exec-opts&quot;: [&quot;native.cgroupdriver=systemd&quot;],<br>&quot;log-driver&quot;: &quot;json-file&quot;,<br>&quot;log-opts&quot;:<br>{ &quot;max-size&quot;: &quot;100m&quot; },<br>&quot;storage-driver&quot;: &quot;overlay2&quot;<br>}<br>EOF</pre><p>Restart Docker and make sure it is up and running</p><blockquote>sudo systemctl restart docker</blockquote><p>You can check if the Cgroup driver is changes to systemd</p><blockquote>sudo docker info | grep -i cgroup</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/671/1*x_ysV8Ga9Ka6xt9jRO5EtA.gif" /></figure><p><strong>Disable swap and enable ip_forward</strong></p><p>Kubernetes cluster needs the to disabled swap for the sake of security issues and performance. To disable swap permanently on all node, run the following commands:</p><blockquote>sudo swapoff -a</blockquote><p>Now to disable swap permanently, open <em>/etc/fstab</em> and comment line containing swap by adding # in the beginning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/671/1*Lf7u9f9pNNwp0UCjOdNp6g.gif" /></figure><p>Now we need to enable IP_Forward in kernel</p><pre>sudo modprobe overlay<br>sudo modprobe br_netfilter<br>sudo tee /etc/sysctl.d/kubernetes.conf&lt;&lt;EOF<br>net.bridge.bridge-nf-call-ip6tables = 1<br>net.bridge.bridge-nf-call-iptables = 1<br>net.ipv4.ip_forward = 1<br>EOF<br>sudo sysctl --system</pre><p><strong>Step 2- Initializing Kubernetes Master Node</strong></p><p>Having are required components installed and configured, We shall initialize the master node. Please note that in our lab and for now, our control plane consists of only one node (master01). All the required components for <a href="https://kubernetes.io/docs/concepts/overview/components/">control plane</a> including etcd will be installed on master node as a docker container.</p><p>Please note that from the first place, the secondary private network was added for internal kubernetes components communications. Therefore we shall bound the <em>api-adrvertisement-address</em> to this network only.</p><pre>sudo kubeadm init --apiserver-advertise-address 192.168.60.11 --control-plane-endpoint 192.168.60.11</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GDIFEb-y4DxgFBC5WYY7QQ.gif" /></figure><p>it takes some time to download all the required docker images and initialize it depends on you Internet connection speed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ABPPGEtQ9NDZVZphTUtuwA.gif" /></figure><p>When the master node initialization process done, the commands to run on workers and probable other masters to join the cluster will be provided by kubeadm</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ith0CDN6-I3tSAulvrwnjA.png" /></figure><p>To connect to our cluster using kubectl, we need to apply following changes on the remote machine.</p><blockquote>mkdir -p $HOME/.kube<br> sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config<br> sudo chown $(id -u):$(id -g) $HOME/.kube/config</blockquote><p>lets setup kubectl installed on master machine and then run <em>kubectl get node command to see the output.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wzljYSsGReXabiNkrcK11A.gif" /></figure><p>As you can see the master node is already added to the cluster.</p><p>Copy and paste the command to join workers to the cluster somewhere. We gonna need it to join workers to our clusters</p><p>Note: All the previous steps should be applied to workers as well, except that workers dont need kubectl at all.</p><p>Having all the above steps done for workers, use the command you copied and join the workers to the cluster. After joining the workers to the cluster, the output of <em>kubectl get node </em>should show new joined nodes as worker.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/1*82P6lGX2BLVC-1ZHqJeTCQ.png" /></figure><p><strong>Step 3- Installing Kubernetes Network</strong></p><p>You probably noticed in kubectl get node command output that all the nodes are in “NotReady” state. The reason is we still did not create any Kubernetes network for the cluster. This network is different from our infrastructure network, and is responsible to create an internal network for all inside-cluster components so that they can communicate with each other through it.</p><p>We are going to use <a href="https://github.com/flannel-io/flannel">Flannel</a>, one of most straight forward networks for kubernetes. There are <a href="https://kubernetes.io/docs/concepts/cluster-administration/addons/">other network-add on</a> with more feature and capabilities. Yet Flannel serve us best and also dosent add up any unnecessary complexity.</p><p>To install Flannel network add-on:</p><ol><li>kubectl apply -f <a href="https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml">https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml</a></li></ol><p>It takes some time to download all required docker images and bring the containers up.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OfHckbX_0Av-U8E1RAvIKA.gif" /></figure><p>Congratulations! The Kubernetes cluster is up and ready to use.</p><p>Final Words: <br>I started to write about what I get during last 3 years working in DevOps in a series. We will go through most of experiences I get by doing. For that purpose, we need this cluster. I started to explain it in hard-way, so we can get what exactly is happening under hood. Because performing all the steps every time manually is a time consuming process, I provided <a href="https://medium.com/@mojabi.rafi/setup-kubernetes-cluster-on-virtual-box-an-automated-way-7a62138ed36c">an automated way to deploy the exact same cluster</a>, which will take few minutes based on your Internet connection speed. We will use this automated way to do our experiments on Kubernetes until we start working with cloud based Kubernetes engines such as GKE.<br>Please provide me feed backs and ask questions to encourage me.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=90a14d791617" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>