<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by The Platform Enthusiast on Medium]]></title>
        <description><![CDATA[Stories by The Platform Enthusiast on Medium]]></description>
        <link>https://medium.com/@PlatformEnthusiast?source=rss-b300b45c8552------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*KrEZY_J6WX01zBTy_xxFTA.png</url>
            <title>Stories by The Platform Enthusiast on Medium</title>
            <link>https://medium.com/@PlatformEnthusiast?source=rss-b300b45c8552------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 21:10:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@PlatformEnthusiast/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Secret Layer: How OpenBao Protects Your Kubernetes Cluster]]></title>
            <link>https://medium.com/@PlatformEnthusiast/the-secret-layer-how-openbao-protects-your-kubernetes-cluster-752549c1c620?source=rss-b300b45c8552------2</link>
            <guid isPermaLink="false">https://medium.com/p/752549c1c620</guid>
            <category><![CDATA[linux-foundation]]></category>
            <category><![CDATA[openbao]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[The Platform Enthusiast]]></dc:creator>
            <pubDate>Fri, 29 May 2026 14:37:04 GMT</pubDate>
            <atom:updated>2026-05-29T16:59:26.871Z</atom:updated>
            <content:encoded><![CDATA[<p>You’re running a few services on Kubernetes. Nothing fancy: a database, an API, maybe a worker that pulls images from a container registry. Each one needs credentials. Database password. API key. Registry access token. So you do what everyone does: you put them in environment variables. A ConfigMap for the non-sensitive stuff. Maybe a Kubernetes Secret for the rest. <em>kubectl apply -f deployment.yaml</em>. Done.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/0*qtfYreR5tJRbUpZh.png" /><figcaption>OpenBao high level flow</figcaption></figure><p>Then one day, you run <em>kubectl get secrets</em> and realize something uncomfortable: every secret in your cluster is sitting in etcd as Base64. Not encrypted. Not hashed. Base64 is encoding, not encryption. Anyone with cluster read access can decode every password, every key, every token your applications use. It takes one command and zero expertise.</p><p>You check for audit logs. There are none. Kubernetes doesn’t log who reads which secret, or when. You check the rotation. When was the last time the database password changed? Six months ago. It’s the same password someone used during initial setup.</p><p>You look at the deployment YAML again. The registry pull credentials are in <em>imagePullSecrets</em>. The database password is in <em>env</em>. The API key is in a ConfigMap because someone needed it during development and it never moved. This is not a rare setup. This is the default. The cluster is healthy. Every pod is running. And the secrets are completely exposed.</p><p>OpenBao is the answer to this specific problem. It replaces Base64-in-etcd with AES-256-GCM encrypted storage. It replaces “everyone with cluster access can read everything” with per-ServiceAccount, per-path policies. It replaces “rotate manually and hope you remember” with dynamic secrets that expire on their own. And it replaces “no audit trail” with a log entry for every single secret access: who requested it, what they requested, and when.</p><p>This article walks through exactly how to set that up on Kubernetes: deploy OpenBao, configure Kubernetes authentication, inject secrets into pods, and verify the full flow end to end. Every command is explained. Every failure mode is called out.</p><h4>What OpenBao actually is</h4><p>You’ve probably heard “OpenBao is a secrets manager.” That’s technically correct and practically useless. It doesn’t tell you what problem it solves or why you’d pick it over the alternatives.</p><p>Here’s the short version: OpenBao solves two problems. First, it stores secrets encrypted at rest, so the database password sitting on disk isn’t readable by anyone who happens to have filesystem access. Second, it controls who gets to read those secrets, using policies tied to specific identities. In Kubernetes, that means specific ServiceAccounts.</p><p>OpenBao is a fork of HashiCorp Vault. When HashiCorp moved from MPL 2.0 to the Business Source License in 2023, the community forked the last MPL-licensed version and brought it under the Linux Foundation. The result is OpenBao: same codebase, same API, same CLI patterns, different license. MPL 2.0 means no BSL restrictions, no enterprise license required, no feature gating.</p><p>The practical implication is that OpenBao maintains roughly 95% API compatibility with Vault. Most Vault CLI commands work by replacing <em>vault</em> with openbao. Most Vault client libraries work with OpenBao by changing the endpoint URL. If you’ve used Vault before, you already know how to use OpenBao.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qZfZYkziqpnyoAfzsZEgzw.png" /><figcaption>OpenBao UI</figcaption></figure><h4>The problems it solves</h4><p>Before diving into setup, it’s worth understanding what you’re gaining. Here’s what changes when you introduce OpenBao into a Kubernetes cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ny72QnGmwsgFjpgTmDuytA.png" /><figcaption>OpenBao vs without OpenBao Table Comparison</figcaption></figure><p>The summary: OpenBao turns secret management from “everyone’s problem” into a platform capability, with encryption, audit, rotation, and access control built in.</p><h4>Deployment modes: Dev, Standalone, HA</h4><p>OpenBao supports three deployment modes, and choosing the right one matters more than you’d think.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2ge8UeDQp9EQ0LiOdEZQ2g.png" /><figcaption>OpenBao Deployment Modes</figcaption></figure><p><strong>Dev mode</strong> is for learning and quick testing. Data lives in memory and is lost on restart. The server auto-initializes and auto-unseals, so there’s no setup friction. The root token is <em>root</em>. Memory footprint is around 50MB. Startup takes about ten seconds. Use this when you want to try OpenBao for the first time or test a configuration change. Do not use this in production. The root token gives full admin access, and data disappears on restart.</p><p><strong>Standalone mode</strong> is for testing with persistence. A single pod with a PersistentVolumeClaim. Data survives restarts. Requires manual initialization and unsealing. Memory footprint is around 150MB. Startup takes about twenty seconds, plus the time to unseal. Use this when you want to test backup and restore procedures, or when you need persistence but don’t need high availability.</p><p><strong>HA Raft mode</strong> is for production. Three pods are working together with Raft consensus. If one pod fails, the other two keep running. Automatic leader election. Memory footprint is around 450MB total (three times 150MB). Startup takes about thirty seconds plus unseal time for each replica. Use this when you need high availability and can tolerate the operational overhead of managing three replicas.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zEcdDt0UBzq1I0P8BBVqZQ.png" /><figcaption>Deployment Method Comparison</figcaption></figure><p>For the rest of this article, we’ll use dev mode for the initial walkthrough and call out where standalone or HA differ.</p><h4>Installation: the commands and what they mean</h4><p>OpenBao runs inside Kubernetes as a Helm chart. The installation is straightforward, but there are a few gotchas that will cost you time if you don’t know about them in advance.</p><p>First, add the Helm repository:</p><pre>helm repo add openbao https://openbao.github.io/openbao-helm<br>helm repo update</pre><p>For dev mode, install with the dev values file:</p><pre>helm upgrade --install openbao openbao/openbao \<br>  --namespace openbao --create-namespace \<br>  --values values-dev.yaml</pre><p><em>helm upgrade — install</em> is idempotent. It installs if the release doesn’t exist, upgrades if it does. This means you can run the same command repeatedly without errors, which matters when you’re iterating on configuration.</p><p>The <em>— create-namespace</em> flag creates the <em>openbao</em> namespace if it doesn’t exist. Without it, the command fails if the namespace hasn’t been created manually first.</p><p>For standalone or HA mode, swap the values file:</p><pre># Standalone<br>--values values-standalone.yaml</pre><pre># HA Raft<br>--values values-ha.yaml</pre><p>After installation, wait for the pod to be ready:</p><pre>kubectl get pods -n openbao -w</pre><p>Press Ctrl+C to stop watching. The pod should show <em>1/1 Running</em>.</p><p><strong>The CSI driver.</strong> If you want pods to mount OpenBao secrets as files (instead of fetching them via API calls), you need the Secrets Store CSI driver and the OpenBao CSI provider. K3s 1.27+ bundles the base CSI driver. Check first:</p><pre>kubectl get pods -n kube-system | grep secrets-store</pre><p>If pods are already running, skip the base driver install. If not, install it:</p><pre>helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts<br>helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver \<br>  --namespace kube-system</pre><p>The OpenBao CSI provider is distributed as a DaemonSet manifest, not a Helm chart. There is no <em>openbao/openbao-csi-provider</em> Helm chart. The upstream manifest references image tag <em>1.4.0</em>, which was never published to Docker Hub. The fix is to patch it to <em>latest</em> during apply:</p><pre>curl -sL https://raw.githubusercontent.com/openbao/openbao-csi-provider/main/deployment/openbao-csi-provider.yaml \<br>  | sed &#39;s|openbao/openbao-csi-provider:1.4.0|openbao/openbao-csi-provider:latest|&#39; \<br>  | kubectl apply -f -</pre><p>This is the kind of thing that eats an hour if you don’t know about it. The manifest applies without error. The pods start. But they’re running a non-existent image, so they crashloop silently. Check the provider pods if secrets aren’t mounting.</p><p><strong>Don’t enable syncSecret unless you need it.</strong> The flag <em>— set syncSecret.enabled=true</em> makes the CSI driver create a Kubernetes Secret synced from the mounted files. That secret lands as Base64 in etcd, exactly what we’re trying to avoid. Only enable it if your application cannot read files and needs <em>envFrom</em> or a Kubernetes Secret reference.</p><h4>Init and unseal: the step dev mode skips</h4><p>Dev mode auto-initializes and auto-unseals. The token is <em>root</em>. You’re done.</p><p>Standalone and HA modes require manual initialization. This is the step that catches people off guard. The pod is running, but OpenBao rejects every request because it has no keys.</p><p>Initialize the cluster:</p><pre>kubectl exec -it -n openbao openbao-0 -- bao operator init -key-shares=3 -key-threshold=2</pre><p>This creates three key shares. You need any two of them to unseal. The command outputs three unseal keys and one initial root token. Save all of them. If you lose them, you must delete the PVC and start over. There is no recovery. Unseal the pod by running the unseal command twice with different keys:</p><pre>kubectl exec -it -n openbao openbao-0 -- bao operator unseal &lt;UNSEAL_KEY_1&gt;<br>kubectl exec -it -n openbao openbao-0 -- bao operator unseal &lt;UNSEAL_KEY_2&gt;</pre><p>OpenBao is sealed when it has no key material in memory. It’s unsealed when it has enough key shares to reconstruct the master key (in this case, 2 of 3). The sealed state is not a degraded mode. It’s a hard rejection. Every request returns a 403 until the vault is unsealed.</p><p>For HA mode, each replica must be unsealed individually:</p><pre># Unseal the leader<br>kubectl exec -it -n openbao openbao-0 -- bao operator unseal &lt;KEY_1&gt;<br>kubectl exec -it -n openbao openbao-0 -- bao operator unseal &lt;KEY_2&gt;</pre><pre># Unseal the standbys<br>kubectl exec -it -n openbao openbao-1 -- bao operator unseal &lt;KEY_1&gt;<br>kubectl exec -it -n openbao openbao-1 -- bao operator unseal &lt;KEY_2&gt;<br>kubectl exec -it -n openbao openbao-2 -- bao operator unseal &lt;KEY_1&gt;<br>kubectl exec -it -n openbao openbao-2 -- bao operator unseal &lt;KEY_2&gt;</pre><p>After unsealing all nodes, check the Raft peers:</p><pre>kubectl exec -it -n openbao openbao-0 -- bao operator raft list-peers</pre><p>The UI reflects the state at each stage. Dev mode shows the login page immediately. Standalone or HA before initialization shows a red “Sealed” banner with no login option. After initialization but before unseal, it shows the unseal form. After unseal, it shows the login page where you enter the initial root token.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/0*s0xu3AHw1x2-zSpl" /><figcaption>OpenBao UI</figcaption></figure><h4>The authentication chain: SA, Role, Policy, Secret</h4><p>This is the concept that determines whether everything works or nothing works. OpenBao’s Kubernetes authentication has four components, and all four must be present.</p><p><strong>ServiceAccount</strong>: Who is this pod? Every pod runs as a ServiceAccount. The default ServiceAccount gives no identity and no access. A named ServiceAccount has a JWT token mounted inside the pod at <em>/var/run/secrets/kubernetes.io/serviceaccount/token</em>. OpenBao calls the Kubernetes API to verify this token. If the token is real, OpenBao knows the pod’s identity.</p><p><strong>Role</strong>: What is this ServiceAccount allowed to do? A role binds specific ServiceAccounts (by name and namespace) to specific policies. Without a role, OpenBao knows who the pod is but grants no permissions.</p><p><strong>Policy</strong>: Which exact secrets can be accessed? A policy defines paths and capabilities. <em>Read</em> lets a pod get a secret. <em>list</em> lets it see available paths. <em>write</em> lets it create or update. <em>delete</em> lets it remove. Without a policy, even an authenticated pod with a role has zero permissions.</p><p><strong>Secret</strong> is the actual data stored in OpenBao’s KV store.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fUrF9OtgnlzDjQcq3fxN1w.png" /><figcaption>OpenBao diagram flow</figcaption></figure><p>The chain works like this: a pod sends its JWT token to OpenBao. OpenBao verifies the token with the Kubernetes API. OpenBao finds the role bound to that ServiceAccount. OpenBao checks the policy attached to that role. If the policy allows the requested path, the secret is returned.</p><p>Missing any link breaks the chain. No ServiceAccount: OpenBao can’t verify the pod. No role: OpenBao knows the pod but grants nothing. No policy: no rules exist to allow access. No secret: there’s nothing to return.</p><h4>Kubernetes auth: the setup</h4><p>With OpenBao running and unsealed, configure Kubernetes authentication. Enable the auth method:</p><pre>kubectl exec -it -n openbao openbao-0 -- bao auth enable kubernetes</pre><p>Tell OpenBao how to reach the Kubernetes API:</p><pre>kubectl exec -it -n openbao openbao-0 -- bao write auth/kubernetes/config \<br>    kubernetes_host=&quot;https://kubernetes.default.svc:443&quot;</pre><p>This endpoint is the in-cluster Kubernetes API server. OpenBao calls it to verify JWT tokens. Without this, OpenBao can’t validate any ServiceAccount token.</p><p>Create a policy that defines what paths a ServiceAccount can access:</p><pre>kubectl exec -it -n openbao openbao-0 -- bao policy write my-app-policy - &lt;&lt;EOF<br>path &quot;secret/data/my-app&quot; {<br>  capabilities = [&quot;read&quot;]<br>}<br>EOF</pre><p>This policy allows read access to <em>secret/data/my-app</em> and nothing else. The <em>secret/data/</em> prefix is required for KV v2. The data lives under <em>secret/data/</em>, while metadata lives under <em>secret/metadata/</em>.</p><p>Create a role that binds a ServiceAccount to the policy:</p><pre>kubectl exec -it -n openbao openbao-0 -- bao write auth/kubernetes/role/my-app-role \<br>    bound_service_account_names=my-app-sa \<br>    bound_service_account_namespaces=default \<br>    policies=my-app-policy \<br>    ttl=24h</pre><p>The <em>bound_service_account_names</em> field lists which ServiceAccounts this role applies to. The <em>bound_service_account_namespaces</em> field lists which namespaces those ServiceAccounts live in. Both must match. If your ServiceAccount is in namespace <em>apps</em> but you set <em>bound_service_account_namespaces=default</em>, authentication fails. This is the most common misconfiguration and the hardest to debug, because the error message doesn’t tell you which field is wrong.</p><p>The <em>ttl</em> controls how long the OpenBao token is valid after authentication. After the TTL expires, the pod must re-authenticate. Useful values: <em>10m</em> for sensitive workloads, <em>1h</em> for standard production, <em>24h</em> for convenience, <em>768h</em> for the maximum.</p><p>One ServiceAccount can access multiple secrets through a single role. Create one policy that covers all the paths:</p><pre>kubectl exec -it -n openbao openbao-0 -- bao policy write my-app-policy - &lt;&lt;EOF<br>path &quot;secret/data/db-config&quot; {<br>  capabilities = [&quot;read&quot;]<br>}<br>path &quot;secret/data/api-keys&quot; {<br>  capabilities = [&quot;read&quot;]<br>}<br>path &quot;secret/data/app-config&quot; {<br>  capabilities = [&quot;read&quot;]<br>}<br>EOF</pre><p>Then attach it to the role:</p><pre>kubectl exec -it -n openbao openbao-0 -- bao write auth/kubernetes/role/my-app-role \<br>    bound_service_account_names=my-app-sa \<br>    bound_service_account_namespaces=default \<br>    policies=my-app-policy \<br>    ttl=24h</pre><p>One ServiceAccount, one role, one policy, three secrets. The pattern scales: add paths to the policy, not new roles.</p><p>If you need to add a new secret path later, read the current policy, append the new path, and write it back. OpenBao replaces the entire policy on write. It doesn’t merge:</p><pre># Read current policy<br>kubectl exec -it -n openbao openbao-0 -- bao policy read my-app-policy</pre><pre># Write updated policy with all paths (old + new)<br>kubectl exec -it -n openbao openbao-0 -- bao policy write my-app-policy - &lt;&lt;EOF<br>path &quot;secret/data/db-config&quot; {<br>  capabilities = [&quot;read&quot;]<br>}<br>path &quot;secret/data/api-keys&quot; {<br>  capabilities = [&quot;read&quot;]<br>}<br>path &quot;secret/data/app-config&quot; {<br>  capabilities = [&quot;read&quot;]<br>}<br>path &quot;secret/data/s3-config&quot; {<br>  capabilities = [&quot;read&quot;]<br>}<br>EOF</pre><p>For the <em>list</em> capability, you need two path entries. One under <em>secret/data/</em> for the data, one under <em>secret/metadata/</em> for listing:</p><pre>path &quot;secret/data/my-app&quot; {<br>  capabilities = [&quot;read&quot;, &quot;list&quot;]<br>}<br>path &quot;secret/metadata/my-app&quot; {<br>  capabilities = [&quot;list&quot;]<br>}</pre><h4>Injection: how pods actually get secrets</h4><p>You have OpenBao running, initialized, unsealed, and configured with Kubernetes authentication. Your ServiceAccount has a role and a policy. Now the pod needs to actually read the secret. There are two ways to do this.</p><p><strong>Method A: CLI and init container.</strong> The pod authenticates to OpenBao using its ServiceAccount JWT token, reads the secret, and writes it to a shared volume. This requires an init container that runs before the application container starts.</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: my-app<br>  namespace: default<br>spec:<br>  template:<br>    spec:<br>      serviceAccountName: my-app-sa<br>      initContainers:<br>        - name: fetch-secret<br>          image: alpine:3<br>          command:<br>            - sh<br>            - -c<br>            - |<br>              apk add --no-cache curl jq<br>              TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)<br>              RESP=$(curl -sk https://openbao.openbao.svc.cluster.local:8200/v1/auth/kubernetes/login \<br>                -d &quot;{\&quot;role\&quot;:\&quot;my-app-role\&quot;,\&quot;jwt\&quot;:\&quot;$TOKEN\&quot;}&quot;)<br>              BAO_TOKEN=$(echo $RESP | jq -r &#39;.auth.client_token&#39;)<br>              SECRET=$(curl -sk https://openbao.openbao.svc.cluster.local:8200/v1/secret/data/my-app \<br>                -H &quot;X-Vault-Token: $BAO_TOKEN&quot;)<br>              echo $SECRET | jq -r &#39;.data.data.apiKey&#39; &gt; /mnt/secrets/apiKey<br>              echo $SECRET | jq -r &#39;.data.data.dbPass&#39; &gt; /mnt/secrets/dbPass<br>          volumeMounts:<br>            - name: secrets<br>              mountPath: /mnt/secrets<br>      containers:<br>        - name: app<br>          image: my-app:1.0<br>          volumeMounts:<br>            - name: secrets<br>              mountPath: /mnt/secrets<br>              readOnly: true<br>      volumes:<br>        - name: secrets<br>          emptyDir:<br>            medium: Memory</pre><p>The init container fetches the JWT token from the ServiceAccount mount, sends it to OpenBao’s Kubernetes auth endpoint, receives an OpenBao token in return, uses that token to read the secret, and writes the values to a shared <em>emptyDir</em> volume backed by tmpfs. The application container reads the files normally. Nothing touches etcd.</p><p>The downside: you’re writing shell scripts in YAML. The authentication flow is manual. If the secret changes, the pod needs to restart to pick up the new value.</p><p><strong>Method B: CSI volume mount.</strong> The CSI driver handles authentication transparently. The pod mounts a CSI volume that maps to an OpenBao secret. No init container, no sidecar, no application changes.</p><pre>apiVersion: secrets-store.csi.x-k8s.io/v1<br>kind: SecretProviderClass<br>metadata:<br>  name: openbao-my-app<br>spec:<br>  provider: openbao<br>  parameters:<br>    roleName: my-app-role<br>    objects: |<br>      - objectName: apiKey<br>        secretPath: secret/data/my-app<br>        secretKey: apiKey<br>      - objectName: dbPass<br>        secretPath: secret/data/my-app<br>        secretKey: dbPass<br>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: my-app<br>  namespace: default<br>spec:<br>  template:<br>    spec:<br>      serviceAccountName: my-app-sa<br>      containers:<br>        - name: app<br>          image: my-app:1.0<br>          volumeMounts:<br>            - name: openbao-secrets<br>              mountPath: /mnt/secrets<br>              readOnly: true<br>      volumes:<br>        - name: openbao-secrets<br>          csi:<br>            driver: secrets-store.csi.k8s.io<br>            readOnly: true<br>            volumeAttributes:<br>              secretProviderClass: openbao-my-app</pre><p>The CSI driver authenticates to OpenBao using the pod’s ServiceAccount token, fetches the secrets defined in the <em>SecretProviderClass</em>, and mounts them as files at the specified path. The application reads files. Nothing is in etcd. No init container. No shell scripts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0YuFDutoqrglAPCuG2gr0A.png" /><figcaption>OpenBao with csi driver flow</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kIbIqRiycIS7rD9e7u4llw.png" /><figcaption>OpenBao method comparison</figcaption></figure><p>Use CSI when the driver is already installed. Use the init container when you can’t install the CSI driver or need more control over the fetch logic.</p><h4>Full setup: start to finish</h4><p>Here’s the complete flow for a real workload. An existing ServiceAccount <em>ecr-app-sa</em> in namespace <em>apps</em> needs access to database credentials stored in OpenBao.</p><p>Verify the ServiceAccount exists:</p><pre>kubectl get sa ecr-app-sa -n apps<br>kubectl describe sa ecr-app-sa -n apps</pre><p>Log into OpenBao (required for all CLI commands):</p><pre>kubectl exec -it -n openbao openbao-0 -- sh -c \<br>  &quot;bao login &lt;ROOT_TOKEN&gt;&quot;</pre><p>Without this, <em>bao</em> commands inside the pod use an empty token and return 403 on everything.</p><p>Enable the KV secrets engine and Kubernetes auth (one-time setup):</p><pre>kubectl exec -n openbao openbao-0 -- bao secrets enable -path=secret kv-v2<br>kubectl exec -n openbao openbao-0 -- bao auth enable kubernetes<br>kubectl exec -n openbao openbao-0 -- bao write auth/kubernetes/config \<br>    kubernetes_host=&quot;https://kubernetes.default.svc:443&quot;</pre><p>Store the secret:</p><pre>kubectl exec -n openbao openbao-0 -- sh -c \<br>  &#39;bao kv put secret/db-config \<br>    db-host=&quot;prod-db.example.com&quot; \<br>    db-username=&quot;app-user&quot; \<br>    db-password=&quot;CHANGE_ME&quot;&#39;</pre><p>Create the policy:</p><pre>kubectl exec -n openbao openbao-0 -- bao policy write db-config-policy - &lt;&lt;EOF<br>path &quot;secret/data/db-config&quot; {<br>  capabilities = [&quot;read&quot;, &quot;list&quot;]<br>}<br>EOF</pre><p>Create the role, binding the existing ServiceAccount to the policy:</p><pre>kubectl exec -n openbao openbao-0 -- bao write auth/kubernetes/role/ecr-app-role \<br>    bound_service_account_names=ecr-app-sa \<br>    bound_service_account_namespaces=apps \<br>    policies=db-config-policy \<br>    ttl=24h</pre><p>Verify everything:</p><pre>bao read auth/kubernetes/role/ecr-app-role<br>bao policy read db-config-policy<br>bao kv get secret/db-config</pre><p>Deploy the application using the existing ServiceAccount:</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: my-app<br>  namespace: apps<br>spec:<br>  template:<br>    spec:<br>      serviceAccountName: ecr-app-sa<br>      containers:<br>        - name: my-app<br>          image: 123456789.dkr.ecr.us-east-1.amazonaws.com/my-app:latest</pre><p>The pod gets ECR pull access through <em>imagePullSecrets</em> on the ServiceAccount and OpenBao secret access through the Kubernetes auth role. All from the same ServiceAccount.</p><p>Now, what happens when you need to rotate a secret? You update the value in OpenBao or you can update via OpenBao UI:</p><pre>kubectl exec -n openbao openbao-0 -- sh -c \<br>  &#39;bao kv put secret/db-config \<br>    db-host=&quot;prod-db.example.com&quot; \<br>    db-username=&quot;app-user&quot; \<br>    db-password=&quot;NEW_PASSWORD&quot;&#39;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UOK5zGEE2TNQ80WooSkWTg.png" /><figcaption>Example service-testing, we can create new version and edit its key value. in the screen shot, the applied value is version 2</figcaption></figure><p>Then roll out the pods so they pick up the new secret:</p><pre>kubectl rollout restart deployment/my-app -n apps</pre><p>After the rollout completes, the pods mount the updated secret value through the CSI driver, and the new version is reflected in the OpenBao UI.</p><h4>Debugging: what broke and where</h4><p>When something doesn’t work, the error message rarely tells you which layer failed. Here’s a map from symptom to cause.</p><p><strong>403 permission denied on every request.</strong> The most common cause is a namespace mismatch. Check that <em>bound_service_account_namespaces</em> matches the namespace where the ServiceAccount actually exists. If the SA is in <em>apps</em> but the role says <em>default</em>, authentication fails silently. The second most common cause: the policy doesn’t include the requested path, or the capabilities don’t include <em>read</em>.</p><p><strong>OpenBao returns “sealed.”</strong> The pod is running but OpenBao was never initialized, or it was restarted and needs to be unsealed again. Run <em>bao status</em> to check. If it says “Initialized: false,” run <em>bao operator init</em>. If it says “Sealed: true,” run <em>bao operator unseal</em> with enough key shares.</p><p><strong>Pod can’t find the secret at the mount path.</strong> Check that the CSI provider pod is running: <em>kubectl get pods -n kube-system | grep csi</em>. Check that the <em>SecretProviderClass</em> references the correct <em>roleName</em> and <em>secretPath</em>. Check that the ServiceAccount mounted on the pod matches the one bound to the OpenBao role.</p><p><strong>Authentication returns 404 on the auth endpoint.</strong> The Kubernetes auth method isn’t enabled. Run <em>bao auth list</em> and verify <em>kubernetes/</em> appears. If not, run <em>bao auth enable kubernetes</em>.</p><p><strong>Token expired.</strong> The TTL on the role is too short, or the pod hasn’t re-authenticated. Check the role’s TTL with <em>bao read auth/kubernetes/role/</em>. Increase it if needed, or configure the application to re-authenticate before expiry.</p><p><strong>Policy write lost existing paths.</strong> OpenBao replaces the entire policy on write. If you write a new policy with only the new path, you lose access to all previous paths. Always read the current policy first, append the new paths, then write the complete thing.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=752549c1c620" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How To Set Up a CI/CD Pipeline with GitLab for Kubernetes Deployment]]></title>
            <link>https://medium.com/@PlatformEnthusiast/how-to-set-up-a-ci-cd-pipeline-with-gitlab-for-kubernetes-deployment-f00540e410cc?source=rss-b300b45c8552------2</link>
            <guid isPermaLink="false">https://medium.com/p/f00540e410cc</guid>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[cicd-pipeline]]></category>
            <category><![CDATA[gitlab]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[dora]]></category>
            <dc:creator><![CDATA[The Platform Enthusiast]]></dc:creator>
            <pubDate>Sun, 03 May 2026 11:46:16 GMT</pubDate>
            <atom:updated>2026-05-25T04:14:50.338Z</atom:updated>
            <content:encoded><![CDATA[<p>Every commit you push should trigger an automatic build, test, and deployment cycle to Kubernetes. No more “it worked on my machine.” No more manually running docker push and then kubectl apply after a long day. This tutorial shows you how to wire that up from scratch: a GitLab CI/CD pipeline that builds a container image, pushes it to ECR, and deploys it to your Kubernetes cluster via Helm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ofVaWbnIw1-IS1xU.png" /><figcaption>CICD Pipeline source: <a href="https://www.incredibuild.com/blog/what-is-a-ci-cd-pipeline">https://www.incredibuild.com/blog/what-is-a-ci-cd-pipeline</a></figcaption></figure><p>The YAML file below is the complete source. I’ll walk through each stage and explain why it’s structured the way it is. Copy what you need.</p><pre><br>stages:<br>  - code-checker<br>  - code-scanner<br>  - sast<br>  - unit-test<br>  - build<br>  - test-docker-image<br>  - scan-docker<br>  - push-image<br>  - deploy<br>  - deploy-status<br></pre><h4>Prerequisites</h4><p>This pipeline deploys to Kubernetes. Before you copy it, make sure you have:</p><ul><li>A Dockerfile in the repository root</li><li>A package.json with a test:coverage script</li><li>A GitLab runner with the “kubernetes” tag registered to your GitLab instance — this is required, not optional</li><li>AWS credentials for ECR push (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)</li><li>A Kubernetes cluster with a namespace already created (the pipeline won’t create it for you)</li><li>kubectl access confirmed and Helm installed locally if you want to test chart rendering before the pipeline runs</li></ul><h3>Introduction</h3><p>If you already know what CI and CD mean, skip this section. If you’ve heard the terms but never quite grasped why they matter, read on for a more detailed explanation. Refer to my other <a href="https://medium.com/@PlatformEnthusiast/the-ci-cd-maturity-gap-thats-costing-your-team-100x-in-bug-fixes-73e280515fbb">article</a></p><p>Think of it like a factory assembly line. Instead of one person trying to assemble an entire car from start to finish (and hoping nothing goes wrong), you have automated stations along the line that catch problems at each step. Engine gets installed? There’s a station that checks the bolts. Wheels go on? There’s a station that checks alignment. If something fails at any station, the whole line stops before the problem reaches the next step.</p><p>That’s this pipeline. The stations are the “stages.” In this case, every “station” runs inside your Kubernetes cluster — the same environment that will eventually host your application. The pipeline doesn’t just test your code; it validates the exact container image and Helm chart that will run in production.</p><p><strong>The core problem CI/CD solves</strong> on Kubernetes is the gap between “it works in my Docker container” and “it works in my cluster.” A container that starts fine locally might fail in production because of resource limits, missing environment variables, or image pull errors from the cluster’s registry policy. The pipeline catches all of that before anything reaches your cluster.</p><p>A CI/CD pipeline closes that gap by:</p><ul><li><strong>Automating environment setup</strong> — The pipeline builds a clean, consistent environment every time. No “it works because my machine has this weird config.”</li><li><strong>Catching problems early</strong> — Tests run automatically on every commit. Problems get caught before they compound.</li><li><strong>Standardizing deployments</strong> — The same artifact that passed testing goes to production. No manual “oh, I forgot to update this config.”</li><li><strong>Making rollback trivial</strong> — If production breaks, you redeploy the previous version in seconds.</li></ul><p>The pipeline in this tutorial sits at DORA’s <strong>Level 3: Defined</strong> for a Kubernetes deployment — all code passes through linting, testing, and SAST before merge. Container scanning catches image vulnerabilities. The Helm chart is validated before deployment. The team trusts the pipeline enough to stop doing manual “kubectl apply” runs &quot;just in case.&quot; That trust is the whole point. Before starting, let’s create a new YAML file called js-stage.yml. Copy and paste the code in every stage below</p><h4>Stage 1: Code Checking</h4><p>Code checking runs before anything else. You catch style violations and secret leaks early before they enter the test matrix.</p><pre>linter:<br>  extends: .node-base<br>  stage: code-checker<br>  allow_failure: false<br>  script:<br>    - set -o pipefail<br>    - npm run lint | tee lint-report.txt<br>  artifacts:<br>    when: always<br>    paths:<br>      - lint-report.txt<br>    expire_in: 1 hour</pre><p>.node-base is a hidden template that every Node.js job extends. It pins the Docker image to a specific Node version, pulls the npm cache, and runs npm ci before every job. One definition, reused across eight jobs.</p><pre>.node-base:<br>  image: node:${NODE_VERSION}-alpine<br>  tags:<br>    - kubernetes<br>  cache:<br>    key:<br>      prefix: &quot;node-v2-${NODE_VERSION}&quot;<br>      files:<br>        - package-lock.json<br>    paths:<br>      - node_modules/<br>      - .npm/<br>    policy: pull<br>  before_script:<br>    - npm ci --prefer-offline || npm install --prefer-offline</pre><p><em>allow_failure: false</em> On the linter means a failing lint breaks the pipeline immediately. No waiting for tests. No deploying with known issues. The <em>artifacts</em> block uploads the lint report to the GitLab UI. Even if the job fails, you can download <em>lint-report.txt </em>and inspect the output in the job log.</p><h4>Stage 2: Secret Detection</h4><p>The secret detection Job runs TruffleHog against your filesystem.</p><pre>secret-detection:<br>  stage: code-checker<br>  tags: [kubernetes]<br>  image: trufflesecurity/trufflehog:latest<br>  script:<br>    - rm -rf .git node_modules trufflehog-report.json<br>    - trufflehog filesystem --fail . &gt; secret-detection-report.json</pre><p>It deletes .git and nodes_module before scanning. You don’t want TruffleHog to flag your own commit history or third-party dependencies as secrets — only the code you’re shipping.</p><p>The dockerfile-lint job uses Hadolint to validate your Dockerfile in the same stage:</p><pre>dockerfile-lint:<br>  stage: code-checker<br>  tags: [kubernetes]<br>  image: hadolint/hadolint:latest-alpine<br>  script:<br>    - set -o pipefail<br>    - hadolint Dockerfile | tee hadolint-report.txt</pre><h4>Stage 3: Dependency and Security Scanning</h4><p>After code checks pass, the pipeline moves to scanning. These jobs are allowed to fail <em>allow_failure: true</em> because you’re doing vulnerability discovery, not gatekeeping.</p><pre>dependency-checker:<br>  stage: code-scanner<br>  needs: [&quot;linter&quot;, &quot;format-checker&quot;, &quot;dockerfile-lint&quot;, &quot;secret-detection&quot;]<br>  allow_failure: true<br>  tags: [kubernetes]<br>  image: alpine:latest<br>  before_script:<br>    - apk add --no-cache osv-scanner nodejs npm<br>  script:<br>    - osv-scanner --recursive --format=json . &gt; osv-report.json || EXIT_CODE=$?<br>    - osv-scanner --recursive --format=table .<br>    - osv-scanner --recursive --format=cyclonedx . &gt; gl-dependency-scanning-report.json || true<br>    - exit ${EXIT_CODE:-0}<br>  reports:<br>    dependency_scanning: gl-dependency-scanning-report.json</pre><p>This job waits for all four code-checker jobs to complete. GitLab starts dependency scanning the moment the linter finishes no waiting for the whole stage to drain.</p><p>The reports: dependency_scanning block injects the CycloneDX report into GitLab’s Dependency Scanning UI. Your vulnerabilities are displayed in the Merge Request sidebar without any additional configuration.</p><p>The SAST job runs Semgrep next:</p><pre>sast:<br>  stage: sast<br>  needs: [&quot;dependency-checker&quot;]<br>  image: returntocorp/semgrep<br>  script:<br>    - semgrep scan --config auto --json -o semgrep-report.json --error --severity ERROR<br>      --exclude=&quot;coverage/&quot; --exclude=&quot;tests/&quot; --exclude=&quot;.gitignore&quot;</pre><p><em>severity ERROR</em> limits findings to actual security issues. Warnings and informational findings don’t block the pipeline. Semgrep produces a JSON report saved as an artifact — the same pattern used by every other scanning tool in this pipeline.</p><h4>Stage 4: Unit Tests</h4><p>Tests run against the source code before any Docker image is built. This is intentional. A broken unit test should never trigger a container build.</p><pre>unit-test:<br>  stage: unit-test<br>  needs: [&quot;sast&quot;]<br>  image: node:${NODE_VERSION}-alpine<br>  before_script:<br>    - npm ci --prefer-offline || npm install --prefer-offline<br>  script:<br>    - npm run test:coverage -- --coverageReporters=text --coverageReporters=cobertura<br>      --coverageReporters=json-summary --json --outputFile=jest-results.json<br>  coverage: &#39;/All files[^|]*\|[^|]*\s+([\d\.]+)/&#39;<br>  allow_failure: false<br>  reports:<br>    junit: junit.xml<br>    coverage_report:<br>      coverage_format: cobertura<br>      path: coverage/cobertura-coverage.xml</pre><p>The coverage regex parses Jest’s output and posts the percentage directly to the GitLab UI. Open any merge request and you’ll see the coverage trend without opening a build log.</p><p>The pipeline enforces an 80% coverage floor using awk:</p><pre>awk -v cov=&quot;$COVERAGE&quot; -v threshold=&quot;$COVERAGE_THRESHOLD&quot; &#39;BEGIN {<br>  if (cov &lt; threshold) { exit 1 }<br>}&#39;</pre><p>If coverage drops below 80%, the job fails. The COVERAGE_THRESHOLD variable at the top of the file makes this threshold configurable per project without touching the job definition.</p><h4>Stage 5: Build and Push the Docker Image</h4><p>With tests passing, the pipeline builds a multi-architecture Docker image and pushes it to the GitLab container registry.</p><pre>dockerize:<br>  stage: build<br>  image: docker:24-dind<br>  services:<br>    - name: docker:24-dind<br>      command: [&quot;--tls=false&quot;]<br>  before_script:<br>    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY<br>    - docker buildx create --use --name multi-platform-builder<br>  script:<br>    - docker buildx build --platform $DOCKER_ARCH $CACHE_OPTS<br>      -t $IMAGE_NAME -t $LATEST_IMAGE --push .<br>  needs: [&quot;unit-test&quot;]</pre><p>DOCKER_ARCH: &quot;linux/amd64,linux/arm64&quot; tells Docker Buildx to build for both AMD and ARM architectures in a single push. Your image runs natively on Apple Silicon and x86_64 EC2 instances without separate build jobs.</p><p>The test-docker-image job runs the container immediately after the build:</p><pre>test-docker-image:<br>  stage: test-docker-image<br>  script:<br>    - CONTAINER_ID=$(docker run -d $LATEST_IMAGE)<br>    - sleep 5<br>    - docker ps | grep $CONTAINER_ID &amp;&amp; echo &quot;Container is running&quot; || echo &quot;Container stopped&quot;<br>    - docker stop $CONTAINER_ID</pre><p>Five seconds of runtime. If the container exits unexpectedly, the job fails and the pipeline never reaches the push stage.</p><h4>Stage 6: Push to AWS ECR</h4><p>The push-docker-image job copies the multi-arch image from GitLab’s registry to AWS ECR. It doesn’t rebuild but it uses <em>docker buildx imagetools create</em> to copy the manifest directly between registries.</p><pre>push-docker-image:<br>  stage: push-image<br>  image: alpine:3.19<br>  before_script:<br>    - apk add --no-cache docker-cli docker-cli-buildx aws-cli<br>    - aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $ECR_REGISTRY<br>  script:<br>    - docker buildx imagetools create -t $ECR_REGISTRY/$ECR_IMAGE_NAME:$CI_COMMIT_SHA $LATEST_IMAGE<br>    - docker buildx imagetools create -t $ECR_REGISTRY/$ECR_IMAGE_NAME:latest $LATEST_IMAGE<br>    - aws ecr describe-images --repository-name $ECR_IMAGE_NAME --image-ids imageTag=$CI_COMMIT_SHA</pre><p><em>imagetools create</em> is the fast path. It copies the existing image manifest rather than pulling, retagging, and repushing each layer. For a cold push of a 500MB image, this shaves 60–90 seconds off the job compared to a <em>docker pull &amp;&amp; docker push</em> sequence.</p><p>The job exports IMAGE_DIGEST and IMAGE_TAG as a dotenv artifact:</p><pre>artifacts:<br>  reports:<br>    dotenv: build.env</pre><p>This artifact passes environment variables to the deploy job downstream.</p><h4>Stage 7: Deploy to Kubernetes with Helm</h4><p>The deploy-helm job runs as a manual step by default (when: manual). You trigger it after reviewing the build log and test results. It deploys into your Kubernetes cluster using a Helm chart stored in your repository.</p><p>Why Helm and not raw <em>kubectl apply</em>? Helm is the standard for production Kubernetes deployments because:</p><ul><li><strong>Releases are versioned</strong> — helm history shows every revision; rollback is a single command</li><li><strong>Templates</strong> — the same chart works across dev, staging, and prod by swapping --set values</li><li><strong>Atomic updates</strong> — helm upgrade --install handles upgrades and initial installs identically</li></ul><pre>deploy-helm:<br>  stage: deploy<br>  needs: [&quot;push-docker-image&quot;]<br>  image: alpine/helm:3.14.0<br>  when: manual<br>  script:<br>    - helm upgrade --install $HELM_RELEASE_NAME $HELM_CHART_PATH<br>      --namespace $NAMESPACE<br>      --set image.repository=$ECR_REGISTRY/$ECR_IMAGE_NAME<br>      --set image.tag=$CI_COMMIT_SHA<br>      --wait --timeout $HELM_WAIT_TIMEOUT</pre><p><em>— wait</em> blocks until all pods are ready. If the deployment stalls for more than 300 seconds (HELM_WAIT_TIMEOUT: 300s), the job fails and GitLab prints pod events and <em>kubectl describe</em> output to the job log — useful for debugging <em>CrashLoopBackOff</em> or <em>ImagePullBackOff </em>errors without leaving the GitLab UI.</p><p>The deploy-status job runs after the Helm deploy:</p><pre>deploy-status:<br>  stage: deploy-status<br>  needs:<br>    - job: deploy-helm<br>      artifacts: true<br>  script:<br>    - kubectl rollout status deployment/$HELM_RELEASE_NAME -n $NAMESPACE --timeout=180s<br>    - kubectl describe pods -n $NAMESPACE | grep -A 20 &quot;Events:&quot;</pre><p>It waits for the rollout to complete. If it fails, it reads the Helm revision history and automatically rolls back to the previous version. No manual intervention needed.</p><pre>REVISION=$(helm history $HELM_RELEASE_NAME -n $NAMESPACE --max 1 --output json | jq &#39;.[0].revision&#39;)<br>if [ &quot;$REVISION&quot; -ge 2 ]; then<br>  helm rollback $HELM_RELEASE_NAME -n $NAMESPACE<br>fi</pre><h4>Key Takeaways</h4><p>This pipeline has one design principle running through it: fail fast, recover automatically on Kubernetes. Tests run before builds. Builds run before deploys. Secrets are caught in the first stage, not after the image is already in the cluster.</p><p>The hidden .node-base template keeps the Node.js configuration in one place. The needs keyword parallelizes stage execution, so dependency scanning starts before unit tests finish. The dotenv artifact passes image digests from the push job to the deploy job without manual variable copying. Helm&#39;s revision history enables automatic rollback when the deployment fails.</p><p>For a new project, copy the js-stage.yml file to your repository root as .gitlab-ci.yml Confirm your Kubernetes runner is registered, set your CI/CD variables, and push. Your first pipeline run will take 12–15 minutes. After that, GitLab caches your npm and Docker layers — subsequent runs finish in 4–6 minutes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f00540e410cc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The CI/CD Maturity Gap That’s Costing Your Team 100x in Bug Fixes]]></title>
            <link>https://medium.com/@PlatformEnthusiast/the-ci-cd-maturity-gap-thats-costing-your-team-100x-in-bug-fixes-73e280515fbb?source=rss-b300b45c8552------2</link>
            <guid isPermaLink="false">https://medium.com/p/73e280515fbb</guid>
            <category><![CDATA[dora-metrics]]></category>
            <category><![CDATA[cicd-pipeline]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[The Platform Enthusiast]]></dc:creator>
            <pubDate>Sat, 25 Apr 2026 09:33:01 GMT</pubDate>
            <atom:updated>2026-04-25T09:33:01.947Z</atom:updated>
            <content:encoded><![CDATA[<p>Here’s the situation most teams recognize: deployment day. Someone on the team, usually the one who set things up originally, SSHs into the server, pulls the latest code, runs a build script, restarts the service, and hopes for the best. Sometimes it works. Sometimes production goes down for an hour. No one really knows what went wrong because the process isn’t documented. It just… happens.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m_i2Xc6-0hNNiMDxY_nwrg.png" /><figcaption>Level 1 vs Level 5 CICD Pipeline (AI Generated Images)</figcaption></figure><p>That team is probably deploying once a month, if they’re lucky. And they’re spending 26% of their engineering time on bugs that shouldn’t have reached production. The teams that deploy multiple times per day? They didn’t get there by hiring more people or working weekends. They built a better system, a CI/CD pipeline that catches the mistakes early and makes deploying feel routine instead of terrifying. This article explains the theory behind CI/CD pipelines, how to assess where your team stands using DORA’s five maturity levels, and the realistic path from manual chaos to elite performance. No prior DevOps experience required.</p><h3><strong>What CI/CD Actually Is (In Plain Terms)</strong></h3><p><strong>CI</strong> stands for Continuous Integration. The core idea: every time a developer pushes code, the system automatically verifies that the code still works. It compiles, runs tests, and reports back usually in under 10 minutes. No manual QA step. If something broke, you find out while the code is still fresh in your mind, not three weeks later when you’re trying to remember what you changed.</p><p><strong>CD</strong> stands for Continuous Delivery or Continuous Deployment, depending on how far you automate. In Continuous Delivery, the pipeline prepares a release-ready artifact after all checks pass, but a human decides when to actually push to production. In Continuous Deployment, the pipeline deploys automatically after all checks pass, with no human intervention needed.</p><blockquote>The goal of both: reduce the friction between writing code and getting it to users.</blockquote><p>Think of it like a factory assembly line. Instead of one person trying to assemble an entire car from start to finish (and hoping nothing goes wrong), you have automated stations along the line that catch problems at each step. Is the engine installed? There’s a station that checks if the bolts are tight. Wheels go on? There’s a station that checks alignment. If something fails at any station, the whole line stops before the problem reaches the next step. That’s CI/CD. The stations are the “stages” in your pipeline.</p><h3><strong>The Theory: Why This Matters</strong></h3><p>The core problem CI/CD solves is the gap between “it works on my machine” and “it works in production.” On your local machine, everything is predictable. You control the environment. You installed the right version of Node. You have the right environment variables. You ran the build command exactly the way it’s supposed to be run.</p><p>Production is a different story. The server has a different OS version. The environment variables are set differently. The load is higher. Users do things you didn’t expect. Dependencies that worked locally might conflict with something else running on the same server. That gap between local and production is where incidents happen.</p><p>A CI/CD pipeline closes that gap by:</p><ol><li><strong>Automating environment setup, </strong>the pipeline builds a clean, consistent environment every time. No, “it works because my machine has this weird config file.”</li><li><strong>Catching problems early, </strong>Tests run automatically on every commit. Problems get caught before they compound.</li><li><strong>Standardizing deployments, </strong>the same artifact that passed testing goes to production. No manual “oh, I forgot to update this config file.”</li><li><strong>Making rollback trivial, </strong>If production breaks, you redeploy the previous version in seconds instead of trying to remember what changed.</li></ol><blockquote>The theory is simple: if you automate the boring parts and make the critical parts reversible, you can move faster without breaking things.</blockquote><h3><strong>The Five Maturity Levels</strong></h3><p>Not all pipelines are created equal. The DORA (DevOps Research and Assessment) team at Google has studied thousands of organizations over the past decade and identified five maturity levels. Each level unlocks new capabilities. Most teams start at Level 1 or 2 and work their way up.</p><p>DORA’s research is the most rigorous study of DevOps practices in existence. They’ve surveyed over 50,000 professionals across industries and correlated pipeline characteristics with outcomes. What they found: the organizations that excel at software delivery don’t just move faster — they recover from failures faster, maintain higher reliability, and respond to market demands more quickly. The maturity levels aren’t just benchmarks; they’re predictors of business performance. Elite performers are twice as likely to exceed their business goals. The pipeline isn’t an IT concern; it’s a competitive advantage.</p><p><strong>Level 1: Initial (Ad-hoc)<br></strong>This is where most teams start. Deployments are manual. A developer SSHs into the server, runs some commands, and hopes for the best. Characteristics:</p><ul><li>No automated tests (or tests that no one runs)</li><li>Deployments require the one person who “knows how it works.”</li><li>Rollbacks are painful and usually involve manually undoing changes</li><li>Deployments happen monthly or less</li><li>When something breaks, nobody is quite sure why</li></ul><p>This level works for prototypes and solo projects. It doesn’t scale to teams.</p><p><strong>Level 2: Managed (Repeatable)<br></strong>Automation enters the picture. Basic scripts handle builds and deployments. Things become more consistent, even if they’re not fully automated. Characteristics:</p><ul><li>Automated builds on code commit</li><li>Basic unit tests run automatically</li><li>Deployments to dev/staging are scripted (if not fully automated)</li><li>Some monitoring exists</li><li>Rollback is still manual but documented</li></ul><p>Most teams plateau here for a while. The pipeline exists but isn’t trusted. Developers still do manual checks before pushing. The automation is treated as a suggestion, not a source of truth.</p><p><strong>Level 3: Defined (Measured)<br></strong>The team commits to the pipeline. All code goes through CI before merge. Deployments to staging are fully automated. Security scanning enters the picture. Characteristics:</p><ul><li>All code passes through linting, testing, and SAST before merging</li><li>Container scanning catches vulnerabilities in images</li><li>Staging deployments are fully automated (canary or blue-green)</li><li>Monitoring dashboards exist for key metrics</li><li>Security gates block known vulnerabilities</li></ul><p>At this level, the pipeline is trusted. Developers stop doing manual checks “just in case.” If the pipeline says it’s good, it goes. This is where the productivity jump happens: engineers stop babysitting deployments and start building features instead.</p><p><strong>Level 4: Quantitatively Managed (Optimizing)<br></strong>Data drives decisions. The team tracks DORA metrics and actively works to improve them. Canary deployments validate with real traffic before full rollout. Observability is comprehensive. Characteristics:</p><ul><li>Canary deployments with automated rollback based on metrics</li><li>Full observability: metrics, logs, traces</li><li>DORA metrics are tracked and reviewed monthly</li><li>Security is integrated at every stage (DevSecOps)</li><li>Incident response is fast problems are detected and triaged quickly</li><li>Deployment frequency: daily to multiple times per day</li></ul><p>This is the target for most production teams. Deployments feel routine. When incidents happen, they’re caught quickly and resolved fast. The team ships continuously without fearing the process.</p><p><strong>Level 5: Optimizing (Continuous Improvement)<br></strong>The pipeline doesn’t just follow rules. It learns. ML-driven anomaly detection identifies problems before they surface. Feature flags decouple deployments from feature releases. Self-healing infrastructure recovers from failures automatically. Characteristics:</p><ul><li>Progressive delivery with automated analysis</li><li>Feature flags for independent releases</li><li>ML-based anomaly detection</li><li>Chaos engineering validates resilience</li><li>On-demand deployments with near-zero failure rates</li><li>Self-healing infrastructure</li></ul><p>Very few organizations reach this level. It requires significant investment in observability, automation, and tooling. But for teams that do, deployment becomes a non-event. The system handles most problems automatically, and humans are alerted only when actual intervention is needed.</p><p><strong>The DORA Metrics: Measuring What Matters<br></strong>How do you know if your pipeline is improving? You track the DORA metrics four measurements that predict overall organizational performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UmOm4eJe8Me_aBrPUjKPuw.png" /><figcaption>DORA Metric (DevOps Research and Assessment)</figcaption></figure><p>Elite teams deploy multiple times per day. When a bug is found, it’s fixed and shipped within an hour. Change failure rates remain under 15% most deployments are successful. When something does break, recovery takes under an hour.</p><p>Low teams deploy monthly. Bugs sit in the queue for months before shipping. When a deployment goes wrong, it takes a week to recover because the process is manual, poorly documented, and relies on a few key individuals who may be on vacation. The metrics aren’t arbitrary. They’re measuring the things that actually affect how fast your team can respond to user needs and recover from problems.</p><h4>How Maturity Affects Your Deployment Strategy</h4><p>Your deployment strategy should match your maturity level. Advanced strategies require higher maturity but provide greater reliability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TA4npd6ii90bdqcDcs3yFA.png" /><figcaption>Deployment Strategy</figcaption></figure><p>Canary deserves special attention. In a canary deployment, you route a small percentage of production traffic to the new version, 5% initially, while the rest stays on the current version. You monitor metrics for a period, then increase to 20%, then 50%, then 100%. If error rates spike at any step, you roll back automatically. The power of canary: it lets you validate with real users before committing fully. A bug that affects 5% of users for 5 minutes is recoverable. A bug that affects 100% of users for an hour is an incident.</p><p>If there’s one idea to take away from this article, it’s this: the cost of fixing a bug increases exponentially the later in the pipeline it’s found. This isn’t speculation. The IBM Systems Sciences Institute’s research coined what’s known as the “Rule of 100” (a bug fixed during design costs roughly $100). The same bug in production? $10,000 to $100,000+. That’s a <strong>100x multiplier </strong>purely from catching it earlier.</p><p>Here’s the other number that should stop your teams spend <strong>26% of developer time </strong>reproducing and fixing failing tests, according to Cambridge Judge Business School research. That’s roughly <strong>$61 billion per year </strong>in salary costs alone, just on test failures that CI should catch automatically.</p><p>The DORA data backs this up. Elite performers recover from failed deployments 2,293x faster than low performers. They deploy 182x more frequently per year. The gap isn’t talent, it’s pipeline maturity.</p><p>The pipeline isn’t about adding steps. It’s about catching problems at the cheapest possible stage. Every stage you add to the front of the pipeline saves multiples in time and cost downstream. This is why “shift left” is a core principle in modern CI/CD: move security and quality checks as early as possible. SAST runs before the build. Dependency scanning happens before the image is created. Container scanning happens before the image is pushed.</p><p>By the time code reaches production, it should have passed through so many gates that deployment feels routine because the hard questions have already been answered automatically.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=73e280515fbb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Chaos to Continuous Delivery: The Power of Structured Commits]]></title>
            <link>https://medium.com/@PlatformEnthusiast/from-chaos-to-continuous-delivery-the-power-of-structured-commits-b4089f4c1e98?source=rss-b300b45c8552------2</link>
            <guid isPermaLink="false">https://medium.com/p/b4089f4c1e98</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[git]]></category>
            <dc:creator><![CDATA[The Platform Enthusiast]]></dc:creator>
            <pubDate>Tue, 24 Mar 2026 16:18:44 GMT</pubDate>
            <atom:updated>2026-03-26T13:37:22.172Z</atom:updated>
            <content:encoded><![CDATA[<p>In software development, having a standardised commit format and message is very useful. It can help any developer to learn and adapt to any task in any organisation. It isn’t just a “nice-to-have” formatting rule; they are the API for your project’s history.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gN_TPt7171NEo-KI6yp3EA.jpeg" /><figcaption>AI Generated Image (chaotic git commits and well-structured Conventional Commits)</figcaption></figure><p>Let’s dive into commits in general. The format is like this</p><pre>&lt;type&gt;[optional scope]: &lt;description&gt;<br><br>[optional body]<br><br>[optional footer(s)]</pre><p>The commit contains the following structural elements:</p><p><strong>1. The Header: </strong>&lt;type&gt;[optional scope]: &lt;description&gt;<br>This is the only <strong>mandatory</strong> line. It must be kept under 50–72 characters so it’s readable in git collapsed views:</p><p>The Type (The “What”), tells the system how to treat the change. here is the must-know type:</p><ul><li><strong>feat</strong>: A new feature for the user, not a new feature for the build script.</li><li><strong>fix</strong>: A bug fix for the user, not a fix to a build script.</li><li><strong>docs</strong>: Changes to the documentation.</li><li><strong>style</strong>: Formatting, missing semi-colons, etc.; no production code change.</li><li><strong>refactor</strong>: Refactoring production code, e.g., renaming a variable.</li><li><strong>test</strong>: Adding missing tests, refactoring tests; no production code change.</li><li><strong>chore</strong>: Updating grunt tasks, etc.; no production code change.</li></ul><p>The Scope (The “Where”) is a noun wrapped in parentheses. It identifies the specific module or package being affected. For <strong>Example:</strong></p><ul><li>feat(auth): add OAuth2 support</li><li>fix(ui-button): resolve padding overlap on mobile</li></ul><blockquote><em>Tip:</em> If you find yourself using multiple scopes (e.g., feat(auth,ui)), your commit is likely too large.</blockquote><p>The Description (The “Action”), A summary of the code changes.</p><ul><li><strong>Use the imperative mood:</strong> “change,” not “changed” or “changes.”</li><li><strong>No period at the end:</strong> Keep it clean.</li></ul><p><strong>2. The Body: </strong>[optional body]</p><p>The body is separated from the header by a <strong>single blank line</strong>. This is where you explain the <strong>“Why”</strong>, not the “How.” The code tells you <em>how</em> it was done; the body tells you the <em>motivation</em> and the <em>intent</em>.</p><blockquote><strong><em>Example:</em></strong><em> </em>fix(api): update rate limiting logic</blockquote><blockquote>The previous algorithm used a fixed window which allowed bursts at the edge of the window. Switching to a sliding log to ensure even distribution of requests.</blockquote><p><strong>3. The Footer: </strong>[optional footer(s)]<strong><br></strong>The footer is also separated by a blank line. It is used for metadata and external references.</p><p>Breaking Changes: This is the most “dangerous” footer. It starts with the uppercase string BREAKING CHANGE: followed by a space and a description of what changed.</p><ul><li><strong>Impact:</strong> This triggers a <strong>Major</strong> version bump (1.0.0 $\rightarrow$ 2.0.0).</li></ul><p>Referencing Issues, We use footers to link to Jira tickets, GitHub issues, or Linear tasks.</p><ul><li>Fixes: #123</li><li>Related-to: PROJ-456</li></ul><p>In short, the only mandatory elements are the header. This is the summary of the section above</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3fQXHpL-26lteYiKvZhbfA.png" /></figure><p>Below is an example of conventional commits:<br>1. Commit message with description and breaking change footer</p><pre>feat: allow provided config object to extend other configs<br>BREAKING CHANGE: `extends` key in config file is now used for extending other config files</pre><p>2. Commit message with scope</p><pre>feat(lang): add Polish language</pre><p>3. Commit message with multi-paragraph body and multiple footers</p><pre>fix: prevent racing of requests<br><br>Introduce a request id and a reference to latest request. Dismiss<br>incoming responses other than from latest request.<br><br>Remove timeouts which were used to mitigate the racing issue but are<br>obsolete now.<br><br>Reviewed-by: Z<br>Refs: #123</pre><p>Here are several importance point and benefits of using conventional commits:</p><ul><li><strong>Standardisation:</strong> Creates a consistent “language” across the entire organisation, regardless of the team.</li><li><strong>Automation Power:</strong> Enables automatic generation of CHANGELOG.md files and automated SemVer releases.</li><li><strong>Searchability:</strong> Makes it incredibly easy to filter git history (e.g., git log — grep=”^feat”) to find when specific functionality was introduced.</li><li><strong>Onboarding:</strong> Helps new engineers understand the project’s evolution by reading a clean, structured timeline.</li><li><strong>Discipline:</strong> Forces developers to consider the “why” and “what” of their changes before pushing, resulting in higher-quality code.</li></ul><p><em>Source: https://www.conventionalcommits.org/en/v1.0.0/</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4089f4c1e98" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering the Three Pillars of Observability in K8s: Metrics, Logs, and Traces with the Grafana…]]></title>
            <link>https://medium.com/@PlatformEnthusiast/mastering-the-three-pillars-of-observability-in-k8s-metrics-logs-and-traces-with-the-grafana-1e1a8317bed0?source=rss-b300b45c8552------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e1a8317bed0</guid>
            <category><![CDATA[grafana-tempo]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[grafana-mimir]]></category>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[grafana-loki]]></category>
            <dc:creator><![CDATA[The Platform Enthusiast]]></dc:creator>
            <pubDate>Mon, 23 Mar 2026 07:00:11 GMT</pubDate>
            <atom:updated>2026-03-23T07:00:11.589Z</atom:updated>
            <content:encoded><![CDATA[<h3>Mastering the Three Pillars of Observability in K8s: Metrics, Logs, and Traces with the Grafana Stack</h3><p>Monitoring and observability tools are very important in any IT system. Monitoring tells you when something is wrong. Observability tells you <em>why</em>. It’s the difference between knowing your application is on fire and understanding which exact line of code handed the match to the kindling.</p><p>As systems grow in complexity, monitoring alone stops being enough. You need observability. The Three Pillars of Observability: Observability is typically built around three types of telemetry data, each answering a different kind of question:</p><ul><li><strong>Metrics</strong> — numerical measurements over time. <em>“How much CPU is this service consuming? How many requests per second is it handling? What’s the error rate?”</em></li><li><strong>Logs</strong> — timestamped records of discrete events. <em>“What exactly happened at 14:32:07? Which user triggered this error? What was the full stack trace?”</em></li><li><strong>Traces</strong> — end-to-end records of a request’s journey across services. <em>“Which service in the chain caused this slowdown? Where did this request spend most of its time?”</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WUk2iFVBTM6iOVssXHM_Xg.png" /><figcaption>(Source: <a href="https://grafana.com/docs/tempo/latest/introduction/telemetry/">https://grafana.com/docs/tempo/latest/introduction/telemetry/</a>)</figcaption></figure><p>No single pillar gives you the full picture. The real power comes when you can correlate all three: findings from a metric spike, to the logs that fired at the same time, to the trace that shows exactly which service was the culprit.</p><p>This post is a continuation of my previous article that explains how to build observability: integration for OTel and Loki. In this post, I’ll go into even more advanced topics to integrate Grafana Mimir and Tempo into the stacks built on five battle-tested open-source tools:</p><ul><li><strong>Prometheus</strong> — collects and stores metrics from your services. Think of it as a time-series database that regularly polls your applications and asks, <em>“how are you doing right now?”</em></li><li><strong>OpenTelemetry (OTel)</strong> — a standardized framework for instrumenting your code and shipping telemetry data (metrics, logs, and traces) to any backend. It’s the universal adapter layer — instrument once, send anywhere.</li><li><strong>Grafana Mimir</strong> — a scalable long-term storage backend for metrics. When Prometheus’s local storage isn’t enough — because you need months of retention or you’re running at large scale — Mimir takes over, storing data cheaply in object storage like S3.</li><li><strong>Grafana Loki</strong> — a log aggregation system built for efficiency. Rather than indexing every word in every log line (which gets expensive fast), Loki indexes only the metadata labels attached to your log streams, keeping costs low while keeping queries fast.</li><li><strong>Grafana Tempo</strong> — a distributed tracing backend that stores and retrieves traces without maintaining a heavy index. It sits on top of cheap object storage, accepts trace data from OTel, and lets you look up any trace instantly by its ID.</li><li><strong>Grafana</strong> — the visualization and alerting frontend that ties everything together. It connects to all the backends above, letting you build dashboards, write queries, correlate signals across pillars, and manage alerts — all in one place.</li></ul><p>Before beginning to explain the implementation method, I will explain about the tech stack flow that we will build</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_ITRyMSdPUuDiRAfpA2XFA.png" /><figcaption>Tech Stack flow for Observability Lab</figcaption></figure><p>How above Tech Observability Stack FlowWork (From App to Dashboard)?</p><p>Think of this diagram as a <strong>data pipeline</strong> — telemetry data (metrics, logs, and traces) is born in your applications, travels through several processing stages, gets stored safely, and finally becomes something you can see and act on in Grafana. Let&#39;s walk through it layer by layer.</p><p><strong>Layer 1 — The Applications (Top)<br></strong>This is where everything starts. You have <strong>App 1</strong> and <strong>App 2</strong> — these are your actual running services, whether they’re APIs, web servers, background workers, or anything else. Each app is instrumented to emit telemetry data using two methods:</p><ul><li><strong>OTLP (OpenTelemetry Protocol)</strong> — the standard format for sending traces, metrics, and logs from your app code.</li><li><strong>HostMetrics</strong> — system-level data automatically collected from the underlying host, like CPU usage, memory, disk I/O, and network traffic.</li></ul><p><strong>Layer 2 — The OTel Collector (Ingestion Gateway)<br></strong>All that raw telemetry flows into the <strong>OpenTelemetry Collector</strong>, which acts as the <strong>central hub</strong> of the entire pipeline. It has two jobs:</p><ul><li><strong>OTLP Receiver</strong> — listens and accepts incoming telemetry data from your apps.</li><li><strong>Processors</strong> — cleans, transforms, filters, and enriches the data before forwarding it. For example, it might add environment labels, drop noisy data, or batch events for efficiency.</li></ul><p>After processing, the Collector splits the data into three separate lanes based on type:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uzrdy0TpaZywEAHq6WcKww.png" /><figcaption>Table Data Types and flows</figcaption></figure><p><strong>Layer 3 — Prometheus (Aggregation &amp; Buffering)</strong><br>Metrics don’t go straight to long-term storage — they first pass through <strong>Prometheus</strong>, which serves as a <strong>short-term buffer and local aggregation layer</strong>. Prometheus does two things here:</p><ul><li><strong>Local Scraping</strong> — it can also pull metrics directly from services that expose a /metrics endpoint (the traditional Prometheus way).</li><li><strong>Alerting</strong> — it evaluates alert rules in real time. If your error rate crosses a threshold, Prometheus fires the alert <em>before</em> the data is archived.</li></ul><p>Once Prometheus has done its job, it forwards (Remote Writes) the metrics to Grafana Mimir for long-term storage.</p><p><strong>Layer 4 — Long-Term Storage (S3-Backed)<br></strong>This is where all three telemetry types land for durable, scalable, long-term storage. All three backends store data in object storage (like AWS S3 or similar), which is cheap and reliable:</p><ul><li><strong>Grafana Loki</strong> — stores your <strong>logs</strong>, indexed by labels (not full text) for cost efficiency.</li><li><strong>Grafana Tempo</strong> — stores your <strong>traces</strong>, retrievable by Trace ID without a heavy index.</li><li><strong>Grafana Mimir</strong> — stores your <strong>metrics</strong>, with full PromQL compatibility and multi-month retention.</li></ul><p>Layer 5 — Grafana (Visualization)<br>Finally, Grafana sits at the bottom as the single pane of glass for everything. It queries all three backends simultaneously:</p><ul><li><strong>Loki</strong> for logs → using <strong>LogQL</strong></li><li><strong>Tempo</strong> for traces → using <strong>TraceQL</strong></li><li><strong>Mimir</strong> for metrics → using <strong>PromQL</strong></li><li><strong>Prometheus</strong> directly for real-time metrics and alert state</li></ul><p>This is where you build dashboards, investigate incidents, and correlate signals. The real power here is being able to jump between all three pillars in one place — spot a metric spike, click through to the related logs, then pull up the trace that shows exactly which service caused the problem.</p><h4>The Full Flow in One Sentence</h4><p>Apps emit telemetry → OTel Collector receives, processes, and routes it → Prometheus buffers and alerts on metrics → Loki, Tempo, and Mimir store logs, traces, and metrics long-term → Grafana queries all of them and shows you everything in one place.</p><h4>The Implementation</h4><h4>Prerequisites — What You Need Before We Begin</h4><p>Before we start deploying anything, let’s make sure your environment is properly set up. This section covers everything you need to have installed, configured, and understood before following along with the hands-on labs:</p><ul><li>Kubernetes cluster</li><li>Kubectl and helm installed on the bastion or server that has access to kubernetes cluster</li><li>S3 bucket (used to store data logs, metrics, etc)</li></ul><p>To download the file provided in this article, you may git clone from this repository <a href="https://gitlab.com/platform-project2/grafana-stack.git">Grafana-stack</a>. This lab is intended to be used for a non-production environment, since we will disable several configurations due to limited resources in this environment</p><p><strong>Step-1: install otel<br></strong>Start by applying this otel-server.YAML file to a Kubernetes cluster. We will use this command:</p><pre>helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts<br>helm repo update<br><br>#create namespace for monitoring stacks<br>Kubectl create ns monitoring<br><br>#apply yaml file<br>helm install otel-collector open-telemetry/opentelemetry-collector \<br>-f otel-server-nonprod.yaml \<br>--namespace monitoring</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2PI3EkdTvytNLuvAYcvtjQ.png" /><figcaption>Otel Collector is running</figcaption></figure><p>We will provision Otel in daemonset mode (each Kubernetes worker will have one Otel pod). There are several important configurations from the Otel YAML file as follows:</p><pre># OpenTelemetry Collector Configuration<br>config:<br> receivers:<br>   # OTLP receiver for traces, metrics, and logs<br>   otlp:<br>     protocols:<br>       grpc:<br>         endpoint: 0.0.0.0:4317<br>       http:<br>         endpoint: 0.0.0.0:4318<br>  <br>   # Prometheus receiver to scrape metrics<br>   prometheus:<br>     config:<br>       scrape_configs:<br>         - job_name: &#39;otel-collector&#39;<br>           scrape_interval: 30s<br>           static_configs:<br>             - targets: [&#39;localhost:8888&#39;]<br>        <br>         # Kubernetes service discovery<br>         - job_name: &#39;kubernetes-pods&#39;<br>           scrape_interval: 30s<br>           kubernetes_sd_configs:<br>             - role: pod<br>           relabel_configs:<br>             - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]<br>               action: keep<br>               regex: true<br>             - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]<br>               action: replace<br>               target_label: __metrics_path__<br>               regex: (.+)<br>             - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]<br>               action: replace<br>               regex: ([^:]+)(?::\d+)?;(\d+)<br>               replacement: $1:$2<br>               target_label: __address__<br>  <br>   # Filelog receiver to tail all pod logs from the node<br>   filelog:<br>     include:<br>       - /var/log/pods/*/*/*.log<br>     exclude:<br>       # Exclude collector logs to avoid loop<br>       - /var/log/pods/monitoring_otel-collector-*/*/*.log<br>     start_at: end<br>     include_file_path: true<br>     include_file_name: false<br>     operators:<br>       # Extract metadata from the path<br>       - type: regex_parser<br>         regex: &#39;^/var/log/pods/(?P&lt;namespace&gt;[^_]+)_(?P&lt;pod_name&gt;[^_]+)_(?P&lt;uid&gt;[a-z0-9-]+)/(?P&lt;container_name&gt;[^_]+)/.+$&#39;<br>         parse_from: attributes[&quot;log.file.path&quot;]<br>       # Move extracted labels to resource attributes for Loki and k8sattributes processor<br>       - type: move<br>         from: attributes.namespace<br>         to: resource[&quot;k8s.namespace.name&quot;]<br>       - type: move<br>         from: attributes.pod_name<br>         to: resource[&quot;k8s.pod.name&quot;]<br>       - type: move<br>         from: attributes.uid<br>         to: resource[&quot;k8s.pod.uid&quot;]<br>       - type: move<br>         from: attributes.container_name<br>         to: resource[&quot;k8s.container.name&quot;]<br>       # Parse container logs (JSON for containerd/k3s)<br>       - type: container<br>         id: container-parser<br>       # Additional parser for service-guess CLF logs to extract labels at ingestion time<br>       - type: regex_parser<br>         id: clf-parser<br>         # skip Date Time prefixes if present, then capture CLF fields<br>         regex: &#39;^(?:(?P&lt;date&gt;\d{4}-\d{2}-\d{2}) (?P&lt;time&gt;\d{2}:\d{2}:\d{2})\t)?(?P&lt;ip&gt;[^ ]+) - - \[(?P&lt;ts&gt;[^\]]+)\] &quot;(?P&lt;method&gt;\w+) (?P&lt;path&gt;[^ ]+) (?P&lt;proto&gt;[^&quot;]+)&quot; (?P&lt;status&gt;\d+) (?P&lt;size&gt;[^ ]+) (?P&lt;latency&gt;\d+) &quot;(?P&lt;msg&gt;[^&quot;]+)&quot; \[TraceID: (?P&lt;tid&gt;[^\]]+)\]$&#39;<br>         parse_from: body<br>         # Apply to all services in the apps namespace following the standardized CLF format<br>         if: &#39;resource[&quot;k8s.namespace.name&quot;] == &quot;apps&quot;&#39;<br> <br>   # Host metrics receiver<br>   hostmetrics:<br>     collection_interval: 30s<br>     scrapers:<br>       cpu:<br>       disk:<br>       filesystem:<br>       load:<br>       memory:<br>       network:<br>       paging:<br><br> processors:<br>   # Batch processor for better performance<br>   batch:<br>     timeout: 10s<br>     send_batch_size: 1024<br>  <br>   # Memory limiter to prevent OOM<br>   memory_limiter:<br>     check_interval: 5s<br>     limit_mib: 400<br>     spike_limit_mib: 100<br>  <br>   # Resource detection for Kubernetes<br>   resourcedetection:<br>     detectors: [env, system]<br>     timeout: 5s<br>  <br>   # Enrich logs/metrics with Kubernetes metadata<br>   k8sattributes:<br>     auth_type: &quot;serviceAccount&quot;<br>     passthrough: false<br>     filter:<br>       node_from_env_var: KUBE_NODE_NAME<br>     extract:<br>       metadata:<br>         - k8s.pod.name<br>         - k8s.pod.uid<br>         - k8s.namespace.name<br>         - k8s.node.name<br>         - k8s.container.name<br>       labels:<br>         - tag_name: app<br>           key: app<br>     pod_association:<br>       - sources:<br>           - from: resource_attribute<br>             name: k8s.pod.name<br>           - from: resource_attribute<br>             name: k8s.namespace.name<br>       - sources:<br>           - from: resource_attribute<br>             name: k8s.pod.ip<br>       - sources:<br>           - from: resource_attribute<br>             name: k8s.pod.uid<br>       - sources:<br>           - from: connection<br><br><br>   # Resource processor to create user-friendly labels for Loki<br>   resource:<br>     attributes:<br>       - key: namespace<br>         from_attribute: k8s.namespace.name<br>         action: insert<br>       - key: pod<br>         from_attribute: k8s.pod.name<br>         action: insert<br>       - key: container<br>         from_attribute: k8s.container.name<br>         action: insert<br>       # Promote extracted CLF fields to indexed labels<br>       - key: status<br>         from_attribute: status<br>         action: insert<br>       - key: method<br>         from_attribute: method<br>         action: insert<br>       - key: path<br>         from_attribute: path<br>         action: insert<br><br>   # Attributes processor for adding metadata<br>   attributes:<br>     actions:<br>       - key: environment<br>         value: non-production<br>         action: insert<br> exporters:<br>   # Prometheus exporter for metrics<br>   prometheus:<br>     endpoint: &quot;0.0.0.0:8889&quot;<br>     namespace: otel<br>     const_labels:<br>       environment: nonprod<br>  <br>   # Prometheus Remote Write (if using Prometheus server)<br>   prometheusremotewrite:<br>     endpoint: http://prometheus-server.monitoring.svc.cluster.local/api/v1/write<br>     tls:<br>       insecure: true<br>     # Use the specific queue for this exporter<br>     remote_write_queue:<br>       enabled: true<br>       queue_size: 10000  # Increase this for cross-region<br>       num_consumers: 5<br>     retry_on_failure:<br>       enabled: true<br>       initial_interval: 5s<br>       max_elapsed_time: 300s<br>  <br>   # Debug exporter for monitoring data flow (replaces deprecated logging exporter)<br>   debug:<br>     verbosity: normal<br>     sampling_initial: 5<br>     sampling_thereafter: 200<br>  <br>   # Loki exporter for logs<br>   otlphttp/loki:<br>     endpoint: http://loki-gateway.monitoring.svc.cluster.local/otlp<br>     tls:<br>       insecure: true<br>     # Optimization for cross-region or network latency<br>     sending_queue:<br>       enabled: true<br>       queue_size: 5000<br>     retry_on_failure:<br>       enabled: true<br>       initial_interval: 5s<br>       max_elapsed_time: 300s<br>  <br>   # Tempo exporter for traces<br>   otlp/tempo:<br>     endpoint: tempo-server.monitoring.svc.cluster.local:4317<br>     tls:<br>       insecure: true<br><br> service:<br>   pipelines:<br>     # Traces pipeline<br>     traces:<br>       receivers: [otlp]<br>       processors: [memory_limiter, resourcedetection, attributes, batch]<br>       exporters: [otlp/tempo, debug]<br>    <br>     # Metrics pipeline<br>     metrics:<br>       receivers: [otlp, prometheus, hostmetrics]<br>       processors: [memory_limiter, resourcedetection, attributes, batch]<br>       exporters: [prometheus, prometheusremotewrite, debug]<br>    <br>     # Logs pipeline<br>     logs:<br>       receivers: [otlp, filelog]<br>       processors: [memory_limiter, k8sattributes, resource, resourcedetection, attributes, batch]<br>       exporters: [otlphttp/loki, debug]</pre><p>Snippet of yaml file above explain about the core functionality config of Otel:</p><ul><li>The first config, receivers: the tools the collector uses to get data. otlp: Listens for data that your “modern” apps send voluntarily. Prometheus: This is the “Collector Bot.” It goes out every 30 seconds, finds all your apps, and “harvests” their numbers (like CPU or request counts). filelog: This is a “File Reader.” It looks at the actual log files saved on the server’s hard drive and reads them line by line. Hostmetrics: This measures the “Vitals” of the server itself for example it show how much RAM is left, how hard the disks are working, etc.</li><li>The second config, Processors: The Sorting Factory. The processors act as the sorting factory where data is cleaned and organized before storage. To ensure high efficiency, the <strong>batch</strong> processor groups data into “boxes” rather than sending individual lines, reducing overhead. For system stability, the <strong>memory_limiter</strong> serves as a safety guard, dropping data if RAM usage spikes to prevent a full server crash. Meanwhile, the <strong>k8sattributes</strong> processor functions as a label maker, tagging data with metadata like app names or namespaces, while the resource processor acts as a translator to rename labels into a consistent format that Grafana prefers.</li><li>The third config, Exporters: The Delivery Trucks. Once the data is refined, exporters act as delivery trucks that transport it to its long-term destination. The prometheusremotewrite exporter handles the delivery of numerical metrics to the Mimir server, while otlphttp/loki and otlp/tempo deliver logs and application traces to Loki and Tempo, respectively. Additionally, the debug exporter prints data directly to the terminal, allowing you to verify that the pipeline is working correctly before it reaches storage.</li><li>The last config, Service Pipelines: The Assembly Lines. The service pipelines represent the assembly lines that plug every component together into a cohesive workflow. This section defines three distinct tracks to manage different data types: the Traces Line labels and routes application behavior to Tempo; the Metrics Line collects and cleans numerical sensor data for Mimir; and the Logs Line reads raw files, attaches Kubernetes context, and sends them to Loki. By defining these paths, the system ensures that every piece of telemetry follows the correct processing and delivery route.</li></ul><p><strong>Step-2: Install Prometheus<br></strong>Start by applying this otel-server.YAML file to a Kubernetes cluster. We will use this command:</p><pre>helm repo add prometheus-community https://prometheus-community.github.io/helm-charts<br>helm repo update<br><br># apply yaml file<br>helm install prometheus prometheus-community/prometheus \<br> -f prometheus-server-agent.yaml \<br> --namespace monitoring</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NusbmGHX80dtKvnou4IC7g.png" /><figcaption>Promtheus pod is running</figcaption></figure><p>We will provision Prometheus in agent mode (the pod will not enable persistent data saving). This mode is a specialized, lightweight deployment mode for Prometheus optimized for service discovery, scraping, and forwarding metrics via Remote Write to a central storage system. It eliminates local storage (TSDB), alerting, and query capabilities to reduce resource usage, providing an efficient alternative for edge scenarios or long-term storage workflows. We use this mode due to mimir existence that will handling saving data mechanism to S3.</p><p><strong>Step-3: Install Mimir<br></strong>Start by applying this mimir-server.YAML file to a Kubernetes cluster. We will use this command:</p><pre># apply yaml file<br>helm install mimir grafana/mimir-distributed \<br>-f mimir-server-nonprod.yaml \<br> --namespace monitoring \<br> --create-namespace</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WrNAgJeXeQWH_tG8s3cRZg.png" /><figcaption>Mimir pod is running</figcaption></figure><p>We have provisioned mimir core components, each component is useful to keep metric data to s3 object: we will explain detail each config in mimir yaml file as follows:</p><p><strong>1. Name Overrides</strong></p><pre>nameOverride: &quot;mimir-nonprod&quot;<br>fullnameOverride: &quot;mimir-nonprod&quot;</pre><p>Sets the name of all Kubernetes resources to mimir-nonprod. Without this, Helm generates names with dots (e.g. mimir.nonprod) which causes DNS label warnings in Kubernetes, since dots are not valid in resource names.</p><p><strong>2. Storage Configuration<br>2a. Common Storage (S3)</strong></p><pre>common:<br>  storage:<br> backend: s3<br> s3:<br>   endpoint: s3.amazonaws.com<br>   region: us-east-1<br>   bucket_name: &lt;empty&gt;<br>   access_key_id: &lt;empty&gt;<br>   secret_access_key: &lt;empty&gt;</pre><p>Sets AWS S3 as the default storage backend for all Mimir components. All components (blocks, ruler, alertmanager) inherit this setting unless overridden individually.</p><blockquote><strong>⚠️ </strong>bucket_name, access_key_id, and secret_access_key are empty. S3 access will fail at runtime unless credentials are injected via environment variables or Kubernetes secrets.</blockquote><p><strong>2b. Blocks Storage</strong></p><pre>blocks_storage:<br>  backend: s3<br>  s3:<br> bucket_name: test-bucket-10122225<br>  tsdb:<br> dir: /data/mimir/tsdb</pre><p>Metric blocks (actual time-series data) are stored in S3 bucket. Locally, TSDB uses /data/mimir/tsdb as a temporary write buffer before flushing to S3. The same bucket is used for Ruler and alertmanager storage — acceptable for non-prod, but in production, you should separate these into different buckets.</p><p><strong>3. Ingest Storage (Kafka)</strong></p><pre>ingest_storage:<br>  enabled: true</pre><p>Enables the Kafka-based ingest path. Instead of Prometheus writing metrics directly to ingesters, data flows through Kafka first:<br>Prometheus → Distributor → Kafka → Ingester → S3</p><p>This adds a buffer layer, preventing metric loss if ingesters temporarily go down. Since this is enabled, Kafka must also be enabled in the Helm values — disabling Kafka while this is on will break the ingest path.</p><p><strong>4. Ring / Memberlist Configuration</strong></p><pre>distributor:<br>  ring:<br> kvstore:<br>   store: memberlist<br> <br>ingester:<br>  ring:<br> kvstore:<br>   store: memberlist<br> replication_factor: 1</pre><p>Memberlist is a gossip protocol — Mimir pods discover and communicate with each other automatically without needing an external key-value store like etcd or Consul. This simplifies the setup significantly.</p><p>replication_factor: 1 means metrics are written to only 1 ingester — no redundancy.</p><blockquote>replication_factor: 1 means if the ingester crashes, recent metrics that haven’t been flushed to S3 yet will be lost. Acceptable for non-prod. In production, always use replication_factor: 3.</blockquote><p><strong>Step-4: Install Loki<br></strong>Start by applying this loki-server-nonprod.YAML file to a Kubernetes cluster. We will use this command:</p><pre># apply yaml file<br>helm install loki grafana/loki \<br>-f loki-server.yaml \<br>--namespace monitoring</pre><p>Below is the detail explanation of each config from loki-server-nonprod.yaml</p><p><strong>Deployment Mode<br></strong>deploymentMode: SingleBinary — Runs all Loki components (ingester, querier, compactor, ruler) inside a single process and a single pod. This is the simplest way to run Loki.</p><p><strong>Storage Configuration</strong><br>Loki uses AWS S3 as its object storage backend. All log chunks, ruler data, and admin data are stored there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MdgH_gm2DbdpFMDFYJolig.png" /><figcaption>Table configuration S3</figcaption></figure><blockquote>All three buckets can point to the same S3 bucket. This is fine for non-prod but in production you should separate them into dedicated buckets for clarity and access control.</blockquote><p><strong>Schema Configuration</strong></p><p>from: 2024–01–01 | store: tsdb | schema: v13 | object_store: s3</p><p>Defines how Loki organizes and indexes log data:</p><ul><li>from: 2024–01–01 — This schema applies to all logs from January 1st 2024 onward</li><li>store: tsdb — Uses TSDB (time-series database format) for the index. This is the modern and recommended index format</li><li>object_store: s3 — Log chunks are stored in S3</li><li>schema: v13 — Uses schema version 13, which is the latest stable Loki schema</li><li>prefix: loki_index_ — All index files in S3 are prefixed with loki_index_</li><li>period: 24h — A new index table is created every 24 hours</li></ul><p><strong>Limits Configuration<br></strong>Controls how much data Loki will accept and how heavy queries can get. These are global defaults applied to all log streams.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vBB6DwH0U3BaNYJvyy1y-g.png" /><figcaption>Table Configurations Limit</figcaption></figure><blockquote>Retention only works when compactor.retention_enabled is set to true (which it is in this config). Without that, the retention_period setting has no effect.</blockquote><p><strong>Backend, Read, and Write Replicas</strong></p><p>backend.replicas: 0 | read.replicas: 0 | write.replicas: 0</p><p>These are components used in Loki’s distributed/microservices deployment mode. Since this config uses SingleBinary mode, all these roles are handled by the single binary pod. Setting them to 0 prevents Helm from spinning up unnecessary extra pods.</p><p>Overall Summary</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qF3B18s5TAzZZ6nWVzzx2Q.png" /><figcaption>Summary configuration of loki</figcaption></figure><p><strong>Step-5: Install Tempo<br></strong>Apply tempo-server-nonprod.yaml to provision in Kubernetes:</p><pre>helm install tempo grafana/tempo \<br>-f tempo-server-nonprod.yaml \<br>--namespace monitoring</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q6gnBQIVE7dI9-xS930TjQ.png" /><figcaption>Pod Tempo is running</figcaption></figure><p>Below is the detail explanation of each config in tempo-server-nonprod.yaml:</p><p><strong>Metrics Generator </strong><br>The Metrics Generator is the most significant addition in this config. It is a Tempo component that reads incoming traces and automatically produces Prometheus metrics from them — without you having to instrument your code separately.</p><p><strong>Enabled and Remote Write</strong></p><pre>enabled: true<br>remoteWriteUrl: &quot;http://prometheus-server.monitoring.svc.cluster.local/api/v1/write&quot;</pre><p>enabled: true<br>Turns on the Metrics Generator component inside the Tempo pod. Without this, no metrics are produced from traces.</p><p>remoteWriteUrl<br>The URL where generated metrics are pushed to. This points to your Prometheus server running inside the same cluster in the monitoring namespace. Tempo acts like a Prometheus remote write client. it pushes metrics directly into Prometheus at regular intervals.</p><p><strong>Ingester Configuration<br></strong>max_block_duration: 5m</p><p>The ingester buffers incoming traces in memory and periodically flushes them as immutable blocks to S3. This setting controls how long a block stays open before being flushed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lqRccJl9Q-WNOpDNaLJ_6A.png" /><figcaption>Table Ingester Config</figcaption></figure><p><strong>Querier Configuration</strong></p><p>max_concurrent_queries: 5<br>Limits how many trace queries (from Grafana or direct API calls) can run at the same time inside Tempo. Queries beyond this limit are queued and wait for a slot.</p><ul><li>5 is appropriate for non-prod with a small number of users</li><li>Increase to 20–50 in production depending on Grafana user count and available memory</li><li>Each concurrent query consumes memory — keeping this low prevents OOM on small nodes</li></ul><p><strong>Persistence</strong></p><p>enabled: true | size: 1Gi | storageClass: “”</p><p>Allocates a PersistentVolumeClaim for Tempo’s local storage. This is used for:</p><ul><li>WAL (Write-Ahead Log) — protects buffered traces from loss on pod restart</li><li>local-blocks processor data — the local-blocks processor stores a short window of raw trace data on disk for TraceQL metric queries</li></ul><h4>Step-6: Install Grafana</h4><p>This is the final installation for this stack. Start by applying grafana-server.yaml. Let’s use these commands:</p><pre># apply yaml file<br>helm install grafana grafana-community/grafana \<br>-f grafana-server.yaml \<br>--namespace monitoring</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ehiVnCddcozXrDNP8vRtnw.png" /></figure><p>After that, try to open grafana ui:<br>http://[public ip or domain]:30085/login</p><p>By default, the username and password are as follows:<br>1. Username: admin<br>2. Password: use this command to get the password by decoding the secret value -&gt; kubectl get secret — namespace monitoring grafana -o jsonpath=”{.data.admin-password}” | base64 — decode ; echo</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*28cpmU1xw9TG-x3Op9mCkQ.png" /><figcaption>Grafana UI</figcaption></figure><p>Based on Grafana Server.yaml there are several highlighted configurations that we use to integrate each service in our stack:</p><pre>datasources:<br> datasources.yaml:<br>   apiVersion: 1<br>   datasources:<br>   - name: Prometheus<br>     type: prometheus<br>     url: http://prometheus-server.monitoring.svc.cluster.local       <br>     access: proxy<br>     isDefault: true<br>   - name: Loki<br>     type: loki<br>     url: http://loki-gateway.monitoring.svc.cluster.local<br>     access: proxy</pre><p>Since we have Prometheus and loki as datasource we will need to add these two configurations in yaml. access: proxy: This is a security and networking setting. It means that when you view a dashboard in your browser, your browser sends the request to Grafana, and then Grafana “proxies” that request to Prometheus/Loki. This is better because your Prometheus/Loki services don’t need to be exposed to the outside world; they only need to be reachable by the Grafana pod..isDefault: true: in prometheus it makes Prometheus the default choice whenever you create a new dashboard or panel.</p><p>After completing the installation of each component, continue access the grafana dashboard. By default in our manifest two dashboard is imported</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OCJIyxlwdySxc25-aNK3Sw.png" /><figcaption>Dashboard Grafana</figcaption></figure><p>To check whether Kubernetes is monitored by Prometheus, choose “Node Exporter Full”. Monitor all important node metric is already there, such as CPU, memory, disk, etc. This functionality shows that prometheus have been successfully scraped data from kubernetes node. Continue to check data source, make sure there are 3 datasource configured: Loki, prometheus and Tempo (previously we have configured these three datasource in Grafana manifest).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wvUxlcXAWVpgoyhLuFJTLA.png" /><figcaption>Data Source Grafana</figcaption></figure><p>To check container metric utilization we can create custom dashboard, import dashboard from internet or check menu drilldown choose metrics and filter container that you want to monitor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xei9YFs_-80jc8T8mNbNNw.png" /><figcaption>Metrics Grafana</figcaption></figure><p>To check loki functionality, we can click explore and choose datasource loki.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zuNjcp1fwRuFaG1u2SP8OA.png" /><figcaption>Loki Grafana</figcaption></figure><p>If you want to check the tempo copy trace_id from loki and go to explore choose tempo as datasource and hit blue refresh button the trace will be shown.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UAGaNcBl8O3m6Uiq-BP2Cw.png" /><figcaption>Grafana Tempo</figcaption></figure><p>The last to check whether Mimir gets data from Prometheus, you can check from s3 storage. Since we have not configured any tenant id so mimir will use the path “fake/” to store our data. If the path is not empty, then the mimir configuration is correct. Then you can also check the metric in the menu drilldown to make sure data is ingested to Mimir.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e1a8317bed0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scaling Your CI/CD: Setting Up GitLab Runner with Kubernetes Executor]]></title>
            <link>https://medium.com/@PlatformEnthusiast/scaling-your-ci-cd-setting-up-gitlab-runner-with-kubernetes-executor-7dcb9ca66385?source=rss-b300b45c8552------2</link>
            <guid isPermaLink="false">https://medium.com/p/7dcb9ca66385</guid>
            <category><![CDATA[gitlab]]></category>
            <category><![CDATA[cicd-pipeline]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[The Platform Enthusiast]]></dc:creator>
            <pubDate>Sun, 22 Feb 2026 17:03:20 GMT</pubDate>
            <atom:updated>2026-02-22T17:03:20.290Z</atom:updated>
            <content:encoded><![CDATA[<p>GitLab is a comprehensive, web-based DevSecOps platform that unifies the entire software development lifecycle into a single application. Originally launched in 2011 as an open-source project, it has evolved from a simple Git repository manager into a robust ecosystem covering everything from project planning and source code management to security, monitoring, and automated CI/CD.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JcDqgjcWd7GnmZxzL5Bfaw.png" /><figcaption>Gitlab &amp; Kubernetes (AI Generated)</figcaption></figure><p>GitLab offers features that no other DevSecOps platform solution can matc,h such as <strong>Unified Toolchain</strong>: By housing SCM, CI/CD, and project management in one place, teams reduce “context switching” and eliminate the need to maintain multiple disparate tools, <strong>Scalable Architecture</strong>: GitLab uses “Runners” — lightweight agents that execute jobs. These can be hosted on-premises, in the cloud, or on Kubernetes clusters, allowing your infrastructure to scale horizontally as your workload grows.</p><p>GitLab also remains cloud-agnostic; you can install it on whatever platform your team has. As your organization’s development velocity increases, traditional static CI/CD runners in GitLab often become a bottleneck, leading to long job queues and inefficient resource usage. in this article, we will explain that transitioning to the <a href="https://docs.gitlab.com/runner/executors/kubernetes/">GitLab Runner Kubernetes executor</a> transforms your pipeline into an elastic, auto-scaling powerhouse that spins up isolated pods on demand for every job and terminates them upon completion.</p><p>This approach not only ensures a clean, consistent environment for every build but also significantly optimizes infrastructure costs by utilizing cluster resources only when they are truly needed. In this guide, we will walk through the step-by-step process of setting up and configuring GitLab Runner with the Kubernetes executor to build a highly available and scalable CI/CD infrastructure that grows alongside your team.</p><p>Before following this guide, ensure you have a Kubernetes cluster running, kubectl, helm installed and GitLab access (you may use a self-hosted GitLab, although in this guide, we will use the free GitLab Cloud). First create a namespace gitlab and add helm repository using Helm with this command</p><pre># Create namespace GitLab<br>kubectl create namespace gitlab<br><br># Verify kubernetes namespace creation<br>kubectl get namespace gitlab<br><br># Add GitLab Helm repository<br>helm repo add gitlab https://charts.gitlab.io<br>helm repo update<br></pre><p>Continue creating gitlab-runner.yaml and rbac.yaml</p><pre># rbac.yaml<br># consits of service account creation, cluster role and cluster role binding<br><br>apiVersion: v1<br>kind: ServiceAccount<br>metadata:<br>  name: gitlab-runner-sa<br>  namespace: gitlab<br>---<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: ClusterRole<br>metadata:<br>  name: gitlab-runner-cluster-role<br>rules:<br>  - apiGroups: [&quot;&quot;, &quot;apps&quot;, &quot;networking.k8s.io&quot;, &quot;extensions&quot;, &quot;traefik.io&quot;, &quot;autoscaling&quot;, &quot;batch&quot;, &quot;policy&quot;]<br>    resources: [&quot;*&quot;]<br>    verbs: [&quot;*&quot;]<br>---<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: ClusterRoleBinding<br>metadata:<br>  name: gitlab-runner-cluster-role-binding<br>subjects:<br>  - kind: ServiceAccount<br>    name: gitlab-runner-sa<br>    namespace: gitlab<br>roleRef:<br>  kind: ClusterRole<br>  name: gitlab-runner-cluster-role<br>  apiGroup: rbac.authorization.k8s.io<br><br></pre><p>let’s explain about files above start with rbac.yaml, Kubernetes environment, <strong>RBAC (Role-Based Access Control)</strong> is the “ID and Badge” system that tells the cluster what specific actions a Pod is allowed to perform. For <strong>GitLab Runner</strong>, RBAC is required because of its unique architecture.</p><ol><li>The “Manager-Worker” Architecture<br>The Manager Pod: This is the persistent pod you installed via Helm. It listens to GitLab for new jobs. The Build Pods: When a job arrives, the Manager Pod instructs the Kubernetes API to create a brand-new Pod for that specific job. To create, monitor, and delete these “Worker” pods, the Manager Pod needs the list, create, get, and delete permissions on pods. Without RBAC, the Kubernetes API would simply say &quot;403 Forbidden&quot; to the Runner Manager.</li><li>Executing Commands inside Pods <br>The Runner needs to stream your script: instructions (like npm install or docker build) into the newly created build pod. This requires the pods/exec permission.</li><li>Deploying your Application (Your specific case)<br>Because the Runner uses a ServiceAccount (gitlab-runner-sa), the job pod inherits the permissions of that account. Your rbac.yaml gives the runner ClusterRole permissions with * on resources like apps, networking.k8s.io, and traefik.io. This is why Gitlab CICD can successfully create <strong>Deployments</strong>, <strong>Services</strong>, and <strong>Ingresses</strong>. If you didn’t have RBAC, your kubectl apply command would fail with a permissions error.</li></ol><p>In summary rbac is used to create this:</p><ul><li><strong>ServiceAccount</strong>: The identity for the Runner.</li><li><strong>ClusterRole</strong>: The list of “Keys” (permissions to access everything in apps, networking, batch, etc.).</li><li><strong>ClusterRoleBinding</strong>: The “Keychain” that attaches those “Keys” to the Runner’s identity.</li></ul><blockquote><strong>Security Pro-tip:</strong> current rbac.yaml above is very powerful (it has * permissions). While this makes everything &quot;just work,&quot; in a production environment with many teams, you might eventually want to restrict the runner to only be able to manage resources inside specific namespaces.</blockquote><p>Here is the refined version of the best practice config. In a professional or production environment, you should avoid &quot;*&quot; using resources and verbs. If a malicious script ever gets into your CI, it could delete your entire cluster logic, but with more control:</p><pre>rules:<br>  # 1. Permissions for the Runner to manage its own Build Pods<br>  - apiGroups: [&quot;&quot;]<br>    resources: [&quot;pods&quot;, &quot;pods/exec&quot;, &quot;pods/attach&quot;, &quot;pods/log&quot;, &quot;pods/status&quot;, &quot;configmaps&quot;, &quot;secrets&quot;, &quot;services&quot;]<br>    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;, &quot;create&quot;, &quot;update&quot;, &quot;patch&quot;, &quot;delete&quot;]<br><br>  # 2. Permissions to manage Deployments and HPAs<br>  - apiGroups: [&quot;apps&quot;]<br>    resources: [&quot;deployments&quot;, &quot;statefulsets&quot;, &quot;replicasets&quot;]<br>    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;, &quot;create&quot;, &quot;update&quot;, &quot;patch&quot;, &quot;delete&quot;]<br>    <br>  - apiGroups: [&quot;autoscaling&quot;]<br>    resources: [&quot;horizontalpodautoscalers&quot;]<br>    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;, &quot;create&quot;, &quot;update&quot;, &quot;patch&quot;, &quot;delete&quot;]<br><br>  # 3. Permissions for Networking (Ingress) eg used traefik api gw<br>  - apiGroups: [&quot;networking.k8s.io&quot;, &quot;traefik.io&quot;]<br>    resources: [&quot;ingresses&quot;, &quot;ingressroutes&quot;, &quot;middleware&quot;]<br>    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;, &quot;create&quot;, &quot;update&quot;, &quot;patch&quot;, &quot;delete&quot;]</pre><p>Change the rules snippet from RBAC with the code above key Differences:</p><ol><li><strong>Verbs</strong>: Instead of &quot;*&quot; (which includes things like &quot;proxy&quot;), we list specific actions (get, create, delete, etc.).</li><li><strong>Resources</strong>: We explicitly list pods, deployments, and ingresses.</li><li><strong>Security</strong>: This prevents the runner from touching things it doesn’t need (like nodes, persistentvolumes, or namespaces).</li></ol><p>Copy and paste, and do helm install and kubectl apply rbac.yaml</p><pre># apply rbac.yaml<br>kubectl apply -f rbac.yaml</pre><p>After that register runner to gitlab, in this guide we will use free gitlab cloud. Click setting -&gt; CI/CD settings choose runner -&gt; click create project runner. Fill up the detail following image below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XBJlWHfNY2ibjA6GToWLtw.png" /></figure><p>Tags: kubernetes, docker<br>Runner description: k8s runner</p><p>Click Create Runner then copy the token value</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cp8IXTeLwhKbUUWehN9Nzg.png" /></figure><p>paste it into runnerToken in gitlab-runner.yaml below</p><pre># gitlab-runner.yaml<br>gitlabUrl: https://gitlab.com/  # use gitlab cloud url or your self-hosted URL<br>runnerToken: &quot;&quot;  # ⚠️ Replace this with token from gitlab!<br>image:<br>  registry: docker.io<br>  image: gitlab/gitlab-runner<br>  tag: v18.3.0<br>hpa:<br>  enabled: true<br>  minReplicas: 2<br>  maxReplicas: 10<br>  metrics:<br>    - type: Resource<br>      resource:<br>        name: cpu<br>        target:<br>          type: Utilization<br>          averageUtilization: 80<br>concurrent: 10         <br>resources:<br>  limits:<br>    memory: 256Mi<br>    cpu: 200m<br>  requests:<br>    memory: 128Mi<br>    cpu: 100m<br>rbac:<br>  create: false<br>  serviceAccountName: gitlab-runner-sa<br>securityContext:<br>  runAsUser: 999<br>  fsGroup: 999<br><br>envVars:<br>  - name: HOME<br>    value: /home/gitlab-runner<br>extraVolumes:<br>  - name: runner-home<br>    emptyDir: {}<br>extraVolumeMounts:<br>  - name: runner-home<br>    mountPath: /home/gitlab-runner<br>  - name: runner-home<br>    mountPath: /.gitlab-runner<br><br># Kubernetes executor config<br>runners:<br>  config: |<br>    [[runners]]<br>      [runners.kubernetes]<br>        namespace = &quot;{{.Release.Namespace}}&quot;<br>        service_account = &quot;gitlab-runner-sa&quot;<br>        helper_image = &quot;registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:arm64-v18.3.0&quot;<br>        image = &quot;alpine:latest&quot;<br><br>        # Resource limits (per job pod)<br>        cpu_limit = &quot;1&quot;<br>        memory_limit = &quot;1Gi&quot;<br>        cpu_request = &quot;100m&quot;<br>        memory_request = &quot;256Mi&quot;<br>        # Service containers (e.g., DinD)<br>        service_cpu_limit = &quot;1&quot;<br>        service_cpu_request = &quot;100m&quot;<br>        service_memory_limit = &quot;1Gi&quot;<br>        service_memory_request = &quot;256Mi&quot;<br>        # Enable privileged mode (needed for Docker-in-Docker)<br>        privileged = true</pre><p>Key takeaway:</p><ol><li>Manager Pod starts up using the image and resources defined.</li><li>It uses the runnerToken to tell GitLab &quot;I am ready.&quot;</li><li>When a job appears, it uses its gitlab-runner-sa permissions to create a Worker Pod.</li><li>The <strong>concurrent: 10</strong>: This is the global limit. This single runner manager will never run more than 10 job pods at the exact same time. The single runner equal to single pod and above you will configure a maximum of 10 replicas, so the maximum concurrent GitLab-Runners can handle is 50</li><li>The Worker Pod uses the runners.config (CPU/Memory) settings to request space on your nodes.</li><li>Runners Config: The “Kubernetes Executor” This is the most critical part. It configures the Worker Pods.<strong> </strong><strong>helper_image</strong>: Every job pod has a small &quot;helper&quot; container that handles Git cloning and artifact uploading. Since you are likely on an ARM-based server (like Graviton or Apple Silicon), you&#39;ve specified the arm64 version. <strong>privileged = true</strong>: <strong>Crucial for Docker-in-Docker.</strong> This gives the job pod host-level permissions, which is required if you want to run docker build inside an alpine job.</li><li>Once the code in your .gitlab-ci.yml finishes, the Manager Pod deletes the Worker Pod.</li></ol><p>After that run this command</p><pre># install gitlab runner<br>helm install gitlab-runner \<br>  --namespace gitlab \<br>  --version 0.80.0 \<br>  -f gitlab-runner.yaml \<br>  gitlab/gitlab-runner<br><br># verify pods is running<br>kubectl get pods -n gitlab</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*8ZD-Mf17YkVt3ZB3w52MNg.png" /><figcaption>sample kubectl get pods from running gitlab-runner pod</figcaption></figure><p>Congratulations, you have provision gitlab runner in Kubernetes. If you want to test this runner copy and paste this <strong>gitlab-ci.yaml </strong>to your repository</p><pre>hello-kubernetes:<br>  stage: test<br>  tags:<br>    - kubernetes<br>  image: alpine:latest<br>  script: |<br>    echo &quot;Hello from Kubernetes Runner!&quot;<br>    echo &quot;Running on pod: $(hostname)&quot;<br>    echo &quot;Date: $(date)&quot;<br>    echo &quot;Testing network:&quot;<br>    ping -c 3 google.com || echo &quot;No internet (might be expected)&quot;</pre><p>and check in pipeline gitlab the output should be success</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n_RmjVt0gTmr6SrkdqnZYQ.png" /><figcaption>test gitlab runner with tag kubernetes</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7dcb9ca66385" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Architecting Observability: A Complete Guide to OTel and Loki Integration]]></title>
            <link>https://medium.com/@PlatformEnthusiast/architecting-observability-a-complete-guide-to-otel-and-loki-integration-23b77109c44f?source=rss-b300b45c8552------2</link>
            <guid isPermaLink="false">https://medium.com/p/23b77109c44f</guid>
            <category><![CDATA[otel]]></category>
            <category><![CDATA[site-reliability-engineer]]></category>
            <category><![CDATA[grafana-loki]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[The Platform Enthusiast]]></dc:creator>
            <pubDate>Mon, 16 Feb 2026 18:56:47 GMT</pubDate>
            <atom:updated>2026-02-18T06:41:41.970Z</atom:updated>
            <content:encoded><![CDATA[<p>There are numerous tools available for implementing observability in Kubernetes. In modern distributed systems, achieving comprehensive observability requires correlating traces, metrics, and logs to understand application behavior and troubleshoot issues effectively. One tool which essentials act as a collector and open source is Otel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z56pdyjWXyhlmPh8EDNzRg.png" /><figcaption>AI generated images OTEL &amp; Loki</figcaption></figure><p>OpenTelemetry (OTel) has emerged as the industry-standard instrumentation framework for collecting distributed traces and metrics, while Grafana Loki provides a horizontally scalable, cost-efficient log aggregation system designed specifically for cloud-native environments. This guide explores how to architect a unified observability stack by integrating OpenTelemetry’s rich telemetry collection capabilities with Loki’s efficient log storage and querying.</p><p>Before starting this guide, make sure you have kubernetes cluster running and cluster access to pull images fromthe internet. We will install OTel &amp; Grafana Loki. You will begin by installing OTel in kubernetes. First create otel-server.yaml.</p><pre>mode: daemonset<br>image:<br>  repository: otel/opentelemetry-collector-contrib<br>  #tag: 0.145.0 (Optional:specify a version, or it defaults to the latest verified by the chart)<br>resources:<br>  limits:<br>    cpu: 500m<br>    memory: 512Mi<br>  requests:<br>    cpu: 200m<br>    memory: 256Mi<br>service:<br>  type: NodePort<br><br>ports:<br>  otlp:<br>    enabled: true<br>    containerPort: 4317<br>    servicePort: 4317<br>    nodePort: 30317<br>    protocol: TCP<br>    appProtocol: grpc<br>  otlp-http:<br>    enabled: true<br>    containerPort: 4318<br>    servicePort: 4318<br>    nodePort: 30318<br>    protocol: TCP<br>  metrics:<br>    enabled: true<br>    containerPort: 8888<br>    servicePort: 8888<br>    protocol: TCP<br><br>#OpenTelemetry Collector Configuration<br>config:<br>  receivers:<br>    otlp:<br>      protocols:<br>        grpc:<br>          endpoint: 0.0.0.0:4317<br>        http:<br>          endpoint: 0.0.0.0:4318<br>  processors:<br>    # Batch processor for better performance<br>    batch:<br>      timeout: 10s<br>      send_batch_size: 1024<br>    # Memory limiter to prevent OOM<br>    memory_limiter:<br>      check_interval: 5s<br>      limit_mib: 400<br>      spike_limit_mib: 100<br>    # Resource detection for Kubernetes<br>    resourcedetection:<br>      detectors: [env, system]<br>      timeout: 5s<br>    # Enrich logs/metrics with Kubernetes metadata<br>    k8sattributes:<br>      auth_type: &quot;serviceAccount&quot;<br>      passthrough: false<br>      filter:<br>        node_from_env_var: KUBE_NODE_NAME<br>      extract:<br>        metadata:<br>          - k8s.pod.name <br>          - k8s.pod.uid<br>          - k8s.namespace.name<br>          - k8s.node.name<br>          - k8s.container.name<br>        labels:<br>          - tag_name: app<br>            key: app<br>      pod_association:<br>        - sources:<br>            - from: resource_attribute<br>               name: k8s.pod.name<br>            - from: resource_attribute<br>              name: k8s.namespace.name<br>        - sources:<br>            - from: resource_attribute<br>              name: k8s.pod.ip<br>        - sources:<br>            - from: resource_attribute<br>              name: k8s.pod.uid<br>        - sources:<br>            - from: connection<br>    # Resource processor to create user-friendly labels for Loki<br>    resource:<br>      attributes:<br>        - key: namespace<br>          from_attribute: k8s.namespace.name<br>          action: insert<br>        - key: pod<br>          from_attribute: k8s.pod.name<br>          action: insert<br>        - key: container<br>          from_attribute: k8s.container.name<br>          action: insert <br>    # Attributes processor for adding metadata<br>    attributes:<br>      actions:<br>        - key: environment<br>          value: non-production<br>          action: insert<br>  exporters:<br>    # Prometheus exporter for metrics<br>    prometheus:<br>      endpoint: &quot;0.0.0.0:8889&quot;<br>      namespace: otel<br>      const_labels:<br>        environment: nonprod<br>    # Prometheus Remote Write (if using Prometheus server)<br>    prometheusremotewrite:<br>      endpoint: http://prometheus-server.monitoring.svc.cluster.local/api/v1/write<br>      tls:<br>        insecure: true<br>    # Loki exporter for logs<br>    otlphttp/loki:<br>      endpoint: http://loki-gateway.monitoring.svc.cluster.local/otlp<br>      tls:<br>        insecure: true<br>      # Optimization for cross-region or network latency<br>      sending_queue:<br>        enabled: true<br>        queue_size: 5000<br>      retry_on_failure:<br>        enabled: true<br>        initial_interval: 5s<br>        max_elapsed_time: 300s<br>  <br>  service:<br>    pipelines:<br>      # Traces pipeline<br>      traces:<br>        receivers: [otlp]<br>        processors: [memory_limiter, resourcedetection, attributes, batch]<br>        exporters: [debug]<br>      # Metrics pipeline<br>      metrics:<br>        receivers: [otlp, prometheus, hostmetrics]<br>        processors: [memory_limiter, resourcedetection, attributes, batch]<br>        exporters: [prometheus, prometheusremotewrite, debug]<br>      # Logs pipeline<br>      logs:<br>        receivers: [otlp, filelog]<br>        processors: [memory_limiter, k8sattributes, resource, resourcedetection, attributes, batch]<br>        exporters: [otlphttp/loki, debug]<br><br># ClusterRole section<br>clusterRole:<br>  create: true<br>  rules:<br>    - apiGroups: [&quot;&quot;]<br>      resources: [&quot;pods&quot;, &quot;namespaces&quot;, &quot;nodes&quot;]<br>      verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]<br>    - apiGroups: [&quot;apps&quot;]<br>      resources: [&quot;replicasets&quot;]<br>      verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]<br>    - apiGroups: [&quot;extensions&quot;]<br>      resources: [&quot;replicasets&quot;]<br>      verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]<br><br># Service Account<br>serviceAccount:<br>  create: true<br>  name: otel-collector<br><br># Security context - Root required to read host logs<br>podSecurityContext:<br>  runAsUser: 0<br>  runAsGroup: 0<br>  fsGroup: 0<br><br># Volumes for host logs<br>extraVolumes:<br>  - name: varlogpods<br>    hostPath:<br>      path: /var/log/pods<br>  - name: varlibdockercontainers<br>    hostPath:<br>      path: /var/lib/docker/containers<br><br>extraVolumeMounts:<br>  - name: varlogpods<br>    mountPath: /var/log/pods<br>    readOnly: true<br>  - name: varlibdockercontainers<br>    mountPath: /var/lib/docker/containers<br>    readOnly: true<br><br># Autoscaling disabled for non-prod<br>autoscaling:<br>  enabled: false<br>  </pre><p>Key takeaways:</p><ol><li>OTel will be deployed as a daemonset in kubernetes cluster</li><li>In the OpenTelemetry collector configuration, it is divided into several points: Receivers (The “Inputs”), Processors (The “Cleaning &amp; Enrichment”), Exporters (The “Outputs”), Service / Pipelines (The “Wiring”). we will explain each function of this config in next section</li><li>ClusterRole section is a part of Kubernetes RBAC (Role-Based Access Control). It is essentially a “permission slip” that tells the Kubernetes API what the OpenTelemetry Collector is allowed to see and do.</li><li>Autoscalling section to manage the pod autoscalling for OTel</li></ol><p>The OpenTelemetry (OTel) Collector acts as a central data pipeline. It sits between your applications and your monitoring tools (Prometheus and Loki), handling the “heavy lifting” of data collection, cleaning, and delivery.</p><p>The configuration is organized into four main sections that represent the flow of data:</p><h4>1. Receivers (The “Inputs”)</h4><p>These define where data comes from. In your setup, the collector listens for:</p><ol><li>otlp: Data sent directly from your apps using the OpenTelemetry SDK.</li><li>hostmetrics: Hardware stats like CPU, RAM, and Disk directly from the Kubernetes node.</li><li>filelog: Tailing the raw log files in /var/log/pods/. We added a specific Regex Parser here to extract the namespace and pod name from the file path so you can filter logs easily.</li></ol><h4>2. Processors (The “Cleaning &amp; Enrichment”)</h4><p>This is where the magic happens. Before sending data out, the collector modifies it:</p><ol><li>k8sattributes: The most important one for you. It talks to the Kubernetes API to find out which pod a log or metric belongs to, adding labels like pod_name and namespace.</li><li>resource: We used this to create “shortcuts” for Loki labels, transforming complex metadata into simple keys like {pod=”my-app”}.</li><li>batch: It waits until it has a “batch” of data (e.g., 8192 entries or 200ms) before sending it. This is much more efficient than sending every single log line one by one.</li><li>memory_limiter: A safety net. If the collector uses too much RAM, it will drop data instead of crashing your node.</li></ol><h4>3. Exporters (The “Outputs”)</h4><p>These define where data goes:</p><ol><li>prometheusremotewrite: Pushes metrics to your Prometheus server.</li><li>otlphttp/loki: Pushes logs to your Loki instance. We added a sending_queue here to handle network glitches.</li><li>debug: Prints the data to the collector’s own console logs — very useful when you aren’t sure why a log isn’t showing up in Grafana.</li></ol><h4>4. Service / Pipelines (The “Wiring”)</h4><p>This section connects everything. It defines the path for each data type:</p><ul><li>Metrics Pipeline: Receivers (OTLP, hostmetrics) → Processors → Exporters (Prometheus).</li><li>Logs Pipeline: Receivers (OTLP, filelog) → Processors → Exporters (Loki).</li></ul><p>Without this configuration, your logs and metrics would be “blind.” They wouldn’t have labels, they wouldn’t be compressed, and they would be much harder to query. This configuration turns raw text and numbers into structured, searchable observability data.</p><p>Continue copy and paste yaml file above, then helm install it use these command</p><pre># add repo OTel<br>helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts<br>helm repo update<br><br># (Optional) create separate ns for monitoring stack<br>kubectl create namespace monitoring<br><br># install OTel<br>helm install otel-collector open-telemetry/opentelemetry-collector \<br>  -f otel-server.yaml \<br>  --namespace monitoring<br></pre><p>After the installation success, make sure the OTel pod is running. second you will continue to install grafana and prometheus. Create two yaml files: prometheus.yaml &amp; grafana.yaml</p><pre># prometheus.yaml<br>server:<br>  service:<br>    type: NodePort<br>    nodePort: 30090<br>  <br>  # Enable Remote Write Receiver for OTel integration<br>  extraFlags:<br>    - web.enable-remote-write-receiver<br>    - web.enable-lifecycle<br><br>  # Enable Persistent Storage<br>  persistentVolume:<br>    enabled: true<br>    size: 4Gi<br>    storageClass: &quot;&quot; # uses default storage class, change if needed<br>  # Data retention<br>  retention: 15d<br>  <br>  # Global settings<br>  global:<br>    scrape_interval: 15s<br>    evaluation_interval: 15s<br># Enable Node Exporter for machine metrics (CPU, Memory, Disk)<br>nodeExporter:<br>  enabled: true<br># Enable Kube State Metrics for cluster level metrics (Deployments, Pods status etc)<br>kubeStateMetrics:<br>  enabled: true</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xNqE7lQWW07ZqNjZyuVV5w.png" /><figcaption>Key Takeaway explanation of script prometheus.yaml</figcaption></figure><pre># grafana.yaml<br>service:<br>  type: NodePort<br>  nodePort: 30085<br><br># Enable Persistent Storage<br>persistence:<br>  enabled: true<br>  type: pvc<br>  size: 2Gi<br><br>initChownData:<br>  enabled: false<br><br>podSecurityContext:<br>  fsGroup: 472<br>  runAsGroup: 472<br>  runAsUser: 472<br><br>containerSecurityContext:<br>  runAsUser: 472<br>  runAsGroup: 472<br>  allowPrivilegeEscalation: false<br>  readOnlyRootFilesystem: false # Required for some Grafana operations if not fully tuned<br><br># Admin configuration<br>adminUser: admin<br>## It is recommended to change this on install via --set or secret, but defaults:<br># adminPassword: strongpassword<br># Provision Prometheus Datasource automatically<br>datasources:<br>  datasources.yaml:<br>    apiVersion: 1<br>    datasources:<br>    - name: Prometheus<br>      type: prometheus<br>      url: http://prometheus-server.monitoring.svc.cluster.local  # Adjust namespace if different<br>      access: proxy<br>      isDefault: true<br>    - name: Loki<br>      type: loki<br>      url: http://loki-gateway.monitoring.svc.cluster.local<br>      access: proxy<br># Dashboard Providers - load dashboards from sidecars or files<br>dashboardProviders:<br>  dashboardproviders.yaml:<br>    apiVersion: 1<br>    providers:<br>    - name: &#39;default&#39;<br>      orgId: 1<br>      folder: &#39;&#39;<br>      type: file<br>      disableDeletion: false<br>      editable: true<br>      options:<br>        path: /var/lib/grafana/dashboards/default<br># Provision Dashboards<br>## We will load a popular Node Exporter dashboard (ID: 1860) and a Kubernetes Pods dashboard<br>dashboards:<br>  default:<br>    node-exporter-full:<br>      gnetId: 1860  # Node Exporter Full<br>      revision: 37<br>      datasource: Prometheus<br>    kubernetes-pods:<br>      gnetId: 6417  # Kubernetes Pods<br>      revision: 1<br>      datasource: Prometheus<br># Security &amp; Tweaks<br>grafana.ini:<br>  server:<br>    protocol: http<br>    domain: &lt;domain grafana&gt;<br>    root_url: http://&lt;domain grafana&gt;/ # replace with your domain<br>  security:<br>    allow_embedding: true<br>    cookie_samesite: lax<br>  auth.anonymous:<br>    enabled: false  # Disable anonymous access for security</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FMUtsh0qf1miT5zwCVrKEg.png" /><figcaption>Key Takeaway explanation of script grafana.yaml</figcaption></figure><p>Continue copy and paste yaml file above, then helm install it use these command</p><pre># add repo OTel<br>helm repo add prometheus-community https://prometheus-community.github.io/helm-charts<br>helm repo add grafana https://grafana.github.io/helm-charts<br>helm repo update<br><br># install Grafana and prometheus<br>helm install grafana grafana/grafana \<br>  -f grafana.yaml \<br>  --namespace monitoring<br><br>helm install prometheus prometheus-community/prometheus \<br>  -f prometheus.yaml \<br>  --namespace monitoring</pre><p>Make sure the pod is running and test access grafana on your browser input url http://&lt;server ip address&gt;:30085/login</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZDXTLYsm3y3DzzLon5F0Wg.png" /><figcaption>grafana login page</figcaption></figure><p>Login to grafana use username: admin and password: strongpassword.<br>After that you can continue install loki.yaml. Create yaml file with the following snippet</p><pre># loki.yaml<br># Deployment mode<br>deploymentMode: SingleBinary<br><br># Single binary configuration for simplicity<br>singleBinary:<br>  replicas: 1<br>  <br>  # Resource allocation for non-production<br>  resources:<br>    limits:<br>      cpu: 500m<br>      memory: 512Mi<br>    requests:<br>      cpu: 200m<br>      memory: 256Mi<br>  <br>  # Persistence for logs<br>  persistence:<br>    enabled: true<br>    size: 5Gi<br>    storageClass: &quot;&quot;  # uses default storage class<br><br># Loki configuration<br>loki:<br>  auth_enabled: false<br>  <br>  commonConfig:<br>    replication_factor: 1<br>  <br>  storage:<br>    type: &#39;filesystem&#39;<br>  <br>  schemaConfig:<br>    configs:<br>      - from: 2024-01-01<br>        store: tsdb<br>        object_store: filesystem<br>        schema: v13<br>        index:<br>          prefix: loki_index_<br>          period: 24h<br>  <br>  # Limits configuration<br>  limits_config:<br>    retention_period: 168h  # 7 days retention<br>    max_query_series: 1000<br>    max_query_parallelism: 16<br>    ingestion_rate_mb: 10<br>    ingestion_burst_size_mb: 20<br>    per_stream_rate_limit: 5MB<br>    per_stream_rate_limit_burst: 10MB<br>  <br>  # Compactor for log retention<br>  compactor:<br>    retention_enabled: true<br>    delete_request_store: filesystem<br>    retention_delete_delay: 2h<br>    retention_delete_worker_count: 150<br><br># Service configuration - NodePort for ALB integration<br>service:<br>  type: NodePort<br>  port: 3100<br>  nodePort: 30100<br><br># Gateway configuration (nginx)<br>gateway:<br>  enabled: true<br>  replicas: 1<br>  <br>  service:<br>    type: NodePort<br>    port: 80<br>    nodePort: 30101<br>  <br>  resources:<br>    limits:<br>      cpu: 200m<br>      memory: 256Mi<br>    requests:<br>      cpu: 100m<br>      memory: 128Mi<br><br># Monitoring configuration<br>monitoring:<br>  selfMonitoring:<br>    enabled: false<br>    grafanaAgent:<br>      installOperator: false<br>  <br>  lokiCanary:<br>    enabled: false<br><br># Test configuration<br>test:<br>  enabled: false<br><br># Backend configuration (not needed for single binary)<br>backend:<br>  replicas: 0<br>read:<br>  replicas: 0<br>write:<br>  replicas: 0<br><br># Disable memory-intensive caching for 4GB node<br>chunksCache:<br>  enabled: false<br>resultsCache:<br>  enabled: false</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7UExZlqr-fe11nlQ7jwA5w.png" /><figcaption>Core configuration for loki.yaml</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jfaW36mk-yTajEIQY_0A0w.png" /><figcaption>Data retention setup in loki.yaml</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S6_HjLqKtryUcflTnd9oCg.png" /><figcaption>service and gateway setup in loki.yaml</figcaption></figure><p>Continue to copy and paste yaml file above, then helm install it using these commands and make surethe pod is running</p><pre># install Grafana loki<br>helm install loki grafana/loki -f loki.yaml -n monitoring</pre><p>To access Grafana Loki, you can go to explore, then choose Loki as a source</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xj6XnZvyAORP6HKDSxvAtA.png" /></figure><p>Then you can start a query by select label, then click the blue button refresh to run the query. Congratulations, you have setup grafana loki and Otel for pod monitoring in kubernetes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DK9MG0QVcQ8k3s7uGMfjmw.png" /><figcaption>Sample query in Grafana Loki</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mPn8pf_v3QwTxcnSOzWOLQ.png" /><figcaption>Sample log from logger service</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=23b77109c44f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting to Know Kyverno “Policy Maintainer for Kubernetes”]]></title>
            <link>https://medium.com/@PlatformEnthusiast/getting-to-know-kyverno-policy-maintainer-for-kubernetes-4d9319afcec7?source=rss-b300b45c8552------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d9319afcec7</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[kyverno]]></category>
            <dc:creator><![CDATA[The Platform Enthusiast]]></dc:creator>
            <pubDate>Sun, 01 Feb 2026 04:46:30 GMT</pubDate>
            <atom:updated>2026-02-01T04:52:58.856Z</atom:updated>
            <content:encoded><![CDATA[<p>Today, we would like to discuss Kyverno. This tool is useful for enforcing and auditing policy for a Kubernetes cluster. Imagine you have a Kubernetes cluster running in production, hosting a mission-critical application. You want Kyverno to guard your cluster by using policy-as-code at the admission control layer to automatically enforce security, compliance, and operational best practices before any workload can run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*dwvdI-qdiAqB93mX" /><figcaption>kyverno logo</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UvEy88kPSQWa-xUJ.png" /><figcaption>kyverno diagram</figcaption></figure><h4>Kyverno becomes your cluster’s unbreakable first line of defense</h4><ol><li>Zero-trust security enforcement<br>Every single Pod, Deployment, ConfigMap, or CRD must pass Kyverno’s validation before hitting etcd. No human review, no exceptions. One policy. Instant effect across 1000s of workloads. No container runs privileged unless you explicitly whitelist it.</li><li>Automatic compliance with regulatory frameworks<br>PCI-DSS, HIPAA, SOC2, CIS benchmarks become declarative YAML instead of manual checklist hell. Kyverno translates “every workload must encrypt data at rest”</li><li>Mission-critical workload protection through resource governance<br>Prevent the “noisy neighbor” problem that kills SLAs.<br>No workload starves your databases. No team accidentally spins up 1000 replicas that bankrupt your cloud bill.</li><li>Self-healing operational guardrails<br>Kyverno doesn’t just block bad configs — it fixes them. for example, the dev forgot to apply probing in the yaml files. Kyverno can add them.</li></ol><p>So how does kyverno works? works by intercepting Kubernetes API requests as a dynamic admission controller during the admission control phase, right before objects are persisted to etcd. Here’s how Kubernetes works normally vs. with Kyverno, with flow diagrams.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/288/1*4ZsbEbisovLZ1QrJGldflQ.png" /><figcaption>Kubectl apply flow without kyverno</figcaption></figure><p>Here is a breakdown of each stage:</p><ol><li>The process starts when a user or client sends a command.</li><li>Authentication (AuthN), The kube-apiserver first verifies who is making the request. It checks for valid client certificates, bearer tokens, or other credentials. If this fails, you get a 401 Unauthorized error</li><li>Authorization (AuthZ), Once the system knows who you are, it checks if you have permission to perform the specific action. It uses RBAC (Role-Based Access Control) to verify if your user or ServiceAccount is allowed to create a Pod in that specific namespace. If this fails, you get a 403 Forbidden error.</li><li>Admission Control, After passing identity and permission checks, the request goes through Admission Controllers. These are plugins that can modify or validate the request. <br>Built-in only: In this specific flow, only standard controllers (like LimitRanger, which ensures containers don’t exceed resource limits) are active. This is the “gatekeeper” that enforces cluster-wide policies.</li><li>Persistence to etcd, Once the request is fully validated, the object (the Pod definition) is written to etcd, which is the Kubernetes cluster’s distributed database.</li><li>Execution (Pod Scheduling), After the data is saved in etcd, the Kube-scheduler notices the new Pod and assigns it to a worker node. The kubelet on that node then pulls the image and creates the container.</li></ol><ul><li>The note <strong>“ANY CONFIG ALLOWED”</strong> implies that since custom validation was skipped in step 3, any configuration successfully stored in etcd will now be attempted by the node.</li><li><strong>Crucial Note:</strong> The diagram highlights that there are <strong>no custom checks</strong> at this stage; once it hits etcd, the request is considered “accepted” and the state is stored.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/1*8qMBmNCGHTImZ53cLMMlKQ.png" /><figcaption>kubectl apply with kyverno</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ldeXKn9MfXF2lxal.png" /><figcaption>How kyverno works</figcaption></figure><p>flow from kubectl apply from the user to authorization is similar to normal kubectl apply without Kyverno. The only difference is that instead of only passing identity and permission checks, it can act as “gatekeepers.” They can reject a pod if it doesn’t meet security standards set by Kyverno (e.g., “all containers must have a ‘readonly’ root filesystem”).</p><p>Without Kyverno: Security = hope + manual reviews + fire drills when breaches happen. With Kyverno: Security = provable guarantees + automated prevention + audit trail.</p><p>After discussing the concept lets continue with the implementation of Kyverno. To set up this lab, you will need to provide the following:</p><ol><li>A Kubernetes cluster (we can use kind, k3s or minikube).</li><li>kubectl is configured to talk to that cluster.</li><li>Helm installed (recommended; Kyverno also supports raw YAML).</li></ol><p>In this lab, we will implement Kyverno for non production and production environment:</p><p>Step-1: Add the Kyverno Helm repo</p><pre># add kyverno repo using helm<br>helm repo add kyverno https://kyverno.github.io/kyverno/<br>helm repo update</pre><p>Step-2: Install Kyverno (standalone / non‑prod)</p><p>Use the standalone installation, which deploys Kyverno with single replicas per controller and default settings.</p><pre># this command will install kyverno and create namespace kyverno<br>helm install kyverno kyverno/kyverno \<br>  -n kyverno --create-namespace</pre><p>Key points:</p><ul><li>Kyverno must run in its own dedicated namespace (here, kyverno) and should not share with kube-system or application workloads.​</li><li>This install brings up the admission, background, cleanup, and reports controllers with one replica each by default.</li></ul><p>Step-3: verify installation</p><pre>kubectl get pods -n kyverno<br>kubectl get deploy -n kyverno<br>kubectl get crds | grep kyverno</pre><p>You should see Kyverno Deployments and Pods (e.g., kyverno-admission-controller, kyverno-background-controller, etc.) in Running state and Kyverno CRDs like clusterpolicies.kyverno.io present.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tdRxnLgr0k91vLbkrueJVg.png" /><figcaption>sample of kyverno non production pods</figcaption></figure><p>Step-4: enable sample policy and test it</p><p>Copy and paste this sample policy-test.yaml and apply using the command kubectl apply -f policy-test.YAML</p><pre>apiVersion: kyverno.io/v1<br>kind: ClusterPolicy<br>metadata:<br>  name: require-labels<br>spec:<br>  validationFailureAction: Enforce<br>  rules:<br>  - name: require-labels<br>    match:<br>      any:<br>        - resources:<br>            kinds:<br>            - Deployment<br>    validate:<br>      message: &quot;label team is required for all deployment&quot;<br>      pattern:<br>        metadata:<br>          labels:<br>            team: &quot;?*&quot;  </pre><p>Key takeaways:</p><ul><li>This policy will enforce that all deployments in any namespace have a label “team: {value}”.</li><li>kyverno have to mechanism to apply policy. Start all policies in “audit” mode (or with validationFailureAction: audit) to see violations without blocking resources.</li><li>Gradually change to enforce for specific namespaces or workloads once you are confident.</li></ul><p>after apply test policy apply this yaml file</p><pre>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: whoami<br>  namespace: traefik-system<br>  labels:<br>    app: whoami<br>spec:<br>  replicas: 2<br>  selector:<br>    matchLabels:<br>      app: whoami<br>  template:<br>    metadata:<br>      labels:<br>        app: whoami<br>    spec:<br>      containers:<br>        - name: whoami<br>          image: traefik/whoami:v1.10<br>          ports:<br>            - containerPort: 80<br>          resources:<br>            requests:<br>              cpu: 100m<br>              memory: 64Mi<br>            limits:<br>              cpu: 200m<br>              memory: 128Mi</pre><p>In yaml file above there is no lables: team kyverno should reject this request from user to apply.</p><p>In mission-critical production, the cost of <em>one breach, one outage, or one compliance violation</em> dwarfs the 15 minutes it takes to install Kyverno and write your first 5 policies. Kyverno isn’t “nice to have.” It’s the engineering discipline that turns “trust but verify” into “verify or die trying” for your production cluster.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d9319afcec7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Kubernetes Non-Production Environments in AWS]]></title>
            <link>https://medium.com/@PlatformEnthusiast/building-kubernetes-non-production-environments-in-aws-c82303b35b01?source=rss-b300b45c8552------2</link>
            <guid isPermaLink="false">https://medium.com/p/c82303b35b01</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[fck-nat]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[The Platform Enthusiast]]></dc:creator>
            <pubDate>Sat, 24 Jan 2026 08:54:29 GMT</pubDate>
            <atom:updated>2026-02-16T03:04:49.396Z</atom:updated>
            <content:encoded><![CDATA[<p>Building cost-effective non-production for Kubernetes in AWS is challenging, especially for an organization or company that has a global team. This situation allows the development team to work according to their own time zone; therefore, the dev environment needs to run 24/7. The non-production Kubernetes environment ideally consists of at least 3 clusters: Pre-production cluster, this cluster has near-identical to production (same node types, configs); Staging cluster, Shared cluster or dedicated namespace for integration testing; Development cluster, is a single-node lightweight cluster (e.g., Minikube, kind, or K3s) for developers to test their code and do SIT before the UAT phase with QA in SDLC lifecycle.</p><p>Today, we will focus on discussing the development cluster using multiple nodes, k3s, alb, and nat service. The infrastructure diagram will be as follows</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pmlcoODfF71LiXnb_EfI4A.png" /><figcaption>The overview diagram of non production environment</figcaption></figure><p>There are many approaches available to set this. Some teams prefer to provision eks cluster and separate each environment with namespaces and Kubernetes RBAC. This approach is inefficient and expensive. It is not recommended because it will require higher costs (~$0.10/hr per cluster + node fees) even for non-prod; namespace isolation risks “noisy neighbor” issues like resource contention or lateral movement. For development and staging, we suggest using the k3s approach. since this approach provides us a better saving rate for about 45% — 60% compared to eks cluster approach.</p><h3>Pre-requities</h3><p>Before getting started, ensure you have these items set up:</p><ol><li>AWS account</li><li>One domain (we will need 2 subdomains created for these labs,s one for monitoring and the other for accessing api)</li><li>Terraform is installed in local machine</li><li>Iam users with permission to provision vpc, ec2 and ALB</li></ol><h3>Guides:</h3><p>first you need to clone this Terraform script to your local machine</p><pre>git clone: https://gitlab.com/platform-project2/aws-script.git</pre><p>The structure of terraform script will be like this</p><pre>aws-script/<br>├── environments/             # Environment-specific configurations<br>│   └── development/          # Development environment (this directory)<br>│       ├── README.md         # This documentation<br>│       └── terraform.tfvars  # file to pass value to k3s/main.tf<br>├── infrastructures/          # Main infrastructure definitions<br>│   └── k3s/                  # k3s cluster infrastructure<br>│       ├── main.tf<br>│       ├── variables.tf<br>│       ├── output.tf<br>│       └── version.tf<br>└── modules/               # Reusable Terraform modules<br>    ├── alb/               # Application Load Balancer module<br>    ├── ec2/               # EC2 instance module (On-Demand/Spot)<br>    ├── ecr/               # Elastic Container Registry module<br>    ├── iam/               # IAM roles and policies module<br>    └── vpc/               # Virtual Private Cloud module</pre><p>In this structure, we will use the value defined in “terraform.tfvars” to provision infratructure. This file will deploy three EC2 instances, one ALB, one IAM user, and an ECR. This infrastructure will be ready to use for the CICD pipeline enable CI to push images to ecr provisioned by Terraform. Before executing this script make sure your local device has aws cli installed and configured. The following AWS managed policies can be attached to the IAM identity (user/role) running Terraform:</p><ul><li>AmazonVPCFullAccess</li><li>AmazonEC2FullAccess</li><li>AmazonElasticLoadBalancingFullAccess</li><li>AmazonEC2ContainerRegistryFullAccess</li><li>IAMFullAccess</li><li>AmazonS3FullAccess (Required for the S3 backend)</li></ul><blockquote><em>For better security, consider creating a custom scoped-down IAM policy that only allows the specific actions listed below:<br></em><strong><em>EC2/VPC</em></strong><em>: Full access to create/manage VPCs, subnets, instances, routes, and security groups.<br></em><strong><em>ELBv2 &amp; ECR</em></strong><em>: Full access to load balancers, target groups, repositories, and tags.<br></em><strong><em>IAM</em></strong><em>: Can create users, keys, policies, and attach or pass roles.<br></em><strong><em>S3</em></strong><em>: Limited to listing, getting, and putting objects in the state bucket.</em></blockquote><p>Before executing code make sure you have provisioned s3 bucket to save terraform state. Replace Bucket name with your available bucket.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*IPJpaLiQn1OTUlYtG1gJgQ.png" /><figcaption>Snippet Bucket s3</figcaption></figure><p>Go to folder aws-script/infrastructure/k3s, then do terraform init. Continue Terraform validate to validate Terraform syntax. After that, execute “terraform plan — out=tfplan -var-file=../../environments/development/terraform.tfvars” continue execute “terraform apply -var-file=../../environments/development/terraform.tfvars tfplan” do provision all required infrastuctures.</p><p>Second, ssh to EC2-bastion then ssh to EC2-k3s then install k3s execute these command:</p><pre># download and install k3s<br>curl -sfL https://get.k3s.io | sh -<br><br># check status of k3s<br>sudo systemctl status k3s<br><br># check status of k3s node<br>sudo kubectl get nodes</pre><p>By default, to communicate with k3s api, you will need sudo access. let’s change its permission</p><pre># Change permission of k3s:<br>sudo chmod 644 /etc/rancher/k3s/k3s.yaml <br><br># test get nodes <br>kubectl get nodes</pre><p>To enable the bastion server communicate with k3s-server, use this command</p><pre>#Copy content of k3s kube config from k3s server:<br>sudo cat /etc/rancher/k3s/k3s.yaml ~&gt; copy contents of this file<br><br>#On bastion node create kubeconfig file:<br>mkdir -p ~/.kube<br><br>#paste content from kubeconfig and edit server with ip private of k3s server<br>sudo vi ~/.kube/config <br>sudo chown $(id -u):$(id -g) ~/.kube/config<br><br>#test command kubectl<br>kubectl get nodes<br></pre><p>and then continue join the k3s worker run the following command, replacing:</p><ul><li>K3S_URL with your controller node’s IP and port (default is 6443)</li><li>K3S_TOKEN with the node token you copied earlier</li></ul><pre># install and join ec2 to k3s server<br>curl -sfL https://get.k3s.io | K3S_URL=https://controller-node-ip:6443 K3S_TOKEN=your-node-token sh -<br><br># go back to bastion server and test this command to check node status<br>kubectl get nodes -o wide</pre><p>Congratulations, your K3s cluster is created. This article explains to you how to set up a simple non production kubernetes cluster using k3s. if you want to continue to setup api gateway for this setup, we recommend using Traefik since this setup is easy to install and maintain. you can follow this article to install it</p><p><a href="https://medium.com/@PlatformEnthusiast/setup-traefik-in-kubernetes-for-beginner-7434dbddd33b">https://medium.com/@PlatformEnthusiast/setup-traefik-in-kubernetes-for-beginner-7434dbddd33b</a></p><p>In the Terraform scripts, we have created alb with 4 target groups</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*ygw3NDkhcohkDevz5pSjGg.png" /><figcaption>snippet 3 target group for alb</figcaption></figure><p>This config enables us to provision 3 services in k3s using 3 different subdomains for grafana, traefik dashboard and api exposed via traefik.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c82303b35b01" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How To Deploy a Basic Amazon EKS Cluster Using Terraform (Non Production Setup)]]></title>
            <link>https://medium.com/@PlatformEnthusiast/how-to-deploy-a-basic-amazon-eks-cluster-using-terraform-non-production-setup-63f63eff0b9e?source=rss-b300b45c8552------2</link>
            <guid isPermaLink="false">https://medium.com/p/63f63eff0b9e</guid>
            <category><![CDATA[aws-eks]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[The Platform Enthusiast]]></dc:creator>
            <pubDate>Mon, 05 Jan 2026 16:30:38 GMT</pubDate>
            <atom:updated>2026-01-05T16:31:50.001Z</atom:updated>
            <content:encoded><![CDATA[<p>This article will walk through deploying a basic Amazon EKS cluster using Terraform, with a simple VPC, managed node group, and minimal IAM permissions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*Y_sc8FovxAufyhos.png" /><figcaption>EKS Logo source: aws doc</figcaption></figure><h3><strong>Prerequisites</strong></h3><p>Before starting, ensure the following are in place.</p><ol><li>An AWS account with programmatic access (access key and secret key) configured via aws configure or environment variables.</li><li>Installed tools:<br>a. Terraform (v1.3+ recommended)<br>b. AWS CLI v2<br>c. kubectl matching at least the EKS minor version (for example, 1.29).</li><li>A non‑administrator IAM user or role with permissions to manage VPC, IAM, and EKS (described in “Minimal IAM roles” ).</li></ol><p>In this article we will use a multi‑environment layout similar to common Terraform patterns, with shared modules and per‑environment configuration.</p><pre>├── modules/<br>│   ├── vpc/<br>│   │   └── main.tf<br>│   ├── eks/<br>│   │   └── main.tf<br>│   └── iam/<br>│       └── main.tf<br>└── environments/<br>    └── dev/<br>        ├── main.tf<br>        ├── variables.tf<br>        └── terraform.tfvars</pre><p>In this structure:</p><ol><li>modules/ contains reusable building blocks (VPC, EKS, IAM).</li><li>environments/dev/main.tf wires the modules together for the dev environment.</li><li>environments/dev/variables.tf declares the variables used in main.tf.</li><li>environments/dev/terraform.tfvars provides environment‑specific values such as region, cluster name, and CIDR ranges.</li></ol><p>We will begin by creating module iam. this module used to manage roles and permission for EKS control plane needs a role to manage resources, and worker nodes need an instance profile for joining the cluster. These policies provide only what the EKS control plane and nodes need, without broad AdministratorAccess.</p><p>Create modules/iam/main.tf:</p><pre>variable &quot;cluster_name&quot; {}<br><br>data &quot;aws_iam_policy_document&quot; &quot;eks_cluster_assume_role&quot; {<br>  statement {<br>    actions = [&quot;sts:AssumeRole&quot;]<br><br>    principals {<br>      type        = &quot;Service&quot;<br>      identifiers = [&quot;eks.amazonaws.com&quot;]<br>    }<br>  }<br>}<br><br>resource &quot;aws_iam_role&quot; &quot;eks_cluster_role&quot; {<br>  name               = &quot;${var.cluster_name}-cluster-role&quot;<br>  assume_role_policy = data.aws_iam_policy_document.eks_cluster_assume_role.json<br>}<br><br>resource &quot;aws_iam_role_policy_attachment&quot; &quot;eks_cluster_AmazonEKSClusterPolicy&quot; {<br>  role       = aws_iam_role.eks_cluster_role.name<br>  policy_arn = &quot;arn:aws:iam::aws:policy/AmazonEKSClusterPolicy&quot;<br>}<br><br>resource &quot;aws_iam_role_policy_attachment&quot; &quot;eks_cluster_AmazonEKS_VPC_ResourceController&quot; {<br>  role       = aws_iam_role.eks_cluster_role.name<br>  policy_arn = &quot;arn:aws:iam::aws:policy/AmazonEKSServicePolicy&quot;<br>}<br><br>data &quot;aws_iam_policy_document&quot; &quot;eks_node_assume_role&quot; {<br>  statement {<br>    actions = [&quot;sts:AssumeRole&quot;]<br><br>    principals {<br>      type        = &quot;Service&quot;<br>      identifiers = [&quot;ec2.amazonaws.com&quot;]<br>    }<br>  }<br>}<br><br>resource &quot;aws_iam_role&quot; &quot;eks_node_role&quot; {<br>  name               = &quot;${var.cluster_name}-node-role&quot;<br>  assume_role_policy = data.aws_iam_policy_document.eks_node_assume_role.json<br>}<br><br>resource &quot;aws_iam_role_policy_attachment&quot; &quot;worker_node_AmazonEKSWorkerNodePolicy&quot; {<br>  role       = aws_iam_role.eks_node_role.name<br>  policy_arn = &quot;arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy&quot;<br>}<br><br>resource &quot;aws_iam_role_policy_attachment&quot; &quot;worker_node_AmazonEC2ContainerRegistryReadOnly&quot; {<br>  role       = aws_iam_role.eks_node_role.name<br>  policy_arn = &quot;arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly&quot;<br>}<br><br>resource &quot;aws_iam_role_policy_attachment&quot; &quot;worker_node_AmazonEKS_CNI_Policy&quot; {<br>  role       = aws_iam_role.eks_node_role.name<br>  policy_arn = &quot;arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy&quot;<br>}<br><br>resource &quot;aws_iam_instance_profile&quot; &quot;eks_node_instance_profile&quot; {<br>  name = &quot;${var.cluster_name}-node-instance-profile&quot;<br>  role = aws_iam_role.eks_node_role.name<br>}<br><br>output &quot;cluster_role_arn&quot; {<br>  value = aws_iam_role.eks_cluster_role.arn<br>}<br><br>output &quot;node_role_arn&quot; {<br>  value = aws_iam_role.eks_node_role.arn<br>}</pre><p>Continue creating vpc modules for simplicity, this guide uses a single VPC with <strong>public</strong> subnets and internet gateways.</p><p>Create modules/vpc/main.tf:</p><pre>terraform {<br>  required_version = &quot;&gt;= 1.3&quot;<br>  required_providers {<br>    aws = {<br>      source  = &quot;hashicorp/aws&quot;<br>      version = &quot;~&gt; 5.0&quot;<br>    }<br>  }<br>}<br><br>provider &quot;aws&quot; {<br>  region = var.aws_region<br>}<br><br>variable &quot;aws_region&quot; {}<br>variable &quot;vpc_cidr&quot; {}<br>variable &quot;public_subnets&quot; {}<br><br>data &quot;aws_availability_zones&quot; &quot;available&quot; {}<br><br>resource &quot;aws_vpc&quot; &quot;this&quot; {<br>  cidr_block           = var.vpc_cidr<br>  enable_dns_support   = true<br>  enable_dns_hostnames = true<br>}<br><br>resource &quot;aws_internet_gateway&quot; &quot;this&quot; {<br>  vpc_id = aws_vpc.this.id<br>}<br><br>resource &quot;aws_subnet&quot; &quot;public&quot; {<br>  count                   = length(var.public_subnets)<br>  vpc_id                  = aws_vpc.this.id<br>  cidr_block              = var.public_subnets[count.index]<br>  map_public_ip_on_launch = true<br>  availability_zone       = data.aws_availability_zones.available.names[count.index]<br>}<br><br>resource &quot;aws_route_table&quot; &quot;public&quot; {<br>  vpc_id = aws_vpc.this.id<br>}<br><br>resource &quot;aws_route&quot; &quot;public_internet&quot; {<br>  route_table_id         = aws_route_table.public.id<br>  destination_cidr_block = &quot;0.0.0.0/0&quot;<br>  gateway_id             = aws_internet_gateway.this.id<br>}<br><br>resource &quot;aws_route_table_association&quot; &quot;public&quot; {<br>  count          = length(aws_subnet.public)<br>  subnet_id      = aws_subnet.public[count.index].id<br>  route_table_id = aws_route_table.public.id<br>}<br><br>output &quot;vpc_id&quot; {<br>  value = aws_vpc.this.id<br>}<br><br>output &quot;public_subnet_ids&quot; {<br>  value = [for s in aws_subnet.public : s.id]<br>}</pre><blockquote><strong>Security warning:</strong> placing EKS worker nodes in public subnets exposes them directly to the internet, which is not recommended for production workloads; use private subnets and NAT gateways for production clusters.</blockquote><p>Continue create eks module with managed groups, this script we will create one nodegroup on demand. Create modules/eks/main.tf:</p><pre>terraform {<br>  required_providers {<br>    aws = {<br>      source  = &quot;hashicorp/aws&quot;<br>      version = &quot;~&gt; 5.0&quot;<br>    }<br>  }<br>}<br><br>variable &quot;cluster_name&quot; {}<br>variable &quot;cluster_version&quot; {<br>  default = &quot;1.29&quot;<br>}<br>variable &quot;vpc_id&quot; {}<br>variable &quot;public_subnet_ids&quot; {}<br>variable &quot;cluster_role_arn&quot; {}<br>variable &quot;node_role_arn&quot; {}<br>variable &quot;node_instance_type&quot; {}<br>variable &quot;desired_capacity&quot; {}<br>variable &quot;aws_region&quot; {}<br><br>provider &quot;aws&quot; {<br>  region = var.aws_region<br>}<br><br>module &quot;eks&quot; {<br>  source  = &quot;terraform-aws-modules/eks/aws&quot;<br>  version = &quot;&gt;= 19.0, &lt; 21.0&quot;<br><br>  cluster_name    = var.cluster_name<br>  cluster_version = var.cluster_version<br><br>  vpc_id     = var.vpc_id<br>  subnet_ids = var.public_subnet_ids<br><br>  cluster_endpoint_public_access = true<br><br>  cluster_role_arn = var.cluster_role_arn<br><br>  eks_managed_node_group_defaults = {<br>    instance_types = [var.node_instance_type]<br>    ami_type       = &quot;AL2_x86_64&quot;<br>  }<br><br>  eks_managed_node_groups = <br>    default = {<br>      min_size     = 1<br>      max_size     = 3<br>      desired_size = var.desired_capacity<br>      iam_role_arn = var.node_role_arn<br>    }<br>  }<br>}<br><br>output &quot;cluster_name&quot; {<br>  value = module.eks.cluster_name<br>}<br><br>output &quot;cluster_endpoint&quot; {<br>  value = module.eks.cluster_endpoint<br>}<br><br>output &quot;cluster_certificate_authority_data&quot; {<br>  value = module.eks.cluster_certificate_authority_data<br>}</pre><blockquote><strong>Security warning:</strong> cluster_endpoint_public_access = true exposes the Kubernetes API publicly; restrict access using CIDR restrictions or private endpoints in production environments.</blockquote><p>the last one continue to wiring up all modules into one single files. Now compose the modules in environments/dev/main.tf:</p><pre>terraform {<br>  required_version = &quot;&gt;= 1.3&quot;<br>  required_providers {<br>    aws = {<br>      source  = &quot;hashicorp/aws&quot;<br>      version = &quot;~&gt; 5.0&quot;<br>    }<br>  }<br>}<br><br>provider &quot;aws&quot; {<br>  region = var.aws_region<br>}<br><br>module &quot;vpc&quot; {<br>  source          = &quot;../../modules/vpc&quot;<br>  aws_region      = var.aws_region<br>  vpc_cidr        = var.vpc_cidr<br>  public_subnets  = var.public_subnets<br>}<br><br>module &quot;iam&quot; {<br>  source       = &quot;../../modules/iam&quot;<br>  cluster_name = var.cluster_name<br>}<br><br>module &quot;eks&quot; {<br>  source                  = &quot;../../modules/eks&quot;<br>  aws_region              = var.aws_region<br>  cluster_name            = var.cluster_name<br>  vpc_id                  = module.vpc.vpc_id<br>  public_subnet_ids       = module.vpc.public_subnet_ids<br>  cluster_role_arn        = module.iam.cluster_role_arn<br>  node_role_arn           = module.iam.node_role_arn<br>  node_instance_type      = var.node_instance_type<br>  desired_capacity        = var.desired_capacity<br>}</pre><p>environments/dev/terraform.tfvars:</p><pre>aws_region        = &quot;us-east-1&quot;<br>cluster_name      = &quot;dev-eks&quot;<br>vpc_cidr          = &quot;10.0.0.0/16&quot;<br>public_subnets    = [&quot;10.0.1.0/24&quot;, &quot;10.0.2.0/24&quot;]<br>node_instance_type = &quot;t3.medium&quot;<br>desired_capacity   = 2</pre><p>script explanation:</p><ol><li>aws_region = “ap-southeast-1”, Uses the us region. with cluster_name = “dev-eks”</li><li>cluster_name = “dev-eks”, Names the cluster “dev‑eks”, clearly signaling this is a development/learning cluster, not production.</li><li>vpc_cidr = “10.0.0.0/16”, Gives you 65,536 private IPs, which is more than enough for a small dev cluster and easy to remember.​<br>public_subnets = [“10.0.1.0/24”, “10.0.2.0/24”]</li><li>Creates two public subnets, each with 256 IPs, typically spread across two Availability Zones for basic high availability.​<br>node_instance_type = “t3.medium”</li><li>Picks a burstable general‑purpose instance type, which is relatively cheap but still good enough for small workloads.​<br>desired_capacity = 2, Starts with 2 worker nodes; you can lower this to 1 to reduce cost, or increase it later if you need more capacity.​</li></ol><p><strong>Deploying the EKS cluster</strong></p><p>From the project root, change into the dev environment and run Terraform:</p><pre>cd environments/dev<br> terraform init<br> terraform plan<br> terraform apply</pre><p>·</p><ol><li>terraform init downloads providers and initializes the working directory.</li><li>terraform plan shows all resources that will be created, including VPC, IAM roles, EKS cluster, and managed node group.</li><li>terraform apply provisions the cluster; this can take 10–15 minutes depending on region and capacity.</li></ol><p>During apply, Terraform will prompt for confirmation; type yes when ready.</p><p><strong>Connecting with kubectl</strong></p><p>After the cluster is created, update your local kubeconfig to point to the new EKS cluster.</p><pre>aws eks update-kubeconfig --region ap-southeast-1 --name dev-eks</pre><p>This command fetches the cluster endpoint and certificate data, then merges them into ~/.kube/config.</p><p>Verify the nodes:</p><pre>kubectl get nodes</pre><p>You should see 1–3 Ready nodes backed by the managed node group.</p><p><strong>Cost and security notes<br></strong>Running an EKS cluster incurs costs for both the control plane and worker nodes.</p><ol><li>EKS control plane has an hourly charge per cluster (in addition to data transfer and other usage‑based costs).</li><li>A t3.medium instance in us-east-1 typically costs several tens of USD per month per node on on‑demand pricing; running 2 instances can quickly add up across months.</li><li>Public subnets and public API endpoints simplify learning but are not suitable for production; prefer private subnets and endpoint access restrictions.</li></ol><p>Always review the AWS pricing page and keep this cluster for as short a time as needed, especially in non‑free‑tier accounts</p><p><strong>Cleanup with terraform destroy</strong></p><p>To avoid ongoing charges, tear down the cluster and associated resources when finished.</p><p>From environments/dev:</p><pre>terraform destroy</pre><ol><li>Review the plan carefully to ensure only expected resources are destroyed, then type yes to confirm.</li><li>This will delete the EKS cluster, managed node group, VPC, and IAM roles/instance profile created by this configuration; any workloads or data stored in the cluster will be lost.</li></ol><p><strong>Troubleshooting</strong></p><p>Common issues when creating an EKS cluster with Terraform include IAM, subnet, and version mismatches.</p><ol><li><strong>AccessDenied errors during apply<br></strong>Ensure the AWS caller identity (user or role) has permissions: eks:* (cluster CRUD), ec2:* (networking resources), iam:CreateRole, iam:AttachRolePolicy, iam:PassRole for the specific EKS and node roles. Limit iam:PassRole to ARNs that match your EKS and node roles instead of wildcard resources.</li><li><strong>Nodes not joining the cluster<br></strong>Confirm the node IAM role has AmazonEKSWorkerNodePolicy, AmazonEKS_CNI_Policy, and AmazonEC2ContainerRegistryReadOnly attached. Check that the subnets used are tagged for Kubernetes (the EKS module normally handles this), and that the security groups allow required traffic to the control plane.</li><li><strong>kubectl timeouts<br></strong>Verify aws eks update-kubeconfig used the same region and cluster name as Terraform. Ensure your IP can reach the public control plane endpoint; corporate networks or VPNs sometimes block outbound traffic to AWS.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=63f63eff0b9e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>