<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by TheInvincible on Medium]]></title>
        <description><![CDATA[Stories by TheInvincible on Medium]]></description>
        <link>https://medium.com/@theinvincibledev?source=rss-ddb6d9aa3746------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*5DS_3rhBuaZmd-KHrCL8Yw.jpeg</url>
            <title>Stories by TheInvincible on Medium</title>
            <link>https://medium.com/@theinvincibledev?source=rss-ddb6d9aa3746------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 21:56:59 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@theinvincibledev/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[[Part 11] Wrapping Up: What We Have Built, the Full Teardown, and Portfolio Value]]></title>
            <link>https://medium.com/@theinvincibledev/part-11-wrapping-up-what-we-have-built-the-full-teardown-and-portfolio-value-fd8a69442681?source=rss-ddb6d9aa3746------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd8a69442681</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[google-kubernetes-engine]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[TheInvincible]]></dc:creator>
            <pubDate>Mon, 24 Aug 2026 10:57:59 GMT</pubDate>
            <atom:updated>2026-08-24T10:57:59.440Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hmBiKTMNa9NqrPfm" /><figcaption>Photo by <a href="https://unsplash.com/@kmuza?utm_source=medium&amp;utm_medium=referral">Carlos Muza</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>And just like that, we are done building a production-grade platform on a free trial budget.</p><p>We have learnt a lot, experienced a lot, and built something fun and large. Now is the time to retire the project. To be honest, I don’t feel like doing this. There is actually real beauty and joy in building something from scratch: I love seeing and reading the charts on Grafana and Datadog; I love going back to the code and pointing out what each piece does in the project, and I hope you love or would love doing the same. You can actually keep the cluster running for a while, but you can’t keep it for too long, or you will quickly use up your credits.</p><p>Before we tear down the infra, let’s go over the stack once again, phase by phase:</p><ol><li><strong>Organization and cost controls.</strong> A Google Cloud organization, IAM, a service account for Terraform, a dedicated project, and the billing discipline that made the whole thing possible: spot nodes, the e2 machine family, project isolation, and a budget alert that stops the cluster before the credits run out.</li><li><strong>The cluster.</strong> A regional GKE cluster with node pools spread across three availability zones, private and spot, provisioned entirely with Terraform. Six nodes: three for system workloads, three for applications.</li><li><strong>The storefront.</strong> The Online Boutique, twelve microservices, behind an Envoy Gateway managed with the Gateway API. Cloudflare provides DNS and HTTPS on a real domain, so the platform was reachable from the public internet from the first phase.</li><li><strong>The WAF.</strong> A Coraza Web Application Firewall with the OWASP Core Rule Set, attached cluster-wide to the gateway. SQL injection payloads and other OWASP Top 10 attacks returned 403 through the public hostname, verified by an automated test script.</li><li><strong>The network.</strong> Cilium as the CNI, eBPF instead of kube-proxy, WireGuard encrypting pod-to-pod traffic, and Hubble observing the flows.</li><li><strong>The database.</strong> CloudNativePG running PostgreSQL with three replicas spread across the zones, a leader with automatic promotion, proven under load with pgbench.</li><li><strong>The failover.</strong> The leader killed, the replica promoted, the service restored, measured before and after. The one-off kubectl delete pod became a repeatable experiment.</li><li><strong>The observability.</strong> Prometheus and Grafana dashboards for the database and the WAF, Datadog with agent and cluster agent, APM and logs, so the platform was not just running but measured.</li><li><strong>The chaos.</strong> Chaos Mesh turning failure injection into declarative experiments: pod kills, CPU stress, with a steady-state hypothesis before every injection and the dashboards as the evidence layer.</li><li><strong>The AI.</strong> An Ollama inference server with a persistent model disk, a Redis queue, a worker, and KEDA scaling the model to zero when idle and waking it when a job arrives. The same queue pattern that runs OpenAI Batch and Anthropic Message Batches, deployed on our own cluster.</li></ol><p>The four pillars of the project and how they are materialized:</p><ul><li>Security: the WAF and the encrypted network</li><li>Reliability: the HA database and the chaos experiments</li><li>Observability: the dashboards and the agent</li><li>Cost-awareness: the spot nodes, the scale-to-zero, and the budget kill switch</li></ul><h4>The cleanup: shutting it all down</h4><p>There are two layers to the teardown, and the choice between them is yours: either wait for the kill switch, and most of the credits are gone by the time it fires; or tear down by hand now, and keep the remaining budget for other projects. You choose. <strong>I’d prefer the latter for you</strong>.</p><h4>The automated kill switch</h4><p>The kill switch is the Cloud Function we deployed earlier, triggered by a Pub/Sub budget alert. The budget publishes at 95% of the credits, and the function stops every running instance in the cluster zones.</p><h4>The ordered teardown</h4><p>For a clean rebuild or a permanent close, the teardown runs in order, from the cluster up to the organization:</p><pre># 1. Destroy the infrastructure (node pools + cluster + networking, ~10 min).<br>cd infra/terraform<br>terraform destroy -var-file=terraform.tfvars<br><br><br># 2. Verify nothing is left running (all three should be empty):<br>gcloud compute instances list<br>gcloud compute addresses list<br>gcloud compute disks list<br><br># 3. Delete the state bucket (holds terraform state and backups):<br>gcloud storage rm -r gs://devops-portfolio-tfstate<br><br># 4. Local cleanup, done once, irreversible:<br>rm -rf .terraform<br>kubectl config delete-context gke_devops-portfolio-prod_us-central1_devops-portfolio<br>kubectl config delete-cluster gke_devops-portfolio-prod_us-central1_devops-portfolio<br><br># 5. Delete the project, only if closing for good (30-day recovery window):<br>gcloud projects delete devops-portfolio-prod<br><br># 6. Delete the folder, which must be empty first (also 30 days):<br>FOLDER_ID=$(gcloud resource-manager folders list \<br>  --organization=$(gcloud organizations list --format=&#39;value(ID)&#39;) \<br>  --filter=&#39;displayName=&quot;Infrastructure Engineering&quot;&#39; \<br>  --format=&#39;value(ID)&#39;)<br>gcloud resource-manager folders delete &quot;$FOLDER_ID&quot;</pre><ul><li><strong>Cloudflare (console):</strong> DNS, remove the boutique A record created for the project. If the domain is reused elsewhere, reset the SSL mode from Flexible back to Full (strict).</li></ul><blockquote>Steps 1–4 remove the infrastructure and the state but leave the folder, the organization, and the identity layer intact: a fresh rebuild needs only a new project, a service account, and a state bucket. Steps 5–6 close the project for good, and both carry a 30-day recovery window. The organization itself cannot be deleted with gcloud; it lives on the Cloud Identity account, and closing it completely means cancelling the account from the admin console.</blockquote><p>The order matters. The cluster is destroyed before the project, the project before the folder, and the folder only when it is empty. Skipping the order leaves orphaned resources, which is exactly what the kill switch exists to prevent, anyway.</p><blockquote>I spent <strong>$165</strong> in total credit building the project twice and keeping it up for over 3 weeks in order to write and demo for the project series. Your spending should be less.</blockquote><h4>After the teardown</h4><p>The cluster is gone, the records are removed, the folder is closed. What remains is what you have learnt and the experience of building an enterprise-grade system from the ground up. So feel free to add the experience to your portfolio, something like:</p><blockquote>“I built a production-grade cloud platform on GKE: twelve microservices behind a WAF, a self-healing PostgreSQL cluster, chaos-validated reliability, and a scale-to-zero AI inference service, all provisioned as code and all under a $300 budget.”</blockquote><p>Or anything you can do to showcase your new knowledge and experience.</p><p>Finally, be on the lookout for more practical posts from me on DevOps and Cloud-Native technologies, and don’t forget to clap and share.</p><p>You can find the entire repository here, plus the ordered list of the blog:</p><p><a href="https://github.com/TheInvincibleRalph/Enterprise-grade-GKE-Project">GitHub - TheInvincibleRalph/Enterprise-grade-GKE-Project: A production-grade cloud platform built on Google Kubernetes Engine on a private multi-zone cluster with eBPF networking and encrypted pod traffic, a WAF-protected edge, highly available PostgreSQL, enterprise observability, chaos engineering, and an inference server that scales to zero when idle.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd8a69442681" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[Part 10] Deploying an Ollama Inference Server on GKE with KEDA]]></title>
            <link>https://medium.com/@theinvincibledev/part-10-deploying-an-ollama-inference-server-on-gke-with-keda-7c89a4081461?source=rss-ddb6d9aa3746------2</link>
            <guid isPermaLink="false">https://medium.com/p/7c89a4081461</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[ollama]]></category>
            <dc:creator><![CDATA[TheInvincible]]></dc:creator>
            <pubDate>Sat, 22 Aug 2026 13:41:05 GMT</pubDate>
            <atom:updated>2026-08-22T13:46:53.883Z</atom:updated>
            <content:encoded><![CDATA[<h4>Event-Driven Autoscaling, Redis Queue, and an Async Inference API</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*xvWqPXjoDC9iD_fk.jpg" /></figure><p>An API inference server exposes a model over HTTP to receive prompts and return text. On its own, an AI model is just a static file of weights; Ollama is the server that actually loads it into memory and runs the inference. In production, this API is typically consumed by code, like backends or data pipelines. Because inference takes time, they don’t wait for a response. They submit a prompt, get a task ID, and fetch the result when it’s ready. This asynchronous pattern is the same contract used by OpenAI and Anthropic for their batch APIs.</p><p>Since our free trial cluster doesn’t have GPUs, we are running inference on CPU using a small, quantized model (<strong>llama3.2:3b</strong>) served by Ollama. It processes one prompt at a time, and each takes a few seconds. The problem is idle cost: keeping 2 GB of weights loaded in memory doing nothing is the most expensive part of a node billed by the hour.</p><p>To solve this, we need an architecture that only exists when there is active work:</p><ul><li>The API accepts a prompt, returns a task ID immediately, and drops the connection.</li><li>Redis holds the incoming prompts in a queue.</li><li>A worker pulls jobs from Redis and calls the Ollama server to process them.</li><li>KEDA (Kubernetes Event-driven Autoscaling) monitors the Redis queue to decide how many model replicas should be running.</li></ul><p>KEDA is the critical component here. The built-in Kubernetes Horizontal Pod Autoscaler (HPA) scales based on CPU and memory, and it will never scale below one replica. KEDA, however, scales based on the actual workload (the queue depth) and can scale to zero. When the queue is empty, the model pod is terminated, and the idle cost drops to zero. This scale-to-zero mechanism is exactly how serverless, pay-per-inference platforms operate.</p><h3>Step 1: Deploy Ollama with a persistent model disk</h3><p>We will be deploying a Deployment named ollama in its own namespace. The manifests expect the namespace to exist, so we create it first, then the disk, followed by the deployment:</p><p><a href="https://github.com/TheInvincibleRalph/Enterprise-grade-GKE-Project/tree/master/kubernetes/ai">Enterprise-grade-GKE-Project/kubernetes/ai at master · TheInvincibleRalph/Enterprise-grade-GKE-Project</a></p><pre>kubectl create namespace ai-inference<br>kubectl apply -f kubernetes/ai/models-pvc.yaml  <br>kubectl apply -f kubernetes/ai/ollama-deployment.yaml</pre><p>The deployment mounts the claim ollama-models, a 10 Gi disk on standard-rwo, at the model directory. The disk is what makes scale-to-zero workable. A pod can be deleted at any moment, and a scale-to-zero cluster deletes pods constantly. If the model lived in the pod, every wake-up would re-download 2 GB before the first answer. On the disk, the cold start is a load from disk, not a download, so the model outlives every pod that serves it.</p><p>Pull the model onto the disk now, before KEDA exists. Once KEDA is live, an empty queue drops the pod to zero after 60 seconds, and with no pod there is nothing to exec into:</p><pre>kubectl exec -n ai-inference deploy/ollama -- ollama pull llama3.2:3b</pre><p>Let’s test that our model is active by asking it something.</p><pre>kubectl port-forward -n ai-inference svc/ollama 11434:11434<br><br># then in a second terminal:<br>curl http://localhost:11434/api/generate \<br>  -d &#39;{&quot;model&quot;: &quot;llama3.2:3b&quot;, &quot;prompt&quot;: &quot;Say hello in five words.&quot;, &quot;stream&quot;: false}&#39;</pre><p>Both should work now, but after Step 3 the exec path will stop being reliable: KEDA may scale the deployment to zero, and there is no pod to exec into. After now, we will be using the Service name, ollama, in anything that needs to talk to the model.</p><h3>Step 2: The queue and the API</h3><p><a href="https://github.com/TheInvincibleRalph/Enterprise-grade-GKE-Project/blob/master/kubernetes/ai/redis-inference-api.yaml">Enterprise-grade-GKE-Project/kubernetes/ai/redis-inference-api.yaml at master · TheInvincibleRalph/Enterprise-grade-GKE-Project</a></p><pre>kubectl apply -f kubernetes/ai/redis-inference-api.yaml</pre><p>This applies Redis, the API, and their Services.</p><p>Why a queue at all? Because the model is slow. An answer takes seconds on CPU, and an HTTP call that blocks for that long ties up the caller. So between the caller and the model sits a Redis list named inference_queue. The API does not talk to the model. It pushes the prompt into the list and answers immediately, with three endpoints:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3IZ4SuTZw5A6aRTe8PeHpg.png" /></figure><ul><li>POST /infer: takes {&quot;prompt&quot;: ...}, pushes a job onto the list, returns a task_id.</li><li>GET /result/&lt;id&gt;: the answer once the worker wrote it, or processing while it is still being worked.</li><li>GET /health: the current queue length.</li></ul><p>The caller never waits on the model; it waits on the queue. And the queue is what KEDA watches. The API is the only entry point: nothing talks to Ollama directly except the worker from the next step.</p><h3>Step 3: KEDA and the worker</h3><p><a href="https://github.com/TheInvincibleRalph/Enterprise-grade-GKE-Project/blob/master/kubernetes/ai/queue-worker.yaml">Enterprise-grade-GKE-Project/kubernetes/ai/queue-worker.yaml at master · TheInvincibleRalph/Enterprise-grade-GKE-Project</a></p><pre>kubectl apply -f kubernetes/ai/queue-worker.yaml<br><br>helm repo add kedacore https://kedacore.github.io/charts<br>helm repo update<br><br>helm install keda kedacore/keda --namespace keda --create-namespace<br>kubectl apply -f kubernetes/ai/scaledobject.yaml</pre><p>The worker moves jobs from the queue to the model. Its loop: BRPOPLPUSH moves a job from inference_queue to inference_processing in one atomic step, it POSTs the prompt to http://ollama:11434/api/generate, stores the answer as result:&lt;id&gt;, and removes the job from the processing list.</p><p>KEDA scales on inference_processing, the list of jobs being worked, not on inference_queue. The raw queue looks empty the moment the worker pops a job, even though the model is still generating an answer. If KEDA watched the raw queue, it would scale the model down mid-generation. The job sits in the processing list from the moment the worker picks it up to the moment the answer is stored, so KEDA keeps the pod alive for exactly as long as there is real work.</p><blockquote>Try to go through the <a href="https://github.com/TheInvincibleRalph/Enterprise-grade-GKE-Project/blob/master/kubernetes/ai/redis-inference-api.yaml">reference-api</a> and <a href="https://github.com/TheInvincibleRalph/Enterprise-grade-GKE-Project/blob/master/kubernetes/ai/queue-worker.yaml">queue-worker</a> and the manifests, and these things will get clearer.</blockquote><p>Now to <strong>ScaledObject</strong>, which is a Kubernetes custom resource that comes with KEDA. It tells KEDA what to scale, how far, and on what signal.</p><p>Here, what to scale is the field scaleTargetRef and it points to the Deployment, ollama. minReplicaCount: 0 and maxReplicaCount: 1 are the bounds, zero at idle, one at most. The redis trigger watches inference_processing, and the tuning fields say how it behaves, cooldownPeriod: 60 (how long after the queue empties before scaling down) and pollingInterval: 10 (how often KEDA checks the queue).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-nohN5dhzAxuc2eVVQs6eg.png" /></figure><h3>Step 4: The scale-to-zero demo</h3><p>Push 20 jobs, from inside the API pod:</p><pre>kubectl port-forward -n ai-inference svc/inference-api 8080:8080<br><br># then in a second terminal:<br>for i in $(seq 1 20); do<br>  curl -s -X POST http://localhost:8080/infer -H &#39;Content-Type: application/json&#39; \<br>    -d &quot;{\&quot;prompt\&quot;: \&quot;What is Kubernetes? job $i\&quot;}&quot;<br>done</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PW-Ar0S4ReXS_0IIOr7HIA.png" /></figure><pre>kubectl get deploy ollama -n ai-inference -w    # 0 -&gt; 1 while a job is in flight, -&gt; 0 after 60s quiet</pre><p>The lifecycle: no jobs means no Ollama at all; the first job wakes it, and 60 seconds without a job shuts it down again. One worker works one job at a time, so one pod is enough, and the disk allows only one anyway.</p><p>The stack works now, but only inside the cluster. The next step gives it a hostname.</p><h3>Step 5: How a backend calls the inference server</h3><p>In production, the consumer of an inference server is almost always code: another service, a pipeline, a backend. A product description generator, a ticket classifier, a nightly summarization job. The pattern is the same in every case: the caller submits a prompt and collects the answer when it is ready, and nobody watches it run.</p><p>The client side is short. This runs from the API pod, the same place a backend service on the cluster would call from:</p><p>Before running the client script, run the reset script below so the queue starts empty, and the answer comes back early.</p><p>Reset script:</p><p><a href="https://github.com/TheInvincibleRalph/Enterprise-grade-GKE-Project/blob/master/scripts/ai-demo-reset.sh">Enterprise-grade-GKE-Project/scripts/ai-demo-reset.sh at master · TheInvincibleRalph/Enterprise-grade-GKE-Project</a></p><pre>./scripts/ai-demo-reset.sh</pre><p>Client script:</p><pre>kubectl exec -n ai-inference deploy/inference-api -- python -c &quot;<br>import urllib.request, json, time<br><br>def submit(prompt):<br>    req = urllib.request.Request(&#39;http://inference-api:8080/infer&#39;,<br>        data=json.dumps({&#39;prompt&#39;: prompt}).encode(),<br>        headers={&#39;Content-Type&#39;: &#39;application/json&#39;})<br>    return json.loads(urllib.request.urlopen(req).read())[&#39;task_id&#39;]<br>def wait_for(task_id):<br>    while True:<br>        res = json.loads(urllib.request.urlopen(<br>            &#39;http://inference-api:8080/result/&#39; + task_id).read())<br>        if &#39;response&#39; in res:<br>            return res[&#39;response&#39;]<br>        print(&#39;job in flight, waiting...&#39;)<br>        time.sleep(2)<br>task = submit(&#39;Write a product description for a reusable water bottle.&#39;)<br>print(wait_for(task))<br>&quot;</pre><p>Run it once, and you see the lifecycle from the caller’s side. The client submits, then prints job in flight, waiting... while the worker and the model start, then the answer appears. To the client, it is a request that takes longer than usual. To the cluster, it is a job that woke the model from zero, and 60 seconds after the queue empties, the model goes back to zero.</p><p>This is the code a pipeline runs in a loop over a dataset: submit all the prompts, collect the results, write them back to the database.</p><blockquote>PS: The code job above might take a while to return if run almost immediatly after the scale-to-zero demo in Step 4 and if didn’t run the reset script above.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0t4a9H3DXOScKk8p6Z21Iw.png" /></figure><h3>What’s next?</h3><p>With that, the project is complete. Every promise from the overview is delivered: the storefront, the database, the observability, the chaos experiments, and the model. In the next post, I’ll be sharing a kill switch to take down all the resources in the cluster in just one click and how the whole project is valuable to your portfolio.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7c89a4081461" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[Part 9] Chaos Engineering with Chaos Mesh]]></title>
            <link>https://medium.com/@theinvincibledev/part-9-chaos-engineering-with-chaos-mesh-80956e7f4236?source=rss-ddb6d9aa3746------2</link>
            <guid isPermaLink="false">https://medium.com/p/80956e7f4236</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[chaos-monkey]]></category>
            <category><![CDATA[chaos-engineering]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[TheInvincible]]></dc:creator>
            <pubDate>Wed, 19 Aug 2026 11:56:13 GMT</pubDate>
            <atom:updated>2026-08-19T11:56:13.715Z</atom:updated>
            <content:encoded><![CDATA[<h4>Proving our cluster can survive real-world uncertainty and stay within defined reliability objectives</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fj50K77XjE92qqWqYqgqXA.png" /></figure><p>In Part 6, the primary was killed with a single kubectl delete pod, run by hand every time. <strong>Chaos Mesh</strong> is the CNCF project that turns fault injection into declarative CRD objects that inject pod kills, network latency, and node failures on command. We will use the dashboards from <a href="https://medium.com/@theinvincibledev/part-7-observability-with-prometheus-grafana-b5159196d598?postPublishedType=initial">Part 7</a> and <a href="https://medium.com/@theinvincibledev/part-8-optional-observability-with-datadog-e8562102988e?sharedUserId=theinvincibledev">Part 8</a> as the evidence layer, and the steady-state hypothesis as the rule: to be measured before, during and after the chaos injection.</p><h3>Step 1: Install Chaos Mesh</h3><pre>helm repo add chaos-mesh https://charts.chaos-mesh.org<br>helm repo update<br><br>helm upgrade --install chaos-mesh chaos-mesh/chaos-mesh \<br>  --namespace chaos-mesh --create-namespace \<br>  --set chaosDaemon.runtime=containerd \<br>  --set chaosDaemon.socketPath=/run/containerd/containerd.sock \<br>  --set dashboard.create=true<br><br>kubectl port-forward -n chaos-mesh svc/chaos-dashboard 2333:2333</pre><p>The Helm chart installs the controller, the web UI, and the chaos-daemon on every node.</p><p>Then visit <a href="http://localhost:2333/">http://localhost:2333</a>. There, you will be asked to generate a token.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9gLcEn5-gMNtsGsmD4A-bw.png" /></figure><p>Once you click on Click here to generate you’ll be taken to another pop-up where an RBAC manifest is already generated. Follow the instructions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9bKzSawGOMAsuHR7RAWlUw.png" /></figure><blockquote>Note that you’d save the manifest as rbac.yaml in the project folder and run the command to apply it. The manifest changes every few seconds, so make sure the commands you are running tally with what is in the RBAC you copied; e.g if your service account name in the manifest you copied is account-cluster-manager-yprvu , make sure the command to generate the token is against that Service Account (the exact random suffix); else you’ll get an error.</blockquote><p>Afterwards, close that pop-up and paste the generated token on your terminal into the token box. You can give it any name. Then <strong>Submit</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VrRD3Jpm3ruRo_L5xtblMQ.png" /></figure><p>You will then be taken to the homepage.</p><h3>Step 2: Chaos Experiment.</h3><p>For a start, we will be injecting one of our database pods with “chaos” to prove the failover. Navigate to <strong>Experiments</strong>. Then <strong>Kubernetes</strong> → <strong>Pod Fault</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b_rvPFyYY-qLrLEvbKJkeQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B4M2DpgJDEzrYlcBKaaVjw.png" /></figure><p>Pick <strong>Pod Kill</strong>, then from the<strong> Namespace </strong>and <strong>Label Selectors </strong>dropdowns, pick<strong> </strong>database<strong> </strong>and ha-postgres-1<strong>, </strong>respectively. Make the Metadata <strong>Name</strong>: database-chaos-injection . Then <strong>Submit</strong>. The experiment will start, and you can check the details by clicking on the experiment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2UCTwL-MdbP5A0Q1EjncOA.png" /></figure><p>To confirm the pod was killed and that our failover mechanism worked, we will check our Datadog logs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dDJC2MzdyYQWlaLBOfztBA.png" /></figure><ol><li><em>1 Applied: Successfully apply chaos for database/ha-postgres-1</em>proves the database-chaos-injection Pod was applied.</li><li><em>FailingOver: Current primary isn’t healthy, initiating a failover from ha-postgres-1 5 FailoverTarget: Failing over from ha-postgres-1 to ha-postgres-2 </em>proves our failover worked and ha-postgres-2 was elected as the new leader after ha-postgres-1 was killed.</li></ol><h3>Step 2: Define the steady state</h3><p>Before any fault, we need to write down what “normal” means; we will do that as a versioned ConfigMap. So these are the targets measured before and after every experiment:</p><pre># chaos/steady-state-check.yaml<br>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: steady-state-check<br>  namespace: chaos-mesh<br>data:<br>  boutique_app: |<br>    p99_latency: &quot;&lt; 500ms&quot;<br>    error_rate: &quot;&lt; 1%&quot;<br>    availability: &quot;&gt; 99.5%&quot;<br>  postgres: |<br>    replication_lag: &quot;&lt; 100ms&quot;<br>    failover_time: &quot;&lt; 30s&quot;<br>    data_loss: &quot;0&quot;</pre><p>Apply it:</p><pre>kubectl apply -f steady-state-check.yaml</pre><h3>Step 3: Experiment 2, killing the storefront</h3><p>Grab the pod-kill-app.yaml here. It kills 50% of the pods in the boutique namespace.</p><p><a href="https://github.com/TheInvincibleRalph/Enterprise-grade-GKE-Project/blob/master/chaos/pod-kill-app.yaml">Enterprise-grade-GKE-Project/chaos/pod-kill-app.yaml at master · TheInvincibleRalph/Enterprise-grade-GKE-Project</a></p><pre>kubectl apply -f pod-kill-app.yaml</pre><p>Apply it and watch: the frontend pods die, and the Deployment replaces them. Before the kill, the pods were <strong>12</strong>, the steady value. But during the kill, the pods increase to <strong>15</strong> because the Deployment immediately spun up more pods as the others were killed.</p><p>Before:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dBrxwUIAUB--DBvxODoZ_A.png" /></figure><p>During:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M2lRY3o9HU7Rld4U4vskFg.png" /></figure><p>Then, after some time, the number of pods will go back to <strong>12</strong>, the steady value.</p><p>Checking against our steady state:</p><ul><li><strong>p99 latency</strong> — the WAF dashboard&#39;s latency panel in <a href="https://medium.com/@theinvincibledev/part-7-observability-with-prometheus-grafana-b5159196d598?postPublishedType=initial">Part 7</a>, read at p99, it should be under <strong>500 ms</strong> before, during, and after. Mine was <strong>25.0</strong> ms before and <strong>49.7</strong> ms during the chaos experiment.</li><li><strong>Availability</strong> — this short probe loop counts the htttp codes:</li></ul><p>for i in $(seq 1 20); do curl -s -o /dev/null -w &#39;%{http_code}\n&#39; https://boutique.invincibledevops.tech/; done | sort | uniq -c</p><blockquote>NB: replace domain with yours.</blockquote><p>Mine was 20 out of 20 200 which is <strong>100% </strong>uptime. Our <strong>99.5%</strong> target allows at most a couple of failures out of hundreds, so our reliability requirement passes.</p><h3>Step 4: Experiment 3, node failure</h3><p>This is an OPERATIONAL script (not a Chaos Mesh manifest): cordon + drain, and it is the realistic way to simulate a node/zone loss.</p><p>Grab the script, then run the command below.</p><p><a href="https://github.com/TheInvincibleRalph/Enterprise-grade-GKE-Project/blob/master/chaos/node-drain.sh">Enterprise-grade-GKE-Project/chaos/node-drain.sh at master · TheInvincibleRalph/Enterprise-grade-GKE-Project</a></p><pre>./chaos/node-drain.sh</pre><p>It cordons an application node and drains it: the storefront pods reschedule, and if the node held the primary, the failover runs again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9rJ4FqMPnoN0ihYK3zl9ow.png" /></figure><h3>What’s next?</h3><p>The fault injection now covers everything this series built: the storefront, the database, the network, the nodes. In the next part, we will deploy an AI inference service on the cluster, with Kubernetes-based Event Driven Autoscaler scaling it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=80956e7f4236" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[Part 8][Optional] Observability with Datadog]]></title>
            <link>https://medium.com/@theinvincibledev/part-8-optional-observability-with-datadog-e8562102988e?source=rss-ddb6d9aa3746------2</link>
            <guid isPermaLink="false">https://medium.com/p/e8562102988e</guid>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[datadog]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[TheInvincible]]></dc:creator>
            <pubDate>Tue, 18 Aug 2026 11:36:53 GMT</pubDate>
            <atom:updated>2026-08-18T11:36:53.858Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Datadog Agents, dashboards, monitors, and event alerts.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xtnCIDrHWzG9QESit_STug.png" /></figure><p><a href="https://medium.com/@theinvincibledev/part-7-observability-with-prometheus-grafana-b5159196d598?sharedUserId=theinvincibledev">In Part 7</a>, Prometheus scraped in-cluster endpoints, and Grafana rendered them. Datadog works differently: it deploys a DaemonSet of Agent pods that stream metrics, logs, and process data to the Datadog SaaS, where dashboards and monitors are created.</p><h4>Step 1: Create the account</h4><ol><li>Sign up at app.datadoghq.com through the GitHub Student Pack’s Datadog benefit, which grants free 2-year credits.</li><li>After logging in, look at the URL. The site segment is your region: datadoghq.com (US1), us3.datadoghq.com, us5.datadoghq.com, datadoghq.eu.</li><li>Grab your API key. Organization Settings → API Keys → New Key. Then copy it.</li></ol><h4>Step 2: Install the Agent</h4><p>The chart installs three things: the Agent as a DaemonSet (one pod per node), the cluster agent, and the cluster agent’s metrics provider.</p><pre>export DD_API_KEY=&quot;&lt;your-key&gt;&quot; # replace with your API key.<br><br>helm repo add datadog https://helm.datadoghq.com<br>helm repo update<br>helm upgrade --install datadog datadog/datadog \<br>  --namespace datadog --create-namespace \<br>  --set datadog.apiKey=&quot;$DD_API_KEY&quot; \<br>  --set datadog.site=&quot;&lt;region&gt;.datadoghq.com&quot; \ # e.g us5.datadoghq.com<br>  --set datadog.apm.enabled=true \<br>  --set datadog.logs.enabled=true \<br>  --set datadog.logs.containerCollectAll=true \<br>  --set datadog.processAgent.enabled=true \<br>  --set datadog.clusterAgent.enabled=true \<br>  --set datadog.clusterAgent.metricsProvider.enabled=true \<br>  --set providers.gke.cos=true</pre><h4>Step 3: Verify</h4><pre>kubectl get pods -n datadog    # one agent per node + the cluster agent</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XYObqKvZBxs8iJargmZuTw.png" /></figure><h4>Step 4: Build the dashboards with Bits</h4><p>We will not be building the dashboards by hand. We will use Bits, Datadog’s in-product AI assistant. <strong>Bits</strong> reads the Datadog API for what the agent in each node ships, writes the dashboard definitions, and saves them.</p><p>Feed Bits with this prompt:</p><blockquote>Build a full-stack monitoring dashboard for my Kubernetes cluster kube_cluster_name:devops-portfolio. Cover infrastructure (CPU, memory, disk, load), Kubernetes health (nodes, pods, deployments, restarts), application workloads (Online Boutique in kube_namespace:boutique), database (ha-postgres in kube_namespace:database), network, and logs. Anything that is trackable.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I8Mej1JV9LKSlYcSiL8fhg.png" /></figure><p>This is the resulting dashboard with the different sections.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w0wbtbl32daKFnlQL-kseA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rgwYjODtfY0Aim91sS6Nbg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eNE11HBPgitvsgIiNBuM_Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pj3J_swzN-IOJrqt92SBOg.png" /></figure><p>You can scope the dashboard view to a namespace or a deployment or host ($kube_namespace, $kube_deployment, $host).</p><p>Build a failover dashboard to track the failover. Feed Bits with this prompt:</p><blockquote>Build a dashboard named “Database Failover Validation” to watch the PostgreSQL failover event for the ha-postgres cluster in kube_namespace:database: transactions per second, active connections, replication lag, and pod readiness, scoped to kube_cluster_name:devops-portfolio.</blockquote><p>Before failover test:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1jqoDlJQnWSkTfUl8bW0Yg.png" /></figure><p>After the failover (in the next section):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EL6igbwU066iNFoM39iUXw.png" /></figure><p>During the failover, CNPG acts on two fronts at once: the operator elects a new primary from the replicas, and the killed pod is recreated immediately with the same name, so the instance count never drops. The recreated pod comes back as a replica and rejoins the cluster.</p><h4>Step 5: Setting Up an Alert for Failover</h4><p>We will be setting up a monitor specifically for our database failover event. A monitor is a query with a threshold and a notification.</p><blockquote>Create a metric monitor named &quot;PostgreSQL Failover Detected&quot;, scoped to kube_cluster_name:devops-portfolio, kube_namespace:database, and condition:true. Group by pod_name so each pod is monitored individually. Use min aggregation over a 1-minute window, alerting when it drops below 1. Set require_full_window: false. Notify @&lt;your-email&gt;, set priority P1, link to the Full-Stack Overview dashboard.</blockquote><p>Replace &lt;your-email&gt; with your email address.</p><p>Then run the test:</p><pre>./scripts/test-failover.sh</pre><p>Monitor screen:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HdpjM_CCl-3bWxJsR-SB3A.png" /></figure><p>I got the alerts in my mailbox, one for the failure and another for the resolution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IWbycX1kgkKCfcsildJofQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y-J6xBk15cAvlKET7wxU5w.png" /></figure><h3>What’s next?</h3><p>Up next, we will be using <strong>Chaos Mesh</strong> for repeatable chaos experiments: pod kills, network latency, CPU stress, node failures, each one validating a steady-state hypothesis.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e8562102988e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[Part 7] Observability with Prometheus & Grafana]]></title>
            <link>https://medium.com/@theinvincibledev/part-7-observability-with-prometheus-grafana-b5159196d598?source=rss-ddb6d9aa3746------2</link>
            <guid isPermaLink="false">https://medium.com/p/b5159196d598</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[TheInvincible]]></dc:creator>
            <pubDate>Mon, 17 Aug 2026 11:32:18 GMT</pubDate>
            <atom:updated>2026-08-17T11:32:18.888Z</atom:updated>
            <content:encoded><![CDATA[<h4>In Part 6, we observed our failover test via a poll loop that printed one line every two seconds. In this part, we replace the terminal with a monitoring stack: <strong>Prometheus</strong> scrapes metrics from every component, and <strong>Grafana</strong> turns the metrics into dashboards.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*euPBiwniZZo22R1Ff0diCQ.png" /></figure><h3>Prometheus Exporters</h3><p>Ever wondered how Grafana can render metrics from across your entire infrastructure without complex logic on your side?</p><p>As we will see with CloudNativePG and Cilium, modern cloud-native tools make this effortless by exposing their telemetry through native exporters, each serving its metrics on an HTTP endpoint. Because Prometheus is a pull-based metrics server, it scrapes those endpoints (/metrics or /stats/prometheus) on a schedule and stores the samples in a time series database.</p><h4>Step 1: Installing Prometheus &amp; Grafana</h4><pre>helm repo add prometheus-community https://prometheus-community.github.io/helm-charts<br>helm repo update<br><br>helm upgrade --install monitoring prometheus-community/kube-prometheus-stack \<br>  --namespace monitoring --create-namespace \<br>  --set grafana.adminPassword=&quot;admin&quot; \<br>  --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \<br>  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false<br><br>kubectl get pods -n monitoring     # prometheus, operator, grafana, alertmanager, exporters Running</pre><h4>Step 2: CloudNativePG Exporter</h4><p><strong>CloudNativePG</strong> ships a metrics exporter in every instance and can generate a PodMonitor for it, but it is off by default. To switch it on, run this command:</p><pre>kubectl patch cluster -n database ha-postgres --type=merge \<br>  -p &#39;{&quot;spec&quot;:{&quot;monitoring&quot;:{&quot;enablePodMonitor&quot;:true}}}&#39;</pre><pre>kubectl get podmonitor -n database # ha-postgres<br>kubectl get podmonitor -n database ha-postgres -o yaml</pre><p>The PodMonitor is a Custom Resource Definition object that tells Prometheus to bypass Kubernetes Services and scrape the individual PostgreSQL pods directly on port 9187. The reason you get ha-postgres as the name of the PodMonitor is that CloudNativePG names it after the cluster.</p><p>Now check that Prometheus sees the exporter</p><pre>kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-prometheus 9090:9090</pre><p>Then <a href="http://localhost:9090/targets">http://localhost:9090/targets</a>. Search for “ha-postgres”. Then you should see targets UP on port 9187.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dtw5tR6QpjlYcoEKiusBuw.png" /></figure><h4>Step 3: Cilium Exporter</h4><p>We installed <strong>Cilium</strong> in Part 4 with prometheus.enabled: true already on, but serviceMonitor deliberately left off with a comment to revisit it.</p><p>Now, let’s switch it on:</p><pre>helm upgrade --install cilium cilium/cilium \<br>  --namespace kube-system \<br>  --version 1.18.12 \<br>  --values kubernetes/cilium/values.yaml \<br>  --set prometheus.serviceMonitor.enabled=true</pre><h4>Step 4: The PostgreSQL dashboard</h4><p>Grafana is now listening on the monitoring-grafana service:</p><pre>kubectl port-forward -n monitoring svc/monitoring-grafana 8080:80</pre><p>Go to <a href="http://localhost:8080/">http://localhost:8080</a>, use admin and the username and admin as the password.</p><blockquote>Set up a new password when prompted, or just skip.</blockquote><p>To import our PostgreSQL dashboards, go to <strong>Dashboards</strong> → <strong>New</strong> → <strong>Import dashboard</strong>→ <strong>20417</strong> as the ID → then, Load. Datasource is Prometheus.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m4nN8CeIFYIwzr9pd7XGkw.png" /></figure><p>The CloudNativePG’s official dashboard will be rendered, which includes transactions per second, connection count, replication lag per replica, cache hit ratio, and WAL generation rate.</p><p>For the <strong>Server Health</strong> section:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L0F_SzpTP6kPGfmbnZ6UPQ.png" /></figure><p>The <strong>Configuration</strong> section:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NZHY8dyWB4QNy8Lh1mjbNA.png" /></figure><p>There are other sections to explore as well: Operational Stats, Storage I/O, Write Ahead Log, etc.</p><p>Let’s test the failover from <a href="https://medium.com/@theinvincibledev/part-6-deploying-a-highly-available-postgresql-on-kubernetes-with-cloudnativepg-load-testing-it-92b64a304393">part 6</a>. Run the failover script, then refresh the dashboard.</p><pre>./scripts/test-failover.sh</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*euPBiwniZZo22R1Ff0diCQ.png" /></figure><p>We see that the leader pod ha-postgres-1 is down because it was killed, and then another leader was elected, which is ha-postgres-2 because it now carries the other 2 replicas. Refresh again, and you see that ha-postgres is back up as a replica. The continuous write load is proven by the high CPU utilization, 157% in this case.</p><h4>Step 5: The Envoy metrics pipeline</h4><p>Envoy Gateway already exports rich metrics, but for the sake of discovery, we need to do two things: deploy a Service that exposes the metrics port, and deploy a ServiceMonitor that tells Prometheus to scrape it.</p><p>Grab the manifest and run:</p><p><a href="https://github.com/TheInvincibleRalph/Enterprise-grade-GKE-Project/blob/master/kubernetes/envoy/exporter.yaml">Enterprise-grade-GKE-Project/kubernetes/envoy/exporter.yaml at master · TheInvincibleRalph/Enterprise-grade-GKE-Project</a></p><pre>kubectl apply -f kubernetes/envoy/exporter.yaml </pre><p>Once again, check to see if the target comes up</p><pre>kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-prometheus 9090:9090</pre><p>Then go to <a href="http://localhost:9090/targets">http://localhost:9090/targets</a> and search for “envoy”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7-_5ZtL4_7bXb1hyxqeaxQ.png" /></figure><p>Then generate traffic so the panels have data:</p><pre>curl -s https://boutique.invincibledevops.tech/ &gt; /dev/null<br>./scripts/waf-test.sh https://boutique.invincibledevops.tech/</pre><blockquote>You can run the script multiple times to see it live on the dashboard coming up next.</blockquote><h4>Step 6: The WAF dashboard</h4><p>The WAF dashboard has three panels; to import it, follow these steps:</p><p><a href="https://github.com/TheInvincibleRalph/Enterprise-grade-GKE-Project/blob/master/kubernetes/envoy/waf-dashboard.json">Enterprise-grade-GKE-Project/kubernetes/envoy/waf-dashboard.json at master · TheInvincibleRalph/Enterprise-grade-GKE-Project</a></p><ol><li>In Grafana: <strong>Dashboards → New → Import</strong>.</li><li>Paste the contents of kubernetes/envoy/waf-dashboard.json into the import box, or upload the file with the upload button.</li><li>Click <strong>Load</strong>. The data source field comes pre-bound to the Prometheus datasource UIDprometheus, so confirm and click <strong>Import</strong>.</li></ol><p>The dashboard renders immediately.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e0Oaw4wIFvfCfwkvQyQvtg.png" /></figure><h3>What’s next?</h3><p>The cluster is now watched continuously with Grafana. In the next part, we will put the same telemetry on <strong>Datadog</strong> for enterprise-grade monitoring; it will, however, be optional for anyone who doesn’t have a Datadog account and cannot afford one.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b5159196d598" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[Part 6] Deploying a Highly Available PostgreSQL on Kubernetes with CloudNativePG, Load Testing It…]]></title>
            <link>https://medium.com/@theinvincibledev/part-6-deploying-a-highly-available-postgresql-on-kubernetes-with-cloudnativepg-load-testing-it-92b64a304393?source=rss-ddb6d9aa3746------2</link>
            <guid isPermaLink="false">https://medium.com/p/92b64a304393</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[TheInvincible]]></dc:creator>
            <pubDate>Fri, 14 Aug 2026 12:19:06 GMT</pubDate>
            <atom:updated>2026-08-14T12:25:48.584Z</atom:updated>
            <content:encoded><![CDATA[<h3>[Part 6] Deploying a Highly Available PostgreSQL on Kubernetes with CloudNativePG, Load Testing It and Proving Failover</h3><h4><em>Leader election, synchronous replication, and self-healing storage.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LNbEtHLUFY5g-6buwhOiRg.png" /></figure><h3>Why CloudNativePG?</h3><p>CloudNativePG is the Kubernetes operator that handles the full lifecycle of a highly available PostgreSQL database cluster. It provides cloud native capabilities like self-healing, high availability, rolling updates, scale up/down of read-only replicas, affinity/anti-affinity/tolerations for scheduling, and everything else a human operator would do to deploy and manage a Postgres database within the Kubernetes environment.</p><p>Features we would be seeing in the implementation:</p><ol><li><strong>Leader election:</strong> This means the operator decides who the leader is, watches it, and promotes a replica automatically when it dies.</li><li><strong>Synchronous replication</strong>: The leader refuses to acknowledge a write until at least one replica has it on disk, so a leader that dies mid-write loses <em>nothing that was never acknowledged</em>.</li><li><strong>Storage grows in place</strong>: The operator expands the underlying disk live when you bump the size in the spec. No downtime.</li></ol><h4>Step 1 — Install the operator</h4><pre>kubectl apply --server-side --force-conflicts \<br>  -f https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.24/releases/cnpg-1.24.0.yaml</pre><pre># verify ALL 6 CRDs landed<br>kubectl get crd | grep cnpg<br><br># verify the controller has come up:<br>kubectl get pods -n cnpg-system</pre><h4>Step 2— Configure a Workload Identity</h4><p>The cluster’s pods need to be able to write to Google Cloud Storage as a service account. The chain:</p><pre># 1. a dedicated SA with objectAdmin on the bucket<br>gcloud iam service-accounts create pg-backup-sa \<br>  --display-name=&quot;CNPG GCS backups&quot;<br><br>gcloud projects add-iam-policy-binding devops-portfolio-prod \<br>  --member=&quot;serviceAccount:pg-backup-sa@devops-portfolio-prod.iam.gserviceaccount.com&quot; \<br>  --role=&quot;roles/storage.objectAdmin&quot;<br><br>gcloud storage buckets create gs://devops-portfolio-backups \<br>  --project=devops-portfolio-prod <br><br># 2. tell the namespace and the SA about the mapping<br>kubectl create ns database<br>kubectl annotate namespace database iam.gke.io/gke-metadata-server-enabled=&quot;true&quot;<br>kubectl annotate serviceaccount default -n database \<br>  iam.gke.io/gcp-service-account=pg-backup-sa@devops-portfolio-prod.iam.gserviceaccount.com<br><br># 3. allow that Kubernetes identity to act as pg-backup-sa<br>gcloud iam service-accounts add-iam-policy-binding \<br>  pg-backup-sa@devops-portfolio-prod.iam.gserviceaccount.com \<br>  --member=&quot;serviceAccount:devops-portfolio-prod.svc.id.goog[database/default]&quot; \<br>  --role=&quot;roles/iam.workloadIdentityUser&quot;</pre><h4>Step 3— Provision the database</h4><p><a href="https://github.com/TheInvincibleRalph/Enterprise-grade-GKE-Project/blob/master/kubernetes/cnpg/postgres-cluster.yaml">Enterprise-grade-GKE-Project/kubernetes/cnpg/postgres-cluster.yaml at master · TheInvincibleRalph/Enterprise-grade-GKE-Project</a></p><pre>kubectl apply -f kubernetes/cnpg/postgres-cluster.yaml<br><br>kubectl get pods -n database -o wide     <br>kubectl get cluster -n database         <br>kubectl get pvc -n database</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jMJCvGd3LahiNKK0KVIz6A.png" /></figure><h4>The following resources were provisioned</h4><ol><li><strong>CloudNativePG operator</strong> (cnpg-system namespace) with its 6 CRDs</li><li><strong>Cluster </strong><strong>ha-postgres</strong> (database namespace), 3 instances, PostgreSQL 16.4</li><li><strong>pg-backup-sa</strong> , a GCP service account bound to the cluster via Workload Identity</li><li><strong>gs://devops-portfolio-backups</strong>, the GCS bucket backups land in</li></ol><p>Here is the entire database topology, replication, storage, and backups</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t85mBr70O07NgB4WkpVdrg.png" /></figure><p><strong>To understand what this config achieves, I will highlight the decisions that shaped it.</strong></p><p>1. instances: 3 : <strong>one leader, and two replicas</strong></p><p>The operator runs three Postgres pods: ha-postgres-1 (the leader), ha-postgres-2 and ha-postgres-3 (replicas). Exactly one pod carries the cnpg.io/role=primary label which tells which of them is the leader. It also creates the read-write service (ha-postgres-rw, always pointed at the leader) and the read-only service (ha-postgres-ro, spread across replicas).</p><p><strong>2. One instance per zone</strong></p><p>enablePodAntiAffinity + topologyKey: topology.kubernetes.io/zone tells the scheduler to never co-locate two of these pods in the same availability zone, so a zone failure takes down exactly one instance.</p><p><strong>3. Synchronous replication: </strong>synchronous_standby_names = &#39;ANY 1 (ha-postgres-2, ha-postgres-3)&#39;</p><p>With this, the leader <strong>waits for at least one replica to confirm a write to disk before acknowledging the client, </strong>so that the unacknowledged commit is not lost. The first replica to ack makes the commit durable; then the third replica catches up asynchronously. The trade-off is a write-latency cost (one round trip to a replica per commit). This makes sure that when the leader dies, the promoted replica already holds every transaction the clients were told was committed.</p><p><strong>4. Backups via barman and Workload Identity</strong></p><p>The barmanObjectStore points at GCS (gs://devops-portfolio-backups/postgres, WAL compressed with gzip, 30-day retention). Barman uses the pod&#39;s default credentials (Workload Identity credentials), bound to a dedicated GCS service account.</p><h3>Replication and Failover</h3><p>To prove our assertion of leader election and synchronous replication, we need to put the database under a continuous write load, kill the leader mid-write, and watch the operator promote another leader.</p><p>But why break things on purpose? Because breaking it allows us to test the reliability of a system.</p><p>These will be our assertions.</p><ul><li><strong>A — </strong>Synchronous replication: a leader that dies mid-write loses <em>nothing that was acknowledged</em>, because no write is acknowledged until at least one replica has it on disk.</li><li><strong>B</strong> — killing the leader mid-load causes <strong>zero failed transactions</strong>: everything the clients were told was committed survives, and the promotion is transparent.</li></ul><p>Two tools that do the work:</p><ul><li><strong>pgbench, p</strong>ostgres load generator.</li><li><strong>kubectl delete pod --grace-period=0 --force</strong> — the chaos injection. The harshest way to kill a pod: no SIGTERM, no shutdown sequence.</li></ul><h4>Step 5 — Initialize the test database</h4><p>The kill test writes through pgbench, so its tables have to exist first:</p><pre># Get the leader pod — the label is the source of truth<br>LEADER_POD=$(kubectl get pods -n database -l cnpg.io/instanceRole=primary \<br>  -o jsonpath=&#39;{.items[0].metadata.name}&#39;)<br><br># Initialize pgbench&#39;s tables (&quot;app&quot; is the database name)<br>kubectl exec -n database $LEADER_POD -- pgbench -i -U postgres app</pre><h4>Step 6: Kill the leader, and watch the cluster heal</h4><p>The experiment, in one script — continuous load in the background, then a forced kill of the leader:</p><p><a href="https://github.com/TheInvincibleRalph/Enterprise-grade-GKE-Project/blob/master/scripts/test-failover.sh">Enterprise-grade-GKE-Project/scripts/test-failover.sh at master · TheInvincibleRalph/Enterprise-grade-GKE-Project</a></p><pre>./scripts/test-failover.sh</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YfGji_L39jec7FYiAS-NMQ.png" /></figure><p>First we see pgbench generating a continuous write load into the leader, and the progress lines show a steady database: <em>319.3 tps at 10 seconds, 464.1 tps at 20 seconds, 465.7 tps at 30 seconds</em>, with 0 failed transactions at every point.</p><p>Then the leader is killed with no grace period, no SIGTERM, and the load dies with the pod. Then we see the new leader election: the cluster phase moves from <strong>Failing</strong> over into <strong>Waiting</strong> for the instances to become active while no pod carries the primary label, and about 20 seconds after the kill, ha-postgres-2 takes over and the cluster reports <strong>Cluster in healthy state</strong>. Every transaction acknowledged before the kill was safe.</p><h4>Growing the database</h4><p>Let’s patch the storage to 8G, then watch the databases grow in size without first shutting down. We’ll see each PVC flip to 8Gi with the pod’s status staying <strong>Running</strong> throughout.</p><pre>kubectl patch cluster -n database ha-postgres --type=merge \<br>  -p &#39;{&quot;spec&quot;:{&quot;storage&quot;:{&quot;size&quot;:&quot;8Gi&quot;,&quot;resizeInUseVolumes&quot;:true}}}&#39;<br><br>kubectl get pvc -n database -w    # watch them grow 1Gi -&gt; 8Gi, pods keep serving</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dmehpdaq72uvQuoeiagW5g.png" /></figure><p>resizeInUseVolumes: true in the operator is what grows the claim, the GCE disk, and the filesystem live, pod by pod, no restart.</p><h3>What’s next?</h3><p>We have proven that our database is fail-proof and resilient. In the next part, we will deploy our <strong>observability stack</strong> (Prometheus + Grafana with database and WAF dashboards, plus the Datadog integration) so we can have a view into our cluster.</p><blockquote><em>Finally, do me a favour. </em><strong><em>Follow</em></strong><em> me so you’d be the first to be notified when I make a Medium post, and give this post as many </em><strong><em>claps</em></strong><em> as you like so this project can reach a wide audience. Thank you!</em></blockquote><p>Let me know if you have any questions in the comment section.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=92b64a304393" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[Part 5] Configuring the Cluster’s Edge: Envoy Gateway with an In-Process Coraza WAF & Deploying a…]]></title>
            <link>https://medium.com/@theinvincibledev/part-5-configuring-the-clusters-edge-envoy-gateway-with-an-in-process-coraza-waf-deploying-a-5797dd62b092?source=rss-ddb6d9aa3746------2</link>
            <guid isPermaLink="false">https://medium.com/p/5797dd62b092</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[envoy-proxy]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[TheInvincible]]></dc:creator>
            <pubDate>Thu, 13 Aug 2026 11:01:01 GMT</pubDate>
            <atom:updated>2026-08-13T11:01:01.704Z</atom:updated>
            <content:encoded><![CDATA[<h3>[Part 5] Configuring the Cluster’s Edge: Envoy Gateway with an In-Process Coraza WAF &amp; Deploying a Microservices Application Behind the WAF.</h3><h4>In Part 4, we deployed Cilium — the eBPF dataplane that replaced kube-proxy, enforces NetworkPolicies, and encrypts pod-to-pod traffic with WireGuard. In this part, we will configure the cluster’s edge and deploy Google’s Online Boutique, and prove that our live infrastructure is attack-proof.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o6RXVSme1COaSlRg70dR3g.png" /><figcaption>Envoy Gateway architecture</figcaption></figure><h3>Why Envoy Gateway?</h3><p>If you have worked with Kubernetes before, you know the classic choice for a proxy was the NGINX Ingress Controller, but in March 2026, the open-source project got retired, and now we have to use something else.</p><p>Envoy Gateway is the actively maintained successor, and it is built around the <strong>Gateway API</strong>.</p><p>According to the Gateway API standard, routing is declared as four objects (GatewayClass → Gateway → HTTPRoute → Service). Envoy Gateway implements this standard and gives room for some extensions, which we would see later.</p><h3>Request Flow</h3><p>Before we go into the weeds of Envoy terminologies, let’s take a look at our architecture and get a glimpse of how traffic flows inside our cluster.</p><p><strong>1. Traffic (the data plane):</strong></p><pre>Browser → Cloudflare → GCP Load Balancer → Envoy (WAF + routing) → Boutique frontend → microservices</pre><ul><li>Cloudflare handles the HTTPS and sends plain HTTP to the load balancer.</li><li>The load balancer forwards to one of the three Envoy pods.</li><li>Inside that Envoy pod, three things happen in order: the <strong>listener</strong> accepts the request, the <strong>WAF</strong> checks it, and the <strong>router</strong> sends it to the frontend.</li><li>Then the frontend calls the other microservices on the backend.</li></ul><p>The function of the Envoy Gateway Controller is to read our YAML (Gateway, HTTPRoute, WAF policy), turn it into Envoy’s config language, and push it to the Envoy pods.</p><h3>Components of our Envoy Setup</h3><p><strong>1. The Worker (Envoy Proxy)</strong></p><p>Envoy Proxy is a high-performance, open-source network proxy. It sits in the “data plane,” meaning it actually handles the live traffic. It receives requests, performs load balancing, and forwards traffic to backend services. But configuring Envoy directly is notoriously difficult and requires deep networking expertise. A single configuration can easily reach 20,000 lines of complex code. That is where the Envoy Gateway chimes in.</p><p><strong>2. The Manager (Envoy Gateway)</strong></p><p>Envoy Gateway is a Kubernetes-native “control plane” built to manage Envoy. It orchestrates and configures Envoy instances. It watches simple Kubernetes commands (using the Gateway API standard) and automatically translates them into the complex instructions Envoy understands. It manages the lifecycle of Envoy, meaning it can automatically provision, update, and scale Envoy worker pods.</p><p><strong>3. The Inspector (Coraza WAF)</strong></p><p>Coraza is an open-source Web Application Firewall engine, compatible with the ModSecurity rule language. It sits in front of the application and inspects every request before the app sees it, running the <strong>OWASP Core Rule Set</strong>, the community rule collection for SQL injection, XSS, path traversal, command injection, and other classic attack classes.</p><p>The WAF is not a separate service. It loads into the Envoy process itself as a native module: the EnvoyProxy resource mounts libcomposer.so into every Envoy pod, and the EnvoyExtensionPolicy attaches the Coraza filter to the Gateway, so every route through it inherits the filter. Inside the pod, the request flow is: the listener accepts, the WAF inspects, the router forwards. A rule hit becomes a 403 before the request reaches any backend service.</p><h4>These are the files that make up our deployment</h4><p><strong>3. GatewayClass (</strong><strong>eg)</strong>: This is just a template that defines a specific type of network load balancer or proxy to be implemented within a cluster.</p><p><strong>Gateway (</strong><strong>boutique-gateway)</strong>: This is the declarative object that makes the controller run Envoy and provision its load balancer. It listens on port 80.</p><p><strong>HTTPRoute (</strong><strong>boutique-route)</strong>: This is a set of rules that define how HTTP traffic coming into a Gateway gets sent to the backend services.</p><p><strong>EnvoyProxy (</strong><strong>envoy-proxy)</strong>: It describes how the Envoy worker pods are built.</p><p><strong>EnvoyExtensionPolicy (</strong><strong>waf-extension)</strong>: the resource that attaches an extension filter, in our case Coraza, to a Gateway, so every route through it inherits the filter.</p><h3>Step 1 —Install Envoy Gateway</h3><p>Grab the script here:</p><p><a href="https://github.com/TheInvincibleRalph/Enterprise-grade-GKE-Project/tree/master">GitHub - TheInvincibleRalph/Enterprise-grade-GKE-Project</a></p><pre>./scripts/install-envoy-gateway.sh</pre><p>Under the hood, the script does three things:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bZniXlWsCdbHOBznqz1RVw.png" /></figure><p>The script then watches for the LoadBalancer IP (2–5 minutes) and prints it, with the next step:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4_Qj1usvMr9W_m6pwtOMOQ.png" /></figure><h3>Deploying a Microservices Application Behind a WAF</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VP8CiOlem_aoelR5rwnXSg.png" /><figcaption>Mcroservice deploment</figcaption></figure><p>We will be deploying Online Boutique, Google’s official microservices demo, which is a full e-commerce application built to show what real microservices look like: many different pods, all speaking to each other over the network.</p><p>It is the perfect application for this cluster, for two reasons:</p><ol><li><strong>It is microservice-shaped.</strong> Dozens of pods, real inter-service calls, and a stateful cache.</li><li><strong>It gives the WAF a real target.</strong> A working storefront with a search endpoint, query parameters, and forms is exactly the attack surface a production app has.</li></ol><h3>Step 2 — Deploy the application</h3><p>The cluster’s edge is already live from Step 1 now let’s deploy the storefront:</p><blockquote>I will be using my domain name from here, don’t forget to change it to yours.</blockquote><pre>BOUTIQUE_HOST=boutique.invincibledevops.tech ./scripts/deploy-boutique.sh</pre><p>Under the hood, the script does three jobs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lQc3BbUkeex5JFAcIOSTfg.png" /></figure><p>The script waits for the frontend to roll out, then shows the deployed resources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*prNqlpXqKYPHk-koZk95sg.png" /></figure><h4>The following resources were deployed</h4><ol><li><strong>Namespace </strong><strong>boutique</strong></li><li><strong>Online Boutique</strong> (frontend + 11 microservices + redis-cart)</li><li><strong>HTTPRoute </strong><strong>boutique-route</strong> — the app&#39;s contract with the edge.</li></ol><h4>Step 3 — Updating Cloudflare A record.</h4><p>In Step 1, our script already gives us the Load Balancer IP to point the domain to, so that boutique.invincibledevops.tech resolves to the Load Balancer (propagation usually takes 1–2 minutes).</p><p>In your Cloudflare account, go to <strong>Records</strong> → <strong>Add record</strong>, then make sure your details align with this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lv-d1ire2EtwFNYdxXLmrw.png" /></figure><h3>Step 4 — Verify</h3><p>Visit: <a href="https://boutique.invincibledevops.tech/">https://boutique.invincibledevops.tech</a>, then voila!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C437QSr8vRqMs-vs0IN09w.png" /></figure><h4>Step 5 — Proving Web Application Firewall</h4><pre>./scripts/waf-test.sh https://boutique.invincibledevops.tech</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dpr3nh1gLRyMgWSRN6Dg0Q.png" /></figure><p>SQL Injection, Cross-site scripting (XSS), Path traversal, Command injection, UNION-based SQL injection — all blocked! Our Web Application Fiewwall is proven to be functioning.</p><h3>What’s next?</h3><p>The storefront is live, and protected by a Web Application Firewall. But the storefront has no database yet, and every cart, every order, every product page depends on one. In the next part, we will deploy an HA PostgreSQL on Kubernetes with CloudNativePG, carry out load testing, and prove failover.</p><blockquote><em>Finally, do me a favour. </em><strong><em>Follow</em></strong><em> me so you’d be the first to be notified when I make a Medium post, and give this post as many </em><strong><em>claps</em></strong><em> as you like so this project can reach a wide audience. Thank you!</em></blockquote><p>Let me know if you have any questions in the comment section.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5797dd62b092" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[Part 4] Deploying Cilium: eBPF Networking, WireGuard Encryption, and Hubble Observability.]]></title>
            <link>https://medium.com/@theinvincibledev/part-4-deploying-cilium-ebpf-networking-wireguard-encryption-and-hubble-observability-360a9bec2dab?source=rss-ddb6d9aa3746------2</link>
            <guid isPermaLink="false">https://medium.com/p/360a9bec2dab</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[encryption]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[TheInvincible]]></dc:creator>
            <pubDate>Wed, 12 Aug 2026 11:01:01 GMT</pubDate>
            <atom:updated>2026-08-12T11:01:01.668Z</atom:updated>
            <content:encoded><![CDATA[<h4>In Part 3, we deployed the bare infrastructure on Google Cloud. In this part, we will install <strong>Cilium</strong> to replace the current Kubernetes-native datapath with eBPF, encrypt every pod-to-pod packet with WireGuard, and use Hubble to see into the cluster.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UDMcczlk-mONP0iYTISPxQ.png" /><figcaption>Cilium architecture</figcaption></figure><h3>Brief Overview.</h3><p>Cilium is a massive leap forward in the world of networking. To appreciate what Cilium does, we have to understand the nightmare it was built to solve. Imagine you are running a massive, highly dynamic Kubernetes cluster. You have thousands of application containers spinning up to handle traffic spikes, doing their job, and then terminating minutes later. The environment is shifting constantly.</p><p>But the underlying plumbing connecting all these workloads relies on architectural concepts built for the 1990s internet, when container technologies did not exist: legacy Linux components like <strong>kube-proxy</strong> and <strong>iptables</strong>.</p><p>In Linux, when a packet of data hits a network interface, the kernel looks at this predefined list of rules to figure out the packet’s fate, like <em>“should it be allowed through?” “Should it be dropped because it’s malicious?”</em> <em>“Does its destination IP address need to be translated before it moves on?”</em> All of these are done using <strong>iptables</strong>. But the bottleneck is how those rules are evaluated.</p><p>iptables processes packets sequentially; that does work for a static website with 10 rules but becomes an Achilles heel in a highly dynamic Kubernetes environment where every new service, every pod, every network policy translates into a barrage of new rules; sometimes hundreds of thousands of them.</p><p>And that right there is where <strong>Cilium </strong>comes in.</p><p>Cilium’s architecture is a departure from traditional Linux networking, replacing legacy tools like iptables and kube-proxy with a high-performance datapath powered by <strong>eBPF</strong> (Extended Berkeley Packet Filter).</p><p>While traditional iptables evaluates rules sequentially (O(N) complexity), which causes performance to degrade as clusters scale; Cilium uses eBPF maps, a highly optimized hash tables in the kernel, to perform lookups for routing and policy permissions in constant time (O(1)), maintaining high throughput regardless of the number of services — like it takes the same time to look up 100,000 rules as it does 10 rules. Massive, innit?</p><h3>Core Architectural Components</h3><p>Cilium is comprised of several core components that work together to manage the cluster’s network and security state:</p><ol><li><strong>Cilium Agent:</strong> A per-node daemon (typically a DaemonSet) that acts as the cluster’s “brain” on each node. It watches the Kubernetes API for events (Pods, Services, Endpoints) and translates that state into eBPF datapath configurations, compiling and loading the necessary programs into the kernel.</li><li><strong>eBPF Datapath:</strong> This is the in-kernel execution layer where packet processing actually occurs. It handles L3/L4 routing, efficient load balancing for Services, and low-overhead policy enforcement.</li><li><strong>Cilium Operator:</strong> A cluster-wide controller responsible for duties that require global knowledge, such as synchronizing node IPs, managing IP address allocation (IPAM), and allocating security identities.</li><li><strong>Hubble:</strong> The observability layer built on top of Cilium. Since Cilium is bypassing every known Linux native tool; if you bypass netfilter, standard monitoring tools go blind, so you can’t just run tcpdump and expect to see anything. Hubble exists to bridge that observability gap. It is Cilium’s own telescope peering into the deep dark space of the Linux kernel — like NASA’s Hubble telescope, but for pods and network packets.</li></ol><h4>Let’s talk a bit about Encryption before getting our hands dirty.</h4><p>Kube-proxy, which is the legacy proxy, does not provide native encryption. To secure internal traffic, you must layer a heavy Service Mesh (like Istio using mTLS sidecars) on top, which injects proxy containers into every pod, really increasing memory consumption and latency. But Cilium achieves encryption cleanly inside the Linux kernel. It intercepts packets before they leave the pod network interface and wraps them using either <strong>WireGuard</strong> (highly optimized, low configuration) or <strong>IPsec</strong> (hardware-accelerated, enterprise standard). This happens transparently, meaning application code remains completely untouched and no sidecar proxies are needed. We will be using WireGuard in this project.</p><h3>Deployment</h3><h4>Step 1 — Grab the script and install Cilium</h4><p><a href="https://github.com/TheInvincibleRalph/Enterprise-grade-GKE-Project/blob/master/scripts/install-cilium.sh">Enterprise-grade-GKE-Project/scripts/install-cilium.sh at master · TheInvincibleRalph/Enterprise-grade-GKE-Project</a></p><pre>chmod +x ./scripts/install-cilium.sh<br>./scripts/install-cilium.sh</pre><p>Under the hood, the script does three things:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2j1B0yxjtGmNsCsNp2x2oQ.png" /></figure><p>The result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Qq8IovdpGaQxv2_kI5WJw.png" /></figure><h4>The following resources were deployed</h4><ol><li><strong>Cilium agent</strong> (DaemonSet, kube-system) + <strong>cilium-operator</strong></li><li><strong>eBPF datapath: </strong>kube-proxy replacement in the kernel</li><li><strong>WireGuard encryption: </strong>transparent pod-to-pod</li><li><strong>Hubble Relay + Hubble UI: </strong>flow aggregation and the telescope</li></ol><p>The following are the important configurations within the installation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rXZbXcK-wJcxjtfVEMcXsQ.png" /></figure><h4>Step 2 — Verify the datapath</h4><pre>cilium status                 <br>kubectl get pods -n kube-system | grep cilium    # one agent per node + operator<br>kubectl get ds -n kube-system cilium</pre><h4>Step 3 — Prove WireGuard encryption</h4><p>Here’s what we want to prove.</p><blockquote>A request leaves pod-A <em>eth0</em> as plaintext, enters <em>cilium_wg0</em> where it’s encrypted and wrapped into a UDP packet, exits the node’s physical <em>eth0</em> as unreadable UDP, then crosses the GCP VPC network between the two nodes’ physical NICs, and enters the other node’s <em>cilium_wg0</em> where it’s decrypted, and arrives through pod-B <em>eth0</em> as plaintext again.</blockquote><p>Now let’s prove we have our node-to-node encryption intact by applying the encryption test suite.</p><p>The test suites apply two test namespaces with <strong>netshoot</strong> pods.</p><blockquote>Netshoot is a Docker and Kubernetes network troubleshooting container image. It is loaded with common networking utilities like dig, tcpdump, curl, ping etc to help dianose cluster or container network.</blockquote><pre>kubectl apply -f kubernetes/cilium/encryption-test.yaml<br><br># Make sure both nodes differ.<br>kubectl get pods -n test-a -o wide      <br>kubectl get pods -n test-b -o wide      </pre><p><strong>One precondition:</strong> WireGuard encrypts traffic that crosses between nodes, so make sure the pods are on different nodes.</p><p>If the node columns match, reschedule one:</p><pre>kubectl delete pod -n test-b netshoot-b<br>kubectl apply -f kubernetes/cilium/encryption-test.yaml    # reapply this<br>kubectl get pods -n test-b -o wide      # re-check</pre><p>To prove the encryption, we watch one HTTP request flow at two layers: Inside pod-A and on the node’s physical NIC (the machine’s real boundary to the outside; every packet that leaves the machine must pass through it).</p><p>First, we need a listener and the traffic flow:</p><p><strong>1. A listener on B.</strong></p><pre>kubectl exec -n test-b netshoot-b -- python3 -m http.server 80</pre><p><strong>2. A sustained flow from A.</strong> From inside netshoot-a, we send HTTP to netshoot-b’s pod IP in a loop.</p><pre>B_IP=$(kubectl get pod -n test-b netshoot-b -o jsonpath=&#39;{.status.podIP}&#39;)<br>kubectl exec -n test-a netshoot-a -- sh -c \<br>  &quot;while true; do curl -s -o /dev/null http://${B_IP}:80; sleep 0.5; done&quot;</pre><p><strong>3. Then we capture the output</strong></p><pre># Point 1 — at the pod&#39;s own interface, the request in plaintext:<br>kubectl exec -n test-a netshoot-a -- tcpdump -i eth0 -nn -c 20 -A port 80</pre><p>You should see a plaintext response:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iti3kxfiiH3iBr-nIHkOlQ.png" /></figure><p>Now let’s check the same flow on the node’s physical NIC, after the packet has crossed the WireGuard device:</p><pre># The node&#39;s physical interface:<br>NODE_A=$(kubectl get pod -n test-a netshoot-a -o jsonpath=&#39;{.spec.nodeName}&#39;)<br>kubectl debug node/${NODE_A} -it --image=nicolaka/netshoot -- tcpdump -i eth0 -nn -c 20 &#39;udp port 51871&#39;</pre><p>Now we see ciphertext.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*81Xp_ns2_Sb67ytk_mZaAQ.png" /></figure><p>What actually happened? The plaintext packet from the pod entered cilium_wg0,the device encrypted it and sealed it in a node-to-node UDP datagram on port 51871, which is exactly what we captured on the node&#39;s NIC. Same packet, before and after the encryption.</p><h4>Step 4 — Our Hubble telescope</h4><p>Remember why Hubble exists? Cilium bypasses Netfilter, so the standard Linux monitoring tools go blind. Hubble is already installed with Cilium, so we are good on the observability side.</p><pre>kubectl port-forward -n kube-system svc/hubble-ui --address 127.0.0.1 12000:80<br># then open http://127.0.0.1:12000</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MeHydxxwc2deP-5HadTxcA.png" /></figure><p>The view above is the inside of the kube-system namespace. We can see the map of pods talking to each other, and not only that, we get the source IP/identity, destination, port, etc. That is the beauty of Hubble; it gives you great insight into the cluster at a single glance.</p><p>While the visual map gives you an excellent bird’s-eye view, the table gives real-time flow of every single network request within the namespace, which is handy when troubleshooting. Currently, the <strong>Verdict</strong> column looks green, meaning traffic is flowing healthily. But say there is a network policy accidentally blocking traffic, or a pod is rejecting connections; you would immediately see red dropped Verdicts there, which quickly gives you an idea of what went wrong and how to go about debugging.</p><h3>What’s next?</h3><p>Pod-to-pod traffic is now encrypted with WireGuard, visible through Hubble, and handled by the eBPF datapath instead of kube-proxy. What the cluster still needs is the workload itself: in the next part, we deploy the Online Boutique storefront behind the Envoy Gateway edge with the WAF.</p><blockquote>Finally, do me a favour. <strong>Follow</strong> me so you’d be the first to be notified when I make a Medium post, and give this post as many <strong>claps</strong> as you like so this project can reach a wide audience. Thank you!</blockquote><p>Let me know if you have any questions in the comment section.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=360a9bec2dab" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[Part 3] Provisioning a Private, Multi-Zone GKE Cluster with Terraform]]></title>
            <link>https://medium.com/@theinvincibledev/part-3-provisioning-a-private-multi-zone-gke-cluster-with-terraform-d1499ece7177?source=rss-ddb6d9aa3746------2</link>
            <guid isPermaLink="false">https://medium.com/p/d1499ece7177</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[google-kubernetes-engine]]></category>
            <dc:creator><![CDATA[TheInvincible]]></dc:creator>
            <pubDate>Tue, 11 Aug 2026 11:01:01 GMT</pubDate>
            <atom:updated>2026-08-13T00:52:58.808Z</atom:updated>
            <content:encoded><![CDATA[<h4>In Part 2, we set up a folder and project on GCP, enabled APIs, and configured a service account identity for Terraform; we did some work on billing and alerts, and finally installed CLI tools we will need for the rest of the series. Now we will go on to deploying the bare infrastructure.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vEN_1KvQA9B5gBzT" /><figcaption>Photo by <a href="https://unsplash.com/@growtika?utm_source=medium&amp;utm_medium=referral">Growtika</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>To understand what our Terraform config achieves, I will highlight important decisions or ‘settings’ that shaped our infrastructure.</p><ol><li><strong>Regional routing mode.</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m7vwZLisaqUN-RusZPcKDg.png" /></figure><p>By default, GCP networks use GLOBAL routing. But you can explicitly configure routing_mode = “REGIONAL” inside your google_compute_network Terraform resource block for several reasons. In our case, that is what best demonstrates HA and also because GLOBAL routing can be expensive. private_ip_google_access = true allows resources within the private VPC subnet to connect outwards to Google APIs over the internal network without needing a public IP address or Cloud NAT, which makes it cheaper.</p><p><strong>2. Cloud NAT.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hCjT_sgxTLpbNQWf9gzEzw.png" /></figure><p>Because our nodes will be private, we need this for egress internet access for our nodes. resource &quot;google_compute_router_nat&quot; &quot;nat&quot; creates the Cloud NAT gateway on the router (defined in the block before). This is what performs <strong>SNAT</strong>: it rewrites the source IP of outbound packets from private node/pod IPs to a public IP, sends them out, and maps the replies back.</p><p><strong>3.</strong> <strong>Remove default node pools.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ocYFy9ec7CMGx6-XuF2wLQ.png" /></figure><p>When you spin up a managed Kubernetes cluster like GKE, a default node pool is automatically created to bootstrap the system. The default pool boots with a small HDD disk (GKE defaults to 100GB pd-balanced, which blows the 250GB SSD_TOTAL_GB quota in us-central1 with 3 zones), hence, to assign our pools with a custom disk size, we need to remove the default node pools.</p><p><strong>4. Disabling GKE Dataplane V2 to install full Cilium with Wireguard.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cmXEGdj-SNEr4BySW4gjDg.png" /></figure><p>Even though GKE Dataplane V2 is built on Cilium, it is fully abstracted. This means you do not get access to advanced self-managed features like the Hubble UI/CLI framework for deep observability. Because we need Hubble for the observability part of this project and WireGuard for encryption, we would install a full Cilium instead. GKE’s own datapath isADVANCED_DATAPATH, so plugging in LEGACY_DATAPATH bypasses it and frees the cluster so we can install Cilium.</p><p><strong>5. Private Nodes.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ne13rKC60Z1W_wsFOcILaw.png" /></figure><p>I had to set enable_private_nodes = true for each node pool because without it the provider sends enable_private_nodes=false and the pool gets public IPs even in a private cluster, which would blow the tiny IN_USE_ADDRESSES quota for the project.</p><p><strong>6. System pool right-sizing and taint/toleration.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5UHPpoIVhuJMhG2PVnWd1g.png" /></figure><p>I used e2-custom-1-4096 (1 vCPU / 4 GB) because 3 zones × 1 vCPU = 3 of the 12 vCPUs, and 4 GB is the floor for GKE&#39;s system pods plus Cilium. The CriticalAddonsOnly taint keeps application workloads off these nodes; only Cilium, CoreDNS, and the operators get the machine to themselves; others go to the application nodes.</p><p><strong>7. Application pool right sizing and taint/toleration.</strong></p><p>I used e2-custom-2-8192 (2 vCPU / 8 GB), autoscaling min 1/max 3, because it fits within the quota — 3 app × 2 vCPU (6) + 3 system × 1 vCPU (3) = 9 vCPU ≤ 12, and 6 instances ≤ 8.</p><p><strong>8. Spot Nodes.</strong></p><p>Spot VMs are on by default (use_spot_vms = true), that&#39;s the 60–91% cost cut that makes this whole project affordable.</p><h3>Now to deployment.</h3><p>Grab the code below and follow the commands to deploy.</p><p><a href="https://github.com/TheInvincibleRalph/Enterprise-grade-GKE-Project">GitHub - TheInvincibleRalph/Enterprise-grade-GKE-Project: A production-grade cloud platform built on Google Kubernetes Engine on a private multi-zone cluster with eBPF networking and encrypted pod traffic, a WAF-protected edge, highly available PostgreSQL, enterprise observability, chaos engineering, and an inference server that scales to zero when idle.</a></p><pre>cd infra/terraform<br>terraform init -backend-config=backend.hcl<br>terraform plan<br>terraform apply -auto-approve</pre><p>A regional cluster takes 10–15 minutes to come up. When it’s done, the outputs hand you a gcloud container clusters get-credentials command which fetches the GKE cluster’s credentials from Google and writes them into your local <strong>kubeconfig</strong>; that way you can run kubectl commands directly from your terminal, so do well to run the command.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k76C_6_oe4Wxw3Z9e8hTLw.png" /></figure><p>Don’t forget to re-authenticate every time you’ve left the terminal for a long time: gcloud auth login .</p><p>Finally, you can confirm from the console that your cluster has been provisioned.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2UG9RofbLj14k9m1GRFr_w.png" /></figure><h4>The following resources were provisioned</h4><ol><li>VPC with custom subnet and secondary ranges for pods/services</li><li>Cloud NAT for private node egress</li><li>Regional GKE cluster with Workload Identity</li><li>Multiple node pools: system, application.</li></ol><h3>What’s next?</h3><p>The cluster is up, but right now its networking is handled by kube-proxy. In the next part, we replace that with Cilium, eBPF-based networking, configure transparent WireGuard encryption between every pod, and deploy a microservice application.</p><p>Part 4: <a href="https://medium.com/@theinvincibledev/360a9bec2dab"><strong>Deploying Cilium: eBPF Networking, WireGuard Encryption, and Hubble Observability</strong>.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d1499ece7177" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an Enterprise-Grade Kubernetes Ecosystem on GKE]]></title>
            <link>https://medium.com/@theinvincibledev/building-an-enterprise-grade-kubernetes-ecosystem-on-gke-d52102d912a7?source=rss-ddb6d9aa3746------2</link>
            <guid isPermaLink="false">https://medium.com/p/d52102d912a7</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[TheInvincible]]></dc:creator>
            <pubDate>Sat, 08 Aug 2026 16:18:22 GMT</pubDate>
            <atom:updated>2026-08-24T11:36:42.107Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>A hands-on engineering series featuring Cilium, Envoy Gateway, and Chaos Mesh.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H5YQGdb3Ed7CDJ1YA5snOQ.png" /><figcaption>Architectural overview of the entire project.</figcaption></figure><h3><strong>First things first: Why this project?</strong></h3><p>Curious how different tools come together into one resilient system, beyond deploying an isolated app to the cloud, beyond spinning up a few pods? Then be my guest.</p><p>This is a hands-on series where I build a resilient, highly available, and secure Kubernetes environment on GKE, from a blank GCP account to a live, WAF-protected website on a 3-AZ cluster.</p><p>This post is the first of an <strong>11-part series</strong>, where I walk you through the whole build, step by step, every command, every decision. Each phase builds on the previous one, and together they culminate in a single, cohesive architecture that demonstrates <strong>security, reliability, observability, and cost-awareness</strong> — the four pillars of production infrastructure.</p><p>I had a lot of fun building this, and I learned even more. I can’t wait to pour it all out here and bring you into the fun.</p><h3><strong>What’s in it for you (and me)?</strong></h3><p>Finish this project end-to-end, and you&#39;ll walk away with senior-level insight into what an enterprise-grade ecosystem actually looks like, and be in the know about some amazing tools and technologies in the cloud ecosystem. The complete project is gold for your DevOps portfolio and gives you a strong talking point in interviews.</p><p>You will also learn the following:</p><p><strong>Infrastructure as Code &amp; GitOps</strong></p><ul><li>Terraform for provisioning GKE clusters, node pools, and networking</li><li>Helm for packaging and deploying complex workloads</li></ul><p><strong>Kubernetes Deep Knowledge</strong></p><ul><li>The operator pattern (CloudNativePG, KEDA)</li><li>Custom Resource Definitions (CRDs) and controllers</li><li>Pod Disruption Budgets, affinity/anti-affinity, topology spread constraints</li><li>Horizontal Pod Autoscaling (HPA) vs. event-driven autoscaling (KEDA)</li><li>Spot VM node pools and graceful termination handling</li></ul><p><strong>Networking &amp; Security</strong></p><ul><li><strong>Cilium</strong> CNI with <strong>eBPF </strong>— replacing kube-proxy entirely</li><li>Transparent <strong>WireGuard </strong>encryption between pods</li><li>Gateway API (GatewayClass/Gateway/HTTPRoute) on Envoy Gateway</li><li>A <strong>Web Application Firewall</strong> (Coraza) with the OWASP Core Rule Set, running in-process as a native Envoy dynamic module</li><li>NetworkPolicy and CiliumNetworkPolicy for microsegmentation</li></ul><p><strong>Observability &amp; Chaos Engineering</strong></p><ul><li><strong>Prometheus</strong> + <strong>Grafana</strong> for metrics collection and visualization</li><li><strong>Datadog</strong> integration for enterprise-grade APM and infrastructure monitoring</li><li><strong>Chaos Mesh</strong> for injecting faults: pod kills, network latency, CPU stress, node failures</li><li>Defining and validating steady-state hypotheses</li></ul><p><strong>AI Infrastructure</strong></p><ul><li>Deploying LLM inference on CPU (Ollama)</li><li><strong>KEDA</strong> event-driven autoscaling — a Redis queue-length scaler driving scale-to-zero</li><li>Model preloading and cold-start optimization on spot instances</li></ul><p><strong>Real-World Enterprise Practices</strong></p><ul><li><strong>Google Cloud Organization</strong> hierarchy, IAM, and service accounts</li><li><strong>Cloudflare</strong> DNS management with free HTTPS proxying</li><li>Cost optimization with spot VMs and scale-to-zero patterns</li><li>Clean project isolation for budget control</li></ul><h3><strong>The constraints around this project</strong></h3><p>This project runs on a <strong>GCP free trial</strong>: 90 days, $300 credit, no quota increases, no GPUs.</p><p>These are the constraints Google applies to free-trial accounts:</p><ul><li><strong>INSTANCES</strong> (us-central1): 8</li><li><strong>CPUS_ALL_REGIONS</strong> (global): 12</li><li><strong>IN_USE_ADDRESSES</strong> (us-central1): 4</li><li><strong>SSD_TOTAL_GB</strong> (us-central1): 250</li><li><strong>Quota increases</strong>: not available on trial</li><li><strong>GPUs</strong>: not attachable on trial</li></ul><p>To stay within these constraints, the final sizing of our resources will be:</p><ul><li><strong>System pool:</strong> 3 × e2-custom-1–4096 (1 vCPU / 4 GB), one per zone → 3 vCPU</li><li><strong>Application pool:</strong> e2-custom-2–8192 (2 vCPU / 8 GB), autoscaling min 1 / max 3 → up to 6 vCPU</li><li><strong>All nodes:</strong> spot, pd-standard 30 GB disks, private (no external IPs)</li><li><strong>Totals:</strong> 9 vCPU of 12 used at max scale, 6 of 8 instances, 2 of 4 addresses (NAT + LoadBalancer)</li></ul><h3><strong>The caveat (read this before you start)</strong></h3><p>Building this end-to-end takes time. If you’re the busy type, budget weeks; if you’re a student with room to stretch, days. Either way, here’s the honest warning: <strong>don’t expect to tear your infra down once you begin. </strong>It’s built to stay up — and that sounds expensive, right?</p><p>You don’t have to worry; we are using Spot instances and running on a generous $300 GCP credit!</p><p>Can I shock you? My full environment — 6 nodes, 90+ pods, 200+ containers, and live AI inference — stayed up for <strong>two weeks on less than $50</strong>. So?</p><p><strong>Let’s get to it.</strong></p><p>These are the different parts of the series.</p><ol><li><a href="https://medium.com/@theinvincibledev/part-2-setting-up-your-environment-prerequisites-and-cost-optimization-64d3c4b9bbf7?sharedUserId=theinvincibledev">[Part 2] Setting Up Your Environment, Prerequisites, and Cost Optimization</a></li><li><a href="https://medium.com/@theinvincibledev/part-3-provisioning-a-private-multi-zone-gke-cluster-with-terraform-d1499ece7177?sharedUserId=theinvincibledev">[Part 3] Provisioning a Private, Multi-Zone GKE Cluster with Terraform</a></li><li><a href="https://medium.com/@theinvincibledev/part-4-deploying-cilium-ebpf-networking-wireguard-encryption-and-hubble-observability-360a9bec2dab?sharedUserId=theinvincibledev">[Part 4] Deploying Cilium: eBPF Networking, WireGuard Encryption, and Hubble Observability.</a></li><li><a href="https://medium.com/@theinvincibledev/part-5-configuring-the-clusters-edge-envoy-gateway-with-an-in-process-coraza-waf-deploying-a-5797dd62b092?sharedUserId=theinvincibledev">[Part 5] Configuring the Cluster’s Edge: Envoy Gateway with an In-Process Coraza WAF &amp; Deploying a Microservices Application Behind the WAF.</a></li><li><a href="https://medium.com/@theinvincibledev/part-6-deploying-a-highly-available-postgresql-on-kubernetes-with-cloudnativepg-load-testing-it-92b64a304393?sharedUserId=theinvincibledev">[Part 6] Deploying a Highly Available PostgreSQL on Kubernetes with CloudNativePG, Load Testing It and Proving Failover</a></li><li><a href="https://medium.com/@theinvincibledev/part-7-observability-with-prometheus-grafana-b5159196d598?sharedUserId=theinvincibledev">[Part 7] Observability with Prometheus &amp; Grafana</a></li><li><a href="https://medium.com/@theinvincibledev/part-8-optional-observability-with-datadog-e8562102988e?sharedUserId=theinvincibledev">[Part 8][Optional] Observability with Datadog</a></li><li><a href="https://medium.com/@theinvincibledev/part-9-chaos-engineering-with-chaos-mesh-80956e7f4236?sharedUserId=theinvincibledev">[Part 9] Chaos Engineering with Chaos Mesh</a></li><li><a href="https://medium.com/@theinvincibledev/part-10-deploying-an-ollama-inference-server-on-gke-with-keda-7c89a4081461?sharedUserId=theinvincibledev">[Part 10] Deploying an Ollama Inference Server on GKE with KEDA</a></li><li><a href="https://medium.com/@theinvincibledev/part-11-wrapping-up-what-we-have-built-the-full-teardown-and-portfolio-value-fd8a69442681?sharedUserId=theinvincibledev">[Part 11] Wrapping Up: What We Have Built, the Full Teardown, and Portfolio Value</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d52102d912a7" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>