<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pascal Töpke on Medium]]></title>
        <description><![CDATA[Stories by Pascal Töpke on Medium]]></description>
        <link>https://medium.com/@pascal.toepke?source=rss-ede1fb33c222------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*6V815BUdIVVfp3cVN-75Bw.jpeg</url>
            <title>Stories by Pascal Töpke on Medium</title>
            <link>https://medium.com/@pascal.toepke?source=rss-ede1fb33c222------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 06:31:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pascal.toepke/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[MariaDB Operator: HA MySQL the Modern Way]]></title>
            <link>https://medium.com/@pascal.toepke/mariadb-operator-ha-mysql-the-modern-way-70ccf85c2e46?source=rss-ede1fb33c222------2</link>
            <guid isPermaLink="false">https://medium.com/p/70ccf85c2e46</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[kubernetes-operator]]></category>
            <category><![CDATA[mysql]]></category>
            <category><![CDATA[mariadb]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Pascal Töpke]]></dc:creator>
            <pubDate>Sun, 26 Jul 2026 12:51:05 GMT</pubDate>
            <atom:updated>2026-07-26T12:51:05.850Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6Ny5i3occoS2tzbjPwEJtQ.png" /><figcaption>The same story as CloudNativePG, applied to the MySQL half of my homelab.</figcaption></figure><p>A while back I wrote about <a href="https://medium.com/@pascal.toepke/cloudnativepg-postgres-database-the-modern-way-6a3bc9cf5eba">CloudNativePG: Postgres database the modern way</a>, and how moving Postgres onto a Kubernetes operator let me delete a pile of Patroni, etcd, HAProxy, and Keepalived glue I had babysat for years.</p><p>That post left one thing unfinished. My homelab runs MySQL too. BookStack, a few legacy apps, the usual suspects; all of them still talk to a set of MariaDB VMs held together with HAProxy and keepalived. I solved Postgres and left its twin running right next to it.</p><p>This is the sequel. Same approach, pointed at MySQL. Time to move the other half onto Kubernetes.</p><h3>The Old Way: A House of Cards (Again)</h3><p>If you read the CloudNativePG post, this will feel familiar. The traditional recipe for highly available MySQL/MariaDB looks a lot like the Postgres one:</p><ul><li><strong>MariaDB</strong> on a couple of dedicated VMs.</li><li><strong>Galera</strong> (or classic primary/replica replication) keeping those nodes in sync.</li><li><strong>HAProxy</strong> in front, routing clients to a healthy node.</li><li><strong>keepalived</strong> holding a floating virtual IP so the HAProxy layer stays redundant.</li></ul><p>It runs, and it has run for a long time. Every one of those boxes is something I own, patch, monitor, and get paged about at the worst possible time.</p><h3>The Pain Points</h3><p>The complaints match the ones I had about the old Postgres stack:</p><ul><li><strong>Too many moving parts.</strong> A database cluster here means four separate pieces of software, each with its own config file, its own failure modes, and its own definition of healthy.</li><li><strong>Configuration hell.</strong> Getting HAProxy’s health checks to agree with Galera’s actual state, and getting keepalived to fail the VIP over cleanly, stays fiddly. That kind of fiddly shows up during an incident.</li><li><strong>Maintenance burden.</strong> OS patching, MariaDB upgrades, and certificate rotation all get choreographed by hand across VMs that live outside my Kubernetes world.</li><li><strong>It’s a snowflake.</strong> None of it looks like the rest of my infrastructure. Everything else is declarative, GitOps-managed, and reconciled by a controller. The database VMs were the one place I still SSH’d in and ran commands like it was 2015.</li></ul><p>The old way runs fine. It costs too much: in complexity, in attention, and in the number of 2am decisions it asks for.</p><h3>The Modern Way: mariadb-operator</h3><p>The operator pattern that made CloudNativePG pleasant has a direct equivalent for MySQL: <a href="https://github.com/mariadb-operator/mariadb-operator"><strong>mariadb-operator</strong></a>. If you have used CNPG, it will feel familiar.</p><p>You install a controller into the cluster. From then on your database becomes an object you declare. You write a MariaDB resource, apply it, and the operator provisions the pods, wires up storage, bootstraps the cluster, exposes stable Services, and reconciles the whole thing continuously.</p><p>The part I like most matches CNPG: the list of things you delete.</p><ul><li>No HAProxy. The operator gives you stable Kubernetes Services (a primary/read-write endpoint and a read endpoint) out of the box.</li><li>No keepalived and no floating VIP. Kubernetes Service networking already does that job.</li><li>No VMs to patch. You get pods, storage, and CRDs, the same primitives as everything else in the cluster.</li></ul><p>Fewer things to own.</p><h3>A Quick Word on Galera</h3><p>The operator leans on <strong>Galera</strong> for high availability, so a short detour helps. Galera works differently from the Postgres model.</p><p>CloudNativePG runs a primary plus replicas topology: one node accepts writes, the others stream and replay the WAL. Galera runs synchronous, virtually-synchronous, multi-master replication. In practice:</p><ul><li><strong>Every node accepts writes.</strong> All members hold a full, consistent copy of the data, so there is no single primary to fail over.</li><li><strong>Writes get certified across the cluster before they commit.</strong> A transaction that commits on one node applies on the others or gets rejected everywhere, which keeps replica-lag surprises out of the picture.</li><li><strong>It runs on quorum.</strong> A typical deployment uses three nodes, so the cluster tolerates losing one member. Lose quorum and it stops accepting writes to avoid a split brain.</li></ul><p>You get HA baked into the replication protocol. The operator manages the Galera lifecycle: bootstrapping the cluster, adding and recovering members, and running the health checks, so you never touch wsrep config by hand.</p><h3>Key Benefits of mariadb-operator</h3><h4>Declarative databases, users, and grants</h4><p>My favorite part of CNPG works the same way here. You declare Database, User, and Grant resources, and the operator reconciles the server to match. No exec into a pod to run CREATE DATABASE, CREATE USER, or GRANT. Your schema of access lives in Git, reviewable and versioned.</p><h4>Backups as a first-class object</h4><p>The operator ships a native <strong>Backup</strong> custom resource. You point it at an S3 bucket, give it a schedule and a retention window, and it runs logical dumps on a timer and pushes them to object storage. It maps directly onto Barman Cloud in the CNPG world.</p><h4>It looks like the rest of the cluster</h4><p>Same PodSpecs, same storage classes, same Prometheus metrics, same NetworkPolicies. The database behaves like any other workload.</p><h4>One pattern, two engines</h4><p>I already run CloudNativePG, so adopting mariadb-operator gave my Postgres and MySQL stacks the same shape: an operator, a cluster CR, declarative access, and native backups to object storage. That consistency pays off on its own.</p><h3>Comparison at a Glance</h3><ul><li><strong>HA and failover.</strong> Old: Galera, HAProxy, and a keepalived VIP. Operator: Galera, managed directly.</li><li><strong>Load balancing.</strong> Old: HAProxy. Operator: native Kubernetes Services.</li><li><strong>Where it runs.</strong> Old: dedicated VMs. Operator: pods and PVCs in-cluster.</li><li><strong>Provisioning databases and users.</strong> Old: manual SQL. Operator: Database, User, and Grant CRs.</li><li><strong>Backups.</strong> Old: cron plus a mariadb-dump script. Operator: native Backup CR to S3.</li><li><strong>Config management.</strong> Old: hand-edited files across VMs. Operator: declarative CRDs in Git.</li><li><strong>Monitoring.</strong> Old: a bolt-on exporter and a hand-rolled DB user. Operator: managed exporter, metrics user, and ServiceMonitor.</li></ul><h3>Let’s Get Our Hands Dirty</h3><p>Everything below runs through Terraform: the operator via Helm, the cluster and its databases via manifests. Names and endpoints are genericized; adapt them to your setup.</p><h4>Step 1: Install the operator</h4><p>The operator ships as an OCI Helm chart. Like several modern operators, it splits the CRDs into a separate release so you can manage their lifecycle independently. Install those first, then the operator itself.</p><pre>resource &quot;helm_release&quot; &quot;mariadb_operator_crds&quot; {<br>  name       = &quot;mariadb-operator-crds&quot;<br>  repository = &quot;oci://ghcr.io/mariadb-operator/charts&quot;<br>  chart      = &quot;mariadb-operator-crds&quot;<br>  version    = &quot;26.6.0&quot;<br>  namespace  = &quot;mariadb-system&quot;<br>}<br><br>resource &quot;helm_release&quot; &quot;mariadb_operator&quot; {<br>  name       = &quot;mariadb-operator&quot;<br>  repository = &quot;oci://ghcr.io/mariadb-operator/charts&quot;<br>  chart      = &quot;mariadb-operator&quot;<br>  version    = &quot;26.6.0&quot;<br>  namespace  = &quot;mariadb-system&quot;<br><br>  set {<br>    name  = &quot;metrics.enabled&quot;<br>    value = &quot;true&quot;<br>  }<br><br>  depends_on = [helm_release.mariadb_operator_crds]<br>}</pre><p>The operator has a <strong>built-in cert-controller</strong> for its admission webhook, so it stands up without a cert-manager dependency. Once it settles, you have the operator, cert-controller, and webhook pods running in mariadb-system.</p><h4>Step 2: Deploy a Galera cluster</h4><p>Now the fun part: declaring an actual highly available database. This is a three-node Galera cluster with persistent storage and a root password sourced from a Secret.</p><pre>apiVersion: k8s.mariadb.com/v1alpha1<br>kind: MariaDB<br>metadata:<br>  name: mariadb-galera<br>  namespace: databases-mariadb<br>spec:<br>  rootPasswordSecretKeyRef:<br>    name: mariadb-galera-root<br>    key: password<br><br>  replicas: 3<br>  galera:<br>    enabled: true<br><br>  storage:<br>    size: 10Gi<br>    storageClassName: longhorn<br><br>  myCnf: |<br>    [mariadb]<br>    character-set-server=utf8mb4<br>    collation-server=utf8mb4_unicode_ci<br><br>  # Per-instance metrics (covered in the Monitoring section)<br>  metrics:<br>    enabled: true<br>    exporter:<br>      # Point the exporter at the cluster&#39;s primary Service<br>      args:<br>        - &quot;--mysqld.address=mariadb-galera-primary.databases-mariadb.svc.cluster.local:3306&quot;</pre><p>Apply that, and the operator bootstraps a three-member Galera cluster and hands you stable Service endpoints: a read-write (primary) Service and a read-only Service, with no HAProxy involved. That is the entire HA layer I used to run on VMs, expressed in about twenty lines of YAML.</p><h4>Step 3: Databases, users, and grants, declaratively</h4><p>Here you stop touching SQL by hand. For a database with a scoped user, declare it:</p><pre>apiVersion: k8s.mariadb.com/v1alpha1<br>kind: Database<br>metadata:<br>  name: bookstack<br>  namespace: databases-mariadb<br>spec:<br>  mariaDbRef:<br>    name: mariadb-galera<br>  characterSet: utf8mb4<br>  collate: utf8mb4_unicode_ci<br>---<br>apiVersion: k8s.mariadb.com/v1alpha1<br>kind: User<br>metadata:<br>  name: bookstack<br>  namespace: databases-mariadb<br>spec:<br>  mariaDbRef:<br>    name: mariadb-galera<br>  passwordSecretKeyRef:<br>    name: bookstack-db<br>    key: password<br>  host: &quot;%&quot;<br>---<br>apiVersion: k8s.mariadb.com/v1alpha1<br>kind: Grant<br>metadata:<br>  name: bookstack<br>  namespace: databases-mariadb<br>spec:<br>  mariaDbRef:<br>    name: mariadb-galera<br>  username: bookstack<br>  host: &quot;%&quot;<br>  database: bookstack<br>  privileges:<br>    - &quot;ALL PRIVILEGES&quot;</pre><p>Your app connects to mariadb-galera-primary.databases-mariadb.svc.cluster.local:3306 with credentials in a Secret and permissions in Git. Adding another app&#39;s database later takes three more small resources. No mysql shell, no GRANT statements typed from memory.</p><h4>Step 4: Backups to object storage</h4><p>The safety net. The native Backup resource runs scheduled logical dumps and ships them to S3-compatible storage. In my case a self-hosted object store, though any S3 endpoint works.</p><pre>apiVersion: k8s.mariadb.com/v1alpha1<br>kind: Backup<br>metadata:<br>  name: mariadb-galera-scheduled<br>  namespace: databases-mariadb<br>spec:<br>  mariaDbRef:<br>    name: mariadb-galera<br>  schedule:<br>    cron: &quot;0 2 * * *&quot;       # daily at 02:00<br>  maxRetention: 2160h        # keep 90 days<br>  compression: gzip<br>  storage:<br>    s3:<br>      bucket: mariadb-backup<br>      endpoint: s3.example.internal:9000<br>      prefix: mariadb-galera<br>      tls:<br>        enabled: true<br>      accessKeyIdSecretKeyRef:<br>        name: mariadb-backup-s3<br>        key: accessKeyId<br>      secretAccessKeySecretKeyRef:<br>        name: mariadb-backup-s3<br>        key: secretAccessKey</pre><p>One detail worth calling out from running this for real: give the backup job its <strong>own bucket-scoped credentials</strong>. If your object store enforces per-bucket policies (as it should), reusing another service’s keys returns a 403 at the worst moment. A dedicated service account and bucket per database keeps it clean.</p><p>Once applied, the operator spins up a dump Job on schedule and lands compressed .sql.gz files in the bucket. I verified mine with a manual run (kubectl create job --from=cronjob/...) and watched the dump, gzip, and upload land in object storage end to end. Trust, but verify.</p><h3>Monitoring</h3><p>The operator handles most of this for you. Two layers of metrics matter:</p><ul><li><strong>Operator metrics.</strong> You enable these on the operator’s Helm release (metrics.enabled=true). They report on the controller itself: reconcile loops, work-queue depth, and errors.</li><li><strong>Instance metrics.</strong> The ones you use day to day: connections, queries per second, InnoDB buffer pool, and the Galera wsrep state (cluster size, flow control, and whether a node sits Synced). They come from the metrics block on the MariaDB resource in Step 2.</li></ul><p>That block does a lot. When you enable instance metrics, the operator:</p><ol><li><strong>Deploys a </strong><strong>mysqld-exporter</strong> alongside the cluster on port 9104. You point it at the cluster with --mysqld.address (the primary Service, as in the Step 2 snippet) so it knows which endpoint to scrape.</li><li><strong>Auto-provisions a scoped metrics user.</strong> It creates the User, the Grant, and a password Secret, so the exporter logs in with least privilege.</li><li><strong>Creates a </strong><strong>ServiceMonitor</strong> so your Prometheus-compatible scraper finds the exporter automatically.</li></ol><p>My scraper is <a href="https://victoriametrics.com/">VictoriaMetrics</a> (the victoria-metrics-k8s-stack). The operator emits a standard ServiceMonitor, the VictoriaMetrics operator converts it into its own scrape object, and vmagent picks it up with no extra config.</p><p>One piece stays on you: <strong>NetworkPolicy.</strong> My database namespaces run default-deny, so scraping starts only after you allow ingress from the monitoring namespace to the exporter’s port:</p><pre>apiVersion: cilium.io/v2<br>kind: CiliumNetworkPolicy<br>metadata:<br>  name: mariadb-metrics-ingress<br>  namespace: databases-mariadb<br>spec:<br>  endpointSelector: {}<br>  ingress:<br>    - fromEndpoints:<br>        - matchLabels:<br>            io.kubernetes.pod.namespace: monitoring<br>      toPorts:<br>        - ports:<br>            - port: &quot;9104&quot;<br>              protocol: TCP</pre><p>If your scraper’s namespace also runs default-deny on egress, and mine does, allow the matching egress to port 9104 on that side too. Metrics need a clear path both ways.</p><h3>The payoff: a dashboard</h3><p>With the exporter feeding data in, you get the full MySQL and Galera picture. The series I watch:</p><ul><li>mysql_up: whether the node responds at all.</li><li>mysql_global_status_threads_connected against mysql_global_variables_max_connections: connection headroom.</li><li>rate(mysql_global_status_questions[5m]): queries per second.</li><li>mysql_global_status_wsrep_cluster_size: Galera members, which should equal your replica count.</li><li>mysql_global_status_wsrep_local_state: a value of 4 means the node sits Synced.</li><li>mysql_global_status_wsrep_flow_control_paused: how much replication throttles writes.</li></ul><p>You do not have to build those panels by hand. I imported the community <a href="https://grafana.com/grafana/dashboards/14057-mysql/">MySQL dashboard 14057</a> into Grafana. It targets these mysqld_exporter metric names directly, so it populated as soon as data arrived. Point it at your VictoriaMetrics datasource and the MySQL fleet appears next to the Postgres dashboards, same look and same alerting.</p><h3>Conclusion</h3><p>CloudNativePG moved Postgres off the VMs. mariadb-operator does the same for MySQL, and the two solutions share a shape: an operator, a cluster CR, declarative databases, users, and grants, native backups to object storage, and metrics that match everything else.</p><p>That symmetry is the win. My homelab used to run two separate, manual database stacks bolted onto the side of an otherwise declarative, GitOps-managed cluster. Now both run as plain Kubernetes workloads.</p><p>If you already run CloudNativePG and still have a MariaDB VM or two in the corner, you know what to do. The other half of the house is waiting.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=70ccf85c2e46" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CloudnativePG: postgres database the modern way]]></title>
            <link>https://medium.com/@pascal.toepke/cloudnativepg-postgres-database-the-modern-way-6a3bc9cf5eba?source=rss-ede1fb33c222------2</link>
            <guid isPermaLink="false">https://medium.com/p/6a3bc9cf5eba</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cloudnativepg]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[postgresql]]></category>
            <dc:creator><![CDATA[Pascal Töpke]]></dc:creator>
            <pubDate>Sun, 01 Mar 2026 19:15:25 GMT</pubDate>
            <atom:updated>2026-03-01T19:15:25.025Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k-V2DGq3G6o1B8TywBTH0Q.png" /></figure><p>For years, running a highly available (HA) PostgreSQL cluster was considered one of the “final bosses” of database administration. If you wanted a setup that could survive a node failure without manual intervention, you had to piece together a complex puzzle of tools.</p><p>I have been managing this “manual way” for a long time now, wrestling with configurations and making sure all moving parts stay in sync. But recently, I finally made the leap and migrated to <strong>CloudNativePG</strong>, and the difference is huge.</p><p>The world has changed. With the rise of Kubernetes and the advent of CloudNativePG, the “modern way” of running Postgres is finally here.</p><h3>The “Old” Way: A House of Cards?</h3><p>Before Kubernetes operators like CloudNativePG, the gold standard for Postgres HA involved a stack that looked something like this:</p><ol><li><strong>PostgreSQL</strong>: The database itself.</li><li><strong>Patroni</strong>: A Python-based template to manage the Postgres lifecycle and replication.</li><li><strong>etcd (or Consul/Zookeeper)</strong>: A distributed consensus store used by Patroni to keep track of who the leader is.</li><li><strong>HAProxy</strong>: To provide a single entry point for applications, routing traffic to the current primary.</li><li><strong>Keepalived</strong>: Often needed to provide a Virtual IP (VIP) for HAProxy itself.</li></ol><h3>The Pain Points</h3><p>While this stack is robust and proven, it comes with significant overhead:</p><ul><li><strong>Complexity</strong>: You aren’t just managing a database; you’re managing a distributed system (etcd) and a load balancer (HAProxy) on top of it.</li><li><strong>Configuration Hell</strong>: Each component has its own configuration file, its own failure modes, and its own scaling requirements.</li><li><strong>Maintenance Burden</strong>: Upgrading Postgres often meant carefully orchestrating updates across Patroni while making sure etcd stayed healthy throughout.</li><li><strong>Monitoring</strong>: You need to monitor four or five different services just to know your database is “up.”</li></ul><h3>The Modern Way: CloudNativePG</h3><p><strong>CloudNativePG (CNPG)</strong> is an open-source operator designed specifically for Kubernetes. It doesn’t just “wrap” Postgres; it treats the database as a native part of the Kubernetes ecosystem.</p><h3>Why it’s different</h3><p>The most radical thing about CNPG is what it <em>doesn’t</em> include. It completely eliminates the need for Patroni, etcd, and HAProxy.</p><p>How? By using what Kubernetes already gives you.</p><ul><li><strong>No etcd needed</strong>: CNPG uses the Kubernetes API server as its source of truth for leader election and cluster state. If you have a working K8s cluster, you already have the consensus layer you need.</li><li><strong>No Patroni needed</strong>: The operator itself manages the instance lifecycle, replication, and failover logic natively.</li><li><strong>No HAProxy needed</strong>: CNPG creates native Kubernetes Services that automatically route traffic to the primary or the replicas.</li></ul><h3>Key Benefits of CloudNativePG</h3><h3>Declarative Management</h3><p>Everything is a Custom Resource (CR). You define your cluster in a YAML file:</p><pre>apiVersion: postgresql.cnpg.io/v1<br>kind: Cluster<br>metadata:<br>  name: my-database<br>spec:<br>  instances: 3<br>  storage:<br>    size: 1Gi</pre><p>Apply that file, and you have a 3-node HA cluster. Need more replicas? Just change instances: 3 to instances: 5.</p><h3>Self-Healing</h3><p>If a node fails, the CNPG operator notices immediately. It promotes a replica, updates the Kubernetes Services, and starts a new instance to maintain your desired count. All automatic, no manual intervention needed.</p><h3>Integrated Backups</h3><p>CNPG has native support for barman-cloud-wal-archive and barman-cloud-backup, so you can stream WAL logs and backups directly to S3, GCS, or Azure Blob Storage with just a few lines of configuration.</p><h3>Zero-Downtime Upgrades</h3><p>The operator can perform rolling updates of the Postgres instances, so your application stays online even while the database engine is being patched.</p><h3>Comparison at a Glance</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uMhGx9rDUAgvvSDUrWkTIw.png" /></figure><h3>Tutorial: Deploying CloudNativePG with Terraform</h3><p>Enough theory, let’s get our hands dirty. In this section, I’ll walk you through deploying a production-ready CNPG cluster using Terraform. If you prefer plain kubectl and YAML, the same concepts apply. Terraform just gives us reproducibility and state management on top.</p><h3>Step 1: Install the CNPG Operator</h3><p>First, we need the operator itself. The Helm chart will create the namespace for us:</p><pre># Deploy CloudNative-PG Operator<br>resource &quot;helm_release&quot; &quot;cloudnative_pg&quot; {<br>  name             = &quot;cloudnative-pg&quot;<br>  repository       = &quot;https://cloudnative-pg.github.io/charts&quot;<br>  chart            = &quot;cloudnative-pg&quot;<br>  version          = &quot;0.27.0&quot;<br>  namespace        = &quot;cnpg-system&quot;<br>  cleanup_on_fail  = true<br>  create_namespace = true<br>}</pre><p>This installs the operator via its official Helm chart. The operator watches for Cluster custom resources across all namespaces and manages PostgreSQL instances accordingly.</p><h3>Step 2: Install the Barman Cloud Plugin</h3><p>Remember the integrated backup support I mentioned earlier? To use it, you need the Barman Cloud plugin:</p><pre># Deploy Barman Cloud Plugin for backup management<br>resource &quot;helm_release&quot; &quot;barman_cloud&quot; {<br>  name             = &quot;plugin-barman-cloud&quot;<br>  repository       = &quot;https://cloudnative-pg.github.io/charts&quot;<br>  chart            = &quot;plugin-barman-cloud&quot;<br>  namespace        = &quot;cnpg-system&quot;<br>  cleanup_on_fail  = true<br>  create_namespace = false<br><br>  depends_on = [<br>    helm_release.cloudnative_pg<br>  ]<br>}</pre><p>This plugin enables WAL archiving and base backups to S3-compatible storage, GCS, or Azure Blob Storage.</p><h3>Step 3: Deploy a Production Cluster</h3><p>To make deploying a full CNPG cluster easier, I wrote a <a href="https://github.com/pascalinthecloud/terraform-module-cnpg-database">Terraform module</a> that wraps the cluster configuration, database creation, backups, and secret management into a single reusable module. It’s still under active development and not perfect yet, but it already handles the heavy lifting for most use cases.</p><p>Here’s a real-world example with persistent storage and automated backups:</p><pre># Generate a secure password for the database user<br>resource &quot;random_password&quot; &quot;my_app_user&quot; {<br>  length  = 32<br>  special = false<br>}<br><br>module &quot;postgres_cluster&quot; {<br>  source = &quot;github.com/pascalinthecloud/terraform-module-cnpg-database?ref=v0.0.6&quot;<br><br>  databases = [<br>    {<br>      name                        = &quot;my-app-postgres&quot;<br>      owner                       = &quot;my_app_user&quot;<br>      password                    = random_password.my_app_user.result<br>      reclaim_policy              = &quot;Retain&quot;<br>      create_connection_secret    = true<br>      connection_secret_namespace = &quot;my-app&quot;<br>    }<br>  ]<br><br>  cluster = {<br>    name          = &quot;postgres-cluster&quot;<br>    namespace     = kubernetes_namespace.databases.metadata[0].name<br>    instances     = 3<br>    storage_class = &quot;your-storage-class&quot;   # e.g. longhorn, rook-ceph, gp3, etc.<br>    storage_size  = &quot;20Gi&quot;<br>    postgresql_max_connections = &quot;200&quot;<br>  }<br><br>  labels = {<br>    app         = &quot;postgres-cluster&quot;<br>    environment = &quot;production&quot;<br>    managed-by  = &quot;terraform&quot;<br>  }<br><br>  backup = {<br>    enabled                 = true<br>    s3_endpoint_url         = &quot;https://s3.example.com:9000&quot;<br>    s3_bucket_name          = var.backup_bucket_name<br>    s3_access_key_id        = var.s3_access_key_id<br>    s3_secret_access_key    = var.s3_secret_access_key<br>    retention_policy        = &quot;90d&quot;<br>    schedule                = &quot;0 2 * * *&quot;   # Daily at 2 AM UTC<br>    wal_compression         = &quot;gzip&quot;<br>    data_compression        = &quot;gzip&quot;<br>    target                  = &quot;prefer-standby&quot;<br>    create_scheduled_backup = true<br>  }<br>}</pre><p>So what does this give us?</p><ul><li><strong>3-instance HA cluster</strong> with automatic failover, one primary and two replicas.</li><li><strong>A dedicated database and user</strong> with a Kubernetes Secret containing the connection string, automatically created in the application’s namespace. Your app just mounts the secret, no hardcoded credentials.</li><li><strong>Automated daily backups</strong> at 2 AM UTC with gzip compression, streamed to S3-compatible storage. WAL archiving is also enabled, giving you point-in-time recovery.</li><li><strong>90-day retention</strong> so you can recover from issues discovered long after they happened.</li><li>The prefer-standby target means backups run against a replica, so your primary isn&#39;t impacted.</li></ul><h3>Adding More Databases</h3><p>Need another application to use the same cluster? Just add another entry to the databases list:</p><pre>databases = [<br>  {<br>    name                        = &quot;my-app-postgres&quot;<br>    owner                       = &quot;my_app_user&quot;<br>    password                    = random_password.my_app_user.result<br>    reclaim_policy              = &quot;Retain&quot;<br>    create_connection_secret    = true<br>    connection_secret_namespace = &quot;my-app&quot;<br>  },<br>  {<br>    name                        = &quot;another-service-postgres&quot;<br>    owner                       = &quot;another_service_user&quot;<br>    password                    = random_password.another_service_user.result<br>    reclaim_policy              = &quot;Retain&quot;<br>    create_connection_secret    = true<br>    connection_secret_namespace = &quot;another-service&quot;<br>  }<br>]</pre><p>Each database gets its own user, its own credentials secret, and its own namespace isolation, all on the same HA cluster.</p><h3>Monitoring</h3><p>Remember how the old way required monitoring four or five different services? With CloudNativePG, monitoring comes almost for free. The operator automatically deploys a PodMonitor resource, which means if you&#39;re running Prometheus in your cluster, it will start scraping metrics from your PostgreSQL instances out of the box. No extra configuration needed.</p><p>To visualize those metrics, you can import the official <a href="https://grafana.com/grafana/dashboards/20417-cloudnativepg/">CloudNativePG Grafana dashboard</a>. It gives you a full overview of your cluster: instance health, replication status, connection counts, CPU and memory usage, transaction rates, and more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P9pNuEWT1134gtzs6GyRsA.png" /></figure><p>This is what my production cluster looks like. All three instances healthy, replication in sync, and resource usage at a glance. Compare that to the old way of checking Patroni logs, querying etcd, and verifying HAProxy backends separately.</p><h3>Conclusion</h3><p>The era of “building your own” Postgres HA stack is coming to an end for those running on Kubernetes. CloudNativePG offers a streamlined, “boring” (in a good way!) approach to database management that lets you focus on your data rather than your infrastructure.</p><p>Going from zero to a production-ready, backed-up, highly available PostgreSQL cluster takes just a few Terraform resources. Compare that to wiring up Patroni, etcd, and HAProxy. It’s not even close.</p><p>If you are running on Kubernetes, CloudNativePG isn’t just an option, it’s the modern standard.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a3bc9cf5eba" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Terraform’s New Search Feature: What It Is And Why You Need It]]></title>
            <link>https://medium.com/@pascal.toepke/terraforms-new-search-feature-what-it-is-and-why-you-need-it-173091f75069?source=rss-ede1fb33c222------2</link>
            <guid isPermaLink="false">https://medium.com/p/173091f75069</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <dc:creator><![CDATA[Pascal Töpke]]></dc:creator>
            <pubDate>Mon, 29 Sep 2025 12:30:39 GMT</pubDate>
            <atom:updated>2025-09-29T12:30:39.285Z</atom:updated>
            <content:encoded><![CDATA[<p>Terraform v1.14.0 (currently in beta) will introduce a game-changing feature that many infrastructure engineers have been waiting for: the ability to search and query existing infrastructure directly from Terraform. This new search capability bridges the gap between what’s running in your cloud environment and what’s defined in your Terraform configuration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rbJeVDfN3TnmnF8Ow1corg.png" /></figure><h3><strong>What’s New: The Search Feature in v1.14.0</strong></h3><p>The latest Terraform beta release brings three key components that work together to enable infrastructure search:</p><ol><li><strong>List Resources</strong>: Define searchable queries in *.tfquery.hcl files</li><li><strong>terraform query Command</strong>: Execute searches against existing infrastructure</li><li><strong>Enhanced Import</strong>: Generate Terraform configuration from search results</li></ol><p>This feature addresses a common pain point: discovering and importing existing infrastructure into Terraform management.</p><h3>How It Works: List Resources and .tfquery.hcl Files</h3><p>The search functionality starts with query definition files. These special .tfquery.hcl files contain list resources that define what you want to search for and how to filter the results.</p><p>Here’s a practical example from my demo repository:</p><pre>list &quot;aws_instance&quot; &quot;all&quot; {<br>  provider = aws<br>  config {<br>    filter {<br>      name   = &quot;tag:Name&quot;<br>      values = [&quot;*${var.keyword}*&quot;]<br>    }<br>  }<br>}<br><br>variable &quot;keyword&quot; {<br>  description = &quot;Search keyword to find in instance names&quot;<br>  type        = string<br>  default     = &quot;terraform-query-demo&quot;<br>}</pre><p>This query searches for AWS EC2 instances where the Name contains a specific keyword, using wildcards for flexible matching.</p><h3>Demo: Searching Real Infrastructure</h3><p>Let’s walk through a real-world example using my demo infrastructure. You can follow along with the complete code at: <a href="https://github.com/pascalinthecloud/terraform-query-demo"><strong>https://github.com/pascalinthecloud/terraform-query-demo</strong></a></p><h3>My Demo Setup</h3><p>My demonstration environment includes:</p><ul><li><strong>5 EC2 instances</strong> with different operating systems (Amazon Linux, Ubuntu, Debian, CentOS, Fedora)</li><li><strong>Local Terraform state backend</strong> for simplicity</li><li><strong>Modular file structure</strong> split across logical components</li><li><strong>Random naming suffixes</strong> for unique instance identification</li></ul><p>Each instance is tagged with the pattern: terraform-query-demo-{os}-{random-suffix}, for example:</p><ul><li>terraform-query-demo-amazon-linux-4d667691</li><li>terraform-query-demo-ubuntu-a1b2c3d4</li></ul><p><strong>Important for the demo:</strong> I intentionally commented out the Amazon Linux instance in myaws_instance.tf file and removed it from the Terraform state to simulate a real-world scenario where infrastructure exists but isn&#39;t managed by Terraform yet. This creates the perfect setup to demonstrate the search and import capabilities:</p><p>I also removed it from the state file using:</p><pre>terraform state rm aws_instance.amazon_linux</pre><p>This means the Amazon Linux instance exists in AWS but is completely “unmanaged” from Terraform’s perspective — exactly the kind of infrastructure the new search feature is designed to discover and import.</p><h3>The Search in Action</h3><p>First, let’s run the query without any specific keywords to see all instances matching our project:</p><pre>terraform query</pre><p>This returns all instances with “terraform-query-demo” in their name (the default keyword):</p><pre>Found existing alias for &quot;terraform&quot;. You should use: &quot;tf&quot;<br>list.aws_instance.all   account_id=472447887151,id=i-0d89aef1cfc12eb3a,region=eu-central-1   terraform-query-demo-centos-649c922e<br>list.aws_instance.all   account_id=472447887151,id=i-0b83637676e50fcf1,region=eu-central-1   terraform-query-demo-debian-663b3e94<br>list.aws_instance.all   account_id=472447887151,id=i-07a4277d2788d5578,region=eu-central-1   terraform-query-demo-fedora-d2819184<br>list.aws_instance.all   account_id=472447887151,id=i-0cbdd767de9b57e98,region=eu-central-1   terraform-query-demo-amazon-linux-4d667691<br>list.aws_instance.all   account_id=472447887151,id=i-0d95262f7800a1259,region=eu-central-1   terraform-query-demo-ubuntu-02223f19</pre><p>Perfect! This shows all five instances in our demo environment, including the Amazon Linux instance that’s not managed by our current Terraform configuration.</p><p>Now, let’s search for a specific instance using its random suffix:</p><pre>terraform query -var &#39;keyword=4d66&#39; -generate-config-out=autogenerated.tf</pre><p>This command returns:</p><pre>list.aws_instance.all   account_id=472447887151,id=i-0cbdd767de9b57e98,region=eu-central-1   terraform-query-demo-amazon-linux-4d667691</pre><p>The output shows:</p><ol><li><strong>Searches</strong> for instances containing “4d66” in their name</li><li><strong>Finds</strong> the Amazon instance with suffix “4d667691”</li><li><strong>Displays</strong> the instance details (account ID, instance ID, region, and name)</li><li><strong>Generates</strong> Terraform configuration in autogenerated.tf</li></ol><h3>What Gets Generated (And What Needs Fixing)</h3><p>The generated configuration is comprehensive but requires cleanup because the feature is still experimental. Here’s what you’ll see:</p><h3>Raw Generated Output</h3><pre># __generated__ by Terraform<br># Please review these resources and move them into your main configuration files.<br># __generated__ by Terraform<br>resource &quot;aws_instance&quot; &quot;all_0&quot; {<br>  provider                             = aws<br>  ami                                  = &quot;ami-0a88743759c81c37e&quot;<br>  arn                                  = &quot;arn:aws:ec2:eu-central-1:472447887151:instance/i-0cbdd767de9b57e98&quot;<br>  associate_public_ip_address          = true<br>  availability_zone                    = &quot;eu-central-1a&quot;<br>  disable_api_stop                     = false<br>  disable_api_termination              = false<br>  ebs_optimized                        = false<br>  force_destroy                        = null<br>  get_password_data                    = false<br>  hibernation                          = false<br>  host_id                              = &quot;&quot;<br>  iam_instance_profile                 = &quot;&quot;<br>  id                                   = &quot;i-0cbdd767de9b57e98&quot;<br>  instance_initiated_shutdown_behavior = &quot;stop&quot;<br>  instance_lifecycle                   = &quot;&quot;<br>  instance_state                       = &quot;running&quot;<br>  instance_type                        = &quot;t2.micro&quot;<br>  ipv6_address_count                   = 0<br>  ipv6_addresses                       = []<br>  key_name                             = &quot;terraform-query-demo-key&quot;<br>  monitoring                           = false<br>  outpost_arn                          = &quot;&quot;<br>  password_data                        = &quot;&quot;<br>  placement_group                      = &quot;&quot;<br>  placement_group_id                   = &quot;&quot;<br>  placement_partition_number           = 0<br>  primary_network_interface_id         = &quot;eni-0f1d7ebefe108fe82&quot;<br>  private_dns                          = &quot;ip-172-31-20-171.eu-central-1.compute.internal&quot;<br>  private_ip                           = &quot;172.31.20.171&quot;<br>  public_dns                           = &quot;ec2-18-197-24-55.eu-central-1.compute.amazonaws.com&quot;<br>  public_ip                            = &quot;18.197.24.55&quot;<br>  region                               = &quot;eu-central-1&quot;<br>  secondary_private_ips                = []<br>  security_groups                      = [&quot;terraform-query-demo-sg&quot;]<br>  source_dest_check                    = true<br>  spot_instance_request_id             = &quot;&quot;<br>  subnet_id                            = &quot;subnet-0161e26dd3ccc033f&quot;<br>  tags = {<br>    Name    = &quot;terraform-query-demo-amazon-linux-4d667691-test-change&quot;<br>    OS      = &quot;Amazon Linux 2&quot;<br>    Project = &quot;terraform-query-demo&quot;<br>  }<br>  tags_all = {<br>    Name    = &quot;terraform-query-demo-amazon-linux-4d667691-test-change&quot;<br>    OS      = &quot;Amazon Linux 2&quot;<br>    Project = &quot;terraform-query-demo&quot;<br>    Managed = &quot;terraform&quot;<br>  }<br>  tenancy                     = &quot;default&quot;<br>  user_data                   = null<br>  user_data_replace_on_change = null<br>  volume_tags                 = null<br>  vpc_security_group_ids      = [&quot;sg-05a7ba188cbb57529&quot;]<br>  capacity_reservation_specification {<br>    capacity_reservation_preference = &quot;open&quot;<br>  }<br>  cpu_options {<br>    amd_sev_snp      = &quot;&quot;<br>    core_count       = 1<br>    threads_per_core = 1<br>  }<br>  credit_specification {<br>    cpu_credits = &quot;standard&quot;<br>  }<br>  enclave_options {<br>    enabled = false<br>  }<br>  maintenance_options {<br>    auto_recovery = &quot;default&quot;<br>  }<br>  metadata_options {<br>    http_endpoint               = &quot;enabled&quot;<br>    http_protocol_ipv6          = &quot;disabled&quot;<br>    http_put_response_hop_limit = 1<br>    http_tokens                 = &quot;optional&quot;<br>    instance_metadata_tags      = &quot;disabled&quot;<br>  }<br>  primary_network_interface {<br>    delete_on_termination = true<br>    network_interface_id  = &quot;eni-0f1d7ebefe108fe82&quot;<br>  }<br>  private_dns_name_options {<br>    enable_resource_name_dns_a_record    = false<br>    enable_resource_name_dns_aaaa_record = false<br>    hostname_type                        = &quot;ip-name&quot;<br>  }<br>  root_block_device {<br>    delete_on_termination = true<br>    device_name           = &quot;/dev/xvda&quot;<br>    encrypted             = false<br>    iops                  = 100<br>    kms_key_id            = &quot;&quot;<br>    tags                  = {}<br>    tags_all              = {}<br>    throughput            = 0<br>    volume_id             = &quot;vol-0b106085f0387c032&quot;<br>    volume_size           = 8<br>    volume_type           = &quot;gp2&quot;<br>  }<br>  timeouts {<br>    create = null<br>    delete = null<br>    read   = null<br>    update = null<br>  }<br>}<br>import {<br>  to       = aws_instance.all_0<br>  provider = aws<br>  identity = {<br>    account_id = &quot;472447887151&quot;<br>    id         = &quot;i-0cbdd767de9b57e98&quot;<br>    region     = &quot;eu-central-1&quot;<br>  }<br>}</pre><h3>Issues You’ll Encounter</h3><p>The experimental nature means several attributes cause errors:</p><p><strong>Read-only attributes that must be removed:</strong></p><ul><li>arn, id, instance_state</li><li>primary_network_interface_id</li><li>private_dns, public_dns, public_ip</li><li>password_data, availability_zone</li></ul><p><strong>Conflicting attributes:</strong></p><ul><li>ipv6_address_count vs ipv6_addresses</li><li>placement_group vs placement_group_id</li><li>security_groups vs vpc_security_group_ids</li></ul><p><strong>Invalid values:</strong></p><ul><li>amd_sev_snp = &quot;&quot; (must be &quot;enabled&quot; or &quot;disabled&quot;)</li><li>Empty strings where specific values are required</li></ul><h3>Cleaned Up Configuration</h3><p>After removing problematic attributes, you get a working configuration:</p><pre>resource &quot;aws_instance&quot; &quot;all_0&quot; {<br>  ami                    = &quot;ami-0a88743759c81c37e&quot;<br>  instance_type          = &quot;t2.micro&quot;<br>  key_name               = &quot;terraform-query-demo-key&quot;<br>  subnet_id              = &quot;subnet-0161e26dd3ccc033f&quot;<br>  vpc_security_group_ids = [&quot;sg-05a7ba188cbb57529&quot;]<br>  associate_public_ip_address = true<br>  disable_api_stop       = false<br>  disable_api_termination = false<br>  ebs_optimized         = false<br>  get_password_data     = false<br>  hibernation           = false<br>  instance_initiated_shutdown_behavior = &quot;stop&quot;<br>  monitoring            = false<br>  source_dest_check     = true<br>  tenancy              = &quot;default&quot;<br>  <br>  tags = {<br>    Name    = &quot;terraform-query-demo-amazon-linux-4d667691-test-change&quot;<br>    OS      = &quot;Amazon Linux 2&quot;<br>    Project = &quot;terraform-query-demo&quot;<br>    Managed = &quot;terraform&quot;<br>  }<br><br>  cpu_options {<br>    core_count       = 1<br>    threads_per_core = 1<br>  }<br>  credit_specification {<br>    cpu_credits = &quot;standard&quot;<br>  }<br>  enclave_options {<br>    enabled = false<br>  }<br>  maintenance_options {<br>    auto_recovery = &quot;default&quot;<br>  }<br>  metadata_options {<br>    http_endpoint               = &quot;enabled&quot;<br>    http_protocol_ipv6          = &quot;disabled&quot;<br>    http_put_response_hop_limit = 1<br>    http_tokens                 = &quot;optional&quot;<br>    instance_metadata_tags      = &quot;disabled&quot;<br>  }<br>  private_dns_name_options {<br>    enable_resource_name_dns_a_record    = false<br>    enable_resource_name_dns_aaaa_record = false<br>    hostname_type                        = &quot;ip-name&quot;<br>  }<br>  root_block_device {<br>    delete_on_termination = true<br>    encrypted             = false<br>    iops                  = 100<br>    volume_size           = 8<br>    volume_type           = &quot;gp2&quot;<br>  }<br>}</pre><h3>Real-World Use Cases and Benefits</h3><p>This search feature solves several common infrastructure challenges:</p><h4>1. Brownfield Infrastructure Import</h4><ul><li>Search for existing resources that need to be brought under Terraform management</li><li>Generate configuration for resources that were created manually or by other tools</li><li>Incrementally adopt Terraform without recreating existing infrastructure</li></ul><h4>2. State Recovery and Reconstruction</h4><ul><li>Rebuild lost or corrupted Terraform state by searching for existing resources</li><li>Generate configuration files when original Terraform code is missing</li><li>Recover from state file disasters by rediscovering infrastructure</li></ul><h4>3. Configuration Generation</h4><ul><li>Automatically generate Terraform configuration from existing resources</li><li>Reduce manual effort in writing resource definitions from scratch</li><li>Ensure configuration matches actual deployed infrastructure settings</li></ul><h4>4. Terraform Migration Projects</h4><ul><li>Import resources from legacy infrastructure as code tools</li><li>Convert manually managed infrastructure to Terraform</li><li>Migrate from other IaC platforms by searching and importing existing resources</li></ul><h3>⚠️ Current Limitations (Experimental Feature)</h3><ul><li><strong>Manual cleanup required</strong>: Generated configs need significant editing</li><li><strong>Provider support varies:</strong> At this stage, only a subset of providers and resources are supported (currently confirmed for AWS).</li><li><strong>Filtering options are limited:</strong> Advanced query patterns are not yet available (e.g., negative queries like <em>“VMs not tagged with </em><em>managed=terraform”</em> are not currently possible).</li></ul><h3>Looking Forward: The Future of Infrastructure Search</h3><p>While still experimental, this feature represents a significant step forward in Terraform’s evolution. Future improvements will likely include:</p><ul><li><strong>Better generated configurations</strong> with fewer manual fixes needed</li><li><strong>Enhanced filtering capabilities</strong> for complex searches</li><li><strong>Support for more resource types</strong> across different providers</li><li><strong>Query optimization</strong> for large infrastructure environments</li></ul><h3>Conclusion</h3><p>Terraform’s new search feature marks an important milestone in infrastructure management tooling. While third-party tools like <a href="https://github.com/GoogleCloudPlatform/terraformer">Terraformer</a> have provided ways to generate Terraform code from existing infrastructure, Terraform now offers this capability natively with more targeted, query-based discovery.</p><p>This provides a middle ground between manual import processes and bulk-generation tools like Terraformer — targeted discovery and generation based on specific queries rather than importing entire resource types.</p><p><em>Note: I haven’t personally used Terraformer in production environments, so I’d greatly appreciate feedback from readers who have experience with both tools to share how they compare in real-world scenarios.</em></p><p>The ability to bridge the gap between existing infrastructure and Terraform configuration with native, query-based discovery will accelerate Terraform adoption and make infrastructure as code more accessible to teams managing complex, pre-existing environments.</p><p>Try it out with my demo repository at <a href="https://github.com/pascalinthecloud/terraform-query-demo"><strong>https://github.com/pascalinthecloud/terraform-query-demo</strong></a> and experience firsthand how Terraform’s native search capabilities can transform your infrastructure management workflow.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=173091f75069" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting Real Client IPs with Istio Gateway and PROXY Protocol v2 on OVH Cloud]]></title>
            <link>https://medium.com/@pascal.toepke/getting-real-client-ips-with-istio-gateway-and-proxy-protocol-v2-on-ovh-cloud-4064fbd2ed51?source=rss-ede1fb33c222------2</link>
            <guid isPermaLink="false">https://medium.com/p/4064fbd2ed51</guid>
            <category><![CDATA[load-balancing]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[ovh]]></category>
            <dc:creator><![CDATA[Pascal Töpke]]></dc:creator>
            <pubDate>Thu, 04 Sep 2025 09:32:37 GMT</pubDate>
            <atom:updated>2025-09-04T09:32:37.039Z</atom:updated>
            <content:encoded><![CDATA[<p><em>How to configure OVH’s OpenStack-based load balancers with Istio to preserve client IP addresses</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Oej40ETB6jv_mzQHYHMSw.png" /></figure><h3>The Problem</h3><p>When running Istio service mesh in production with external load balancers on <strong>OVH Cloud</strong>, you quickly run into a common issue: <strong>your applications only see the load balancer’s IP address, not the real client IP</strong>. This breaks logging, rate limiting, geolocation, and security policies that depend on knowing where requests actually come from.</p><p>In a typical OVH Cloud setup:</p><ul><li>Client (IP: 203.0.113.50) → OVH Load Balancer → Istio Gateway → Your App</li><li>Your app sees: 10.0.1.100 (load balancer IP) ❌</li><li>Your app should see: 203.0.113.50 (real client IP) ✅</li></ul><h3>Why This Happens</h3><p>Most cloud load balancers, including <strong>OVH’s Octavia load balancers</strong> (based on OpenStack), perform Network Address Translation (NAT), replacing the original client IP with their own. While HTTP headers like X-Forwarded-For can help, they&#39;re not always reliable and can be spoofed.</p><p><strong>PROXY Protocol</strong> solves this by having the load balancer prepend connection metadata (including the original client IP) to the data stream in a standardized format that the backend can trust.</p><h3>The Solution: PROXY Protocol v2 + Istio on OVH Cloud</h3><p>Here’s how to configure your infrastructure to preserve real client IPs using <strong>OVH’s OpenStack-based Octavia load balancers</strong> with Istio Gateway.</p><h3>Step 1: Configure Your OVH Load Balancer Service</h3><p>First, enable PROXY Protocol v2 on your Kubernetes LoadBalancer service using OVH-specific annotations:</p><pre>service:<br>  type: LoadBalancer<br>  annotations:<br>    # OVH Cloud specific annotations<br>    loadbalancer.ovhcloud.com/class: octavia<br>    loadbalancer.ovhcloud.com/flavor: small<br>    service.beta.kubernetes.io/openstack-internal-load-balancer: &quot;true&quot;<br>    # OpenStack/Octavia PROXY Protocol configuration<br>    loadbalancer.openstack.org/proxy-protocol: &quot;v2&quot;<br>  spec:<br>    externalTrafficPolicy: Local  # Critical for IP preservation</pre><p><strong>Key points for OVH Cloud:</strong></p><ul><li>loadbalancer.ovhcloud.com/class: octavia specifies OVH&#39;s Octavia service</li><li>loadbalancer.ovhcloud.com/flavor controls the load balancer size on OVH</li><li>loadbalancer.openstack.org/proxy-protocol: &quot;v2&quot; enables PROXY Protocol v2 (OpenStack standard)</li><li>service.beta.kubernetes.io/openstack-internal-load-balancer keeps traffic within OVH&#39;s private network</li><li>externalTrafficPolicy: Local prevents additional network hops that would lose the client IP</li></ul><h3>Step 2: Configure Your Istio Gateway</h3><p>Add the PROXY Protocol annotation to your Gateway resource:</p><pre>apiVersion: networking.istio.io/v1beta1<br>kind: Gateway<br>metadata:<br>  name: my-gateway<br>  namespace: istio-system<br>  annotations:<br>    &quot;proxy.istio.io/config&quot;: &#39;{&quot;gatewayTopology&quot; : { &quot;proxyProtocol&quot;: {} }}&#39;<br>spec:<br>  selector:<br>    istio: gateway<br>  servers:<br>    - port:<br>        number: 80<br>        name: http<br>        protocol: HTTP<br>      hosts:<br>        - myapp.example.com<br>    - port:<br>        number: 443<br>        name: https<br>        protocol: HTTPS<br>      tls:<br>        mode: SIMPLE<br>        credentialName: my-tls-cert<br>      hosts:<br>        - myapp.example.com</pre><h3>Step 3: Configure Envoy with EnvoyFilter</h3><p>This is the most critical step. Istio’s Envoy proxy needs to understand PROXY Protocol headers sent by OVH’s Octavia load balancers:</p><pre>apiVersion: networking.istio.io/v1alpha3<br>kind: EnvoyFilter<br>metadata:<br>  name: proxy-protocol<br>  namespace: istio-system<br>spec:<br>  workloadSelector:<br>    labels:<br>      istio: gateway<br>  configPatches:<br>    # HTTPS listener configuration<br>    - applyTo: LISTENER<br>      match:<br>        context: GATEWAY<br>        listener:<br>          portNumber: 443<br>      patch:<br>        operation: MERGE<br>        value:<br>          listener_filters:<br>            - name: envoy.filters.listener.proxy_protocol<br>              typed_config:<br>                &quot;@type&quot;: type.googleapis.com/envoy.extensions.filters.listener.proxy_protocol.v3.ProxyProtocol<br>            - name: envoy.filters.listener.tls_inspector<br>              typed_config:<br>                &quot;@type&quot;: type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector<br>            - name: envoy.filters.listener.http_inspector<br>              typed_config:<br>                &quot;@type&quot;: type.googleapis.com/envoy.extensions.filters.listener.http_inspector.v3.HttpInspector<br>    # HTTP listener configuration<br>    - applyTo: LISTENER<br>      match:<br>        context: GATEWAY<br>        listener:<br>          portNumber: 80<br>      patch:<br>        operation: MERGE<br>        value:<br>          listener_filters:<br>            - name: envoy.filters.listener.proxy_protocol<br>              typed_config:<br>                &quot;@type&quot;: type.googleapis.com/envoy.extensions.filters.listener.proxy_protocol.v3.ProxyProtocol<br>            - name: envoy.filters.listener.http_inspector<br>              typed_config:<br>                &quot;@type&quot;: type.googleapis.com/envoy.extensions.filters.listener.http_inspector.v3.HttpInspector</pre><p><strong>Why this configuration matters:</strong></p><ul><li>The PROXY Protocol filter must be <strong>first</strong> in the listener filter chain</li><li>For HTTPS, it must come before the TLS inspector</li><li>Both HTTP (80) and HTTPS (443) listeners need configuration</li></ul><h3>Understanding the Flow on OVH Cloud</h3><p>After configuration, here’s what happens in the OVH Cloud environment:</p><ol><li><strong>Client</strong> sends request to your domain</li><li><strong>DNS</strong> resolves to OVH load balancer IP</li><li><strong>OVH’s Octavia Load Balancer</strong> receives request, prepends PROXY Protocol v2 header with client IP</li><li><strong>Istio Gateway</strong> (running on OVH’s OpenStack infrastructure) receives PROXY Protocol header, extracts client IP</li><li><strong>Envoy</strong> sets proper X-Forwarded-For headers</li><li><strong>Your Application</strong> receives request with real client IP in headers</li></ol><h3>Private vs Public Load Balancers</h3><ul><li>Use service.beta.kubernetes.io/openstack-internal-load-balancer: &quot;true&quot; for internal load balancers</li><li>Remove this annotation for public-facing load balancers</li><li>Internal load balancers are more secure and cost-effective for private applications</li></ul><h3>Health Checks on OVH</h3><p>OVH’s Octavia load balancers automatically configure health checks on port 15021 (Istio’s status port). This port doesn’t use PROXY Protocol, so health checks work seamlessly.</p><h3>Troubleshooting on OVH Cloud</h3><h3>Common Issues</h3><p><strong>“Bad Request” errors after enabling PROXY Protocol:</strong></p><ul><li>Check that the EnvoyFilter is applied: kubectl get envoyfilter -n istio-system</li><li>Verify the workload selector matches your gateway pods</li><li>Ensure PROXY Protocol filter comes first in the chain</li></ul><p><strong>Load balancer shows “degraded” member status:</strong></p><ul><li>This is normal during initial configuration</li><li>OVH’s health checks run on port 15021 without PROXY Protocol</li><li>Status should become healthy within a few minutes</li></ul><p><strong>Still seeing OVH load balancer IPs:</strong></p><ul><li>Verify externalTrafficPolicy: Local is set</li><li>Check that the EnvoyFilter targets the correct gateway pods</li><li>Enable debug logging to see what Envoy receives</li></ul><h3>Debug Commands</h3><pre># Check load balancer status<br>kubectl get service -n istio-system -o yaml | grep -A 10 annotations<br><br># Enable debug logging on Istio gateway<br>kubectl exec &lt;gateway-pod&gt; -n istio-system -- \<br>  curl -X POST &quot;localhost:15000/logging?level=debug&quot;<br><br># Check health status<br>kubectl exec &lt;gateway-pod&gt; -n istio-system -- \<br>  curl localhost:15021/healthz/ready<br><br># Verify EnvoyFilter is applied<br>kubectl get envoyfilter -n istio-system -o yaml</pre><h3>Why Choose OVH Cloud for This Setup?</h3><p><strong>OVH Cloud’s advantages for PROXY Protocol:</strong></p><ul><li><strong>OpenStack Native</strong>: Built on proven OpenStack technology</li><li><strong>Octavia Integration</strong>: Full support for advanced load balancing features</li><li><strong>European Data Sovereignty</strong>: Servers located in Europe with GDPR compliance</li><li><strong>Cost Effective</strong>: Competitive pricing for enterprise-grade infrastructure</li><li><strong>No Vendor Lock-in</strong>: Standard OpenStack APIs and Kubernetes integration</li></ul><h3>Security Considerations</h3><ul><li><strong>Network isolation</strong>: Use OVH’s private networks for internal communication</li><li><strong>Monitoring</strong>: Monitor for unusual IP patterns that might indicate spoofing attempts</li><li><strong>GDPR compliance</strong>: OVH’s European infrastructure helps with data residency requirements</li></ul><h3>Conclusion</h3><p>Implementing PROXY Protocol v2 with Istio on <strong>OVH Cloud</strong> requires coordination between OVH’s OpenStack-based load balancers, Kubernetes services, and Envoy configuration. While it involves several moving parts, the result is accurate client IP tracking that’s essential for production applications.</p><p>The key takeaways for OVH Cloud:</p><ul><li>Use OVH-specific annotations for Octavia load balancers</li><li>Leverage externalTrafficPolicy: Local to preserve client IPs through Kubernetes</li><li>Configure both OVH’s load balancer and Envoy to handle PROXY Protocol</li><li>Order matters: PROXY Protocol filter must come first</li><li>Health checks automatically work on port 15021</li></ul><p>With this setup on OVH Cloud, your applications will finally see real client IPs, enabling better logging, security policies, and user analytics while benefiting from OVH’s reliable OpenStack infrastructure.</p><p><em>Running Istio on OVH Cloud? Share your experiences and configurations in the comments below!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4064fbd2ed51" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Enable Prometheus Scraping with Istio mTLS]]></title>
            <link>https://medium.com/@pascal.toepke/how-to-enable-prometheus-scraping-with-istio-mtls-746747864fac?source=rss-ede1fb33c222------2</link>
            <guid isPermaLink="false">https://medium.com/p/746747864fac</guid>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[metrics]]></category>
            <dc:creator><![CDATA[Pascal Töpke]]></dc:creator>
            <pubDate>Wed, 11 Jun 2025 10:42:50 GMT</pubDate>
            <atom:updated>2025-09-04T09:15:41.514Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wqXojWi3_uVEaVHocu_qOA.png" /></figure><p>When you’re running applications in Kubernetes with Istio and have <strong>mutual TLS (mTLS)</strong> enabled, one of the first things you’ll notice is that Prometheus scraping stops working. Especially if you’re monitoring services like Keycloak, which run behind an Istio sidecar, accessing their /metrics endpoint becomes a bit more involved.</p><p>In this post, I’ll walk you through how I configured Prometheus to scrape metrics securely over mTLS with Istio. This setup uses the Prometheus Operator and assumes sidecar injection is enabled for Prometheus.</p><h3>📉 The Problem</h3><p>Once mTLS is enforced (e.g. using a PeerAuthentication with STRICT mode), Istio requires encrypted communication between all workloads. Prometheus, however, doesn’t natively speak Istio mTLS, and even when using client certificates, it can’t verify Istio’s SPIFFE-based SANs.</p><p>So when scraping, you might see:</p><pre>tls: failed to verify certificate: x509: certificate is not valid for any names<br>or <br>http: TLS handshake error from 127.0.0.1:54256: client sent an HTTP request to an HTTPS server&quot;</pre><h3>✅ The Goal</h3><p>Allow Prometheus to scrape <strong>/metrics</strong> (e.g., from Keycloak) over HTTPS using Istio-issued mTLS certificates.</p><h3>🛠️ Step-by-Step Setup</h3><h4>1. Annotate the Prometheus Pod for Certificate Access</h4><p>Istio can output mTLS certificates into a volume if configured via annotations. Here’s what the prometheus.prometheusSpec.podMetadata section might look like in your values.yaml or Prometheus CR:</p><pre>podMetadata:<br>  annotations:<br>    proxy.istio.io/config: |<br>      proxyMetadata:<br>        OUTPUT_CERTS: /etc/istio-output-certs<br>    sidecar.istio.io/userVolumeMount: |<br>      [{&quot;name&quot;: &quot;istio-certs&quot;, &quot;mountPath&quot;: &quot;/etc/istio-output-certs&quot;}]<br>  labels:<br>    sidecar.istio.io/inject: &quot;true&quot;</pre><p>Mount the certs in Prometheus:</p><pre>volumeMounts:<br>  - mountPath: /etc/prom-certs/<br>    name: istio-certs<br>volumes:<br>  - name: istio-certs<br>    emptyDir:<br>      medium: Memory</pre><blockquote>This makes Istio-issued <strong>cert-chain.pem</strong>, <strong>key.pem</strong>, and <strong>root-cert.pem</strong> available to Prometheus.</blockquote><h4>2. Configure ServiceMonitor with mTLS</h4><pre>endpoints:<br>  - port: http<br>    path: /realms/master/metrics<br>    interval: 15s<br>    scheme: https<br>    tlsConfig:<br>      caFile: /etc/prom-certs/root-cert.pem<br>      certFile: /etc/prom-certs/cert-chain.pem<br>      keyFile: /etc/prom-certs/key.pem<br>      insecureSkipVerify: true</pre><ul><li><em>scheme: https</em> is required.</li><li><em>insecureSkipVerify: true</em> is necessary due to Istio&#39;s non-standard SANs.</li><li>The port name <strong>must match</strong> the name in the target service (e.g., port: http).</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=746747864fac" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>