<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by alexis richardson on Medium]]></title>
        <description><![CDATA[Stories by alexis richardson on Medium]]></description>
        <link>https://medium.com/@monadic?source=rss-c1c94bbe89c2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*jv8lZ1VThdiO9QYR.jpg</url>
            <title>Stories by alexis richardson on Medium</title>
            <link>https://medium.com/@monadic?source=rss-c1c94bbe89c2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 01:09:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@monadic/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[No human in the loop? Agents manage a Sveltos fleet in ConfigHub]]></title>
            <link>https://medium.com/@monadic/no-human-in-the-loop-agents-manage-a-sveltos-fleet-in-confighub-178205600338?source=rss-c1c94bbe89c2------2</link>
            <guid isPermaLink="false">https://medium.com/p/178205600338</guid>
            <dc:creator><![CDATA[alexis richardson]]></dc:creator>
            <pubDate>Fri, 06 Nov 2026 11:46:01 GMT</pubDate>
            <atom:updated>2026-11-06T11:46:01.532Z</atom:updated>
            <content:encoded><![CDATA[<p>Can AI SRE manage a Kubernetes fleet with no human in the loop?</p><p>We ran a real example to find out how ConfigHub approvals can direct and constrain agents on a Sveltos Kubernetes fleet. Every time something goes wrong, an agent fixes the problem and then writes a policy to stop it happening again.</p><h3>How the AI works</h3><p>• A <strong>Devil Agent</strong> introduces outages on purpose.</p><p>• An <strong>Angel Agent</strong> proposes a fix, and a new policy that prevents recurrence.</p><p>• Neither the Devil Agent nor the Angel Agent can approve anything, including its own work, so a third agent, <strong>Milton</strong>, approves or refuses. In our second run it approved nineteen changes and refused one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/1*6Q75HI8kX0rc1Xl8hgZglQ.png" /><figcaption>a simple agentic loop</figcaption></figure><h3>Why Sveltos and ConfigHub</h3><p>ConfigHub is a control system for agents on Kubernetes. It lets operations and approval gates be applied in bulk across a fleet. For example such as promotions and rollouts. In ConfigHub we maintain a representation of the fleet as live graph of configuration objects. Each object maps 1–1 to a live object (and vice versa). Changes are made to live clusters by updating the configuration and reconciling with GitOps over OCI. <a href="https://confighub.com/blog/simpler-faster-and-safer-changes-with-confighub-ba1bc7328aef">As shown here</a> you can insert ConfigHub wherever you have an OCI image store and GitOps.</p><p>In this blog we use Sveltos as our GitOps deployment agent, since it can deliver and coordinate changes across a fleet. You could also use Argo or Flux and we have experimental support for additional technologies.</p><p>Sveltos and ConfigHub work together through a plugin <a href="https://confighub.com/blog/simpler-faster-and-safer-fleets-with-confighub-and-sveltos-d74b9808ccee">we described in a recent blog post</a>. They do three jobs at once on one Kubernetes fleet: <strong>fleets</strong>, <strong>compliance</strong> and <strong>agentic operations</strong>. Sveltos delivers each cluster’s configuration and reports its health.</p><p>ConfigHub holds the configuration, the policies, and the record of every change and approval, the agents’ own included.</p><h3>The fleet</h3><p>A small web ‘shop’ application has two services, api and web, both Python. Plus one platform component: Stakater Reloader, which restarts pods when a secret changes. Five Kubernetes (kind) clusters: one management cluster running Sveltos, one staging and three labelled ‘production’.</p><p>Every cluster has exactly the desired configuration that was asked for but two nodes are down. They are labelled <strong><em>region=us</em></strong> below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/1*nAaJ8TP2JczgsYBE1Ds5nA.png" /><figcaption>Every cluster has the desired configuration but two labelled region=us are down</figcaption></figure><h3>The three outage scenarios</h3><p>In each outage the web shop stops working for the people using it. And so we need to fix the fleet and set policies to prevent recurrence.</p><ol><li><strong>A rotated token is “missed”.</strong> The shop stops serving on all four workload clusters. Its API token has been rotated in each cluster directly, which is how a secret store or a security team would do it. The web service reads that token once when it starts and holds it for the life of the process, so it carries on with the old value and its pods stop reporting as available. ConfigHub UI displays the web shop as broken a minute later</li><li><strong>A change that only reaches half the fleet.</strong> The web shop goes down on the two clusters in the United States and stays up on the two in Europe. A Sveltos delivery profile has been <strong><em>applied by hand on the management cluster</em></strong>. It selects clusters by the label region=us and delivers a network policy to them that blocks traffic to the shop. Two of the four workload clusters are carrying that label. Nothing was stored in the source repository to recorded that this profile existed.</li><li><strong>The nightly repave goes wrong.</strong> The three production clusters are rebuilt from scratch overnight, as part of a regular security practice in which the whole stack is re-based onto newly hardened images. They come back with no shop running on them. This must be fixed.</li></ol><h3>When a repave goes wrong</h3><p>Security teams ask for the <a href="https://blogs.vmware.com/tanzu/the-three-r-s-of-enterprise-security-rotate-repave-and-repair/">three Rs: rotate credentials often, repave servers from a known good state, and repair what is vulnerable quickl</a>y. Outage 1 was a rotation example. Outage 3 shows arepave example.</p><p>Ideally a cluster rebuilt from nothing should re-fetch everything from its source of truth but for some reason two things went wrong:</p><ol><li>Sveltos sent nothing to the rebuilt clusters. It still held its own record saying it had already delivered everything, and nothing had told it these clusters were new. ConfigHub reads that record, so the dashboard said Healthy while production was empty.</li><li>Once delivery <strong><em>does</em></strong> resume, the shop comes back without its API token. That token was created by hand in each cluster, and no release contained it.</li></ol><p><em>A monitoring system can confidently reporting health for a cluster with nothing running on it.</em></p><p>Below: An agent’s repair is shown as a config change promotion. There is a named change on two fields. This is seen moving through the same stages as any other change, and stopped at a gate. The header will read Degraded until the change has been shipped to the correct live clusters by Sveltos.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/1*N1J6yMHSqu_itzd4L5hIoA.png" /><figcaption>An agent’s repair is moving through the same stages as any other change, and stopped at a gate. The header still reads Degraded, because nothing has shipped to any live clusters yet.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/1*OmhV9JAnN7KGBFetCd_vXA.png" /><figcaption>agents using confighub via the CLI — transcript</figcaption></figure><h3>What we learnt</h3><p>Nobody was watching when the first outage happened. The clusters reported it themselves, and then ConfigHub showed the web shop as broken around 80 seconds later.</p><p>Before a new policy was allowed out, we ran it against every object already running on the fleet, on a spare cluster kept for nothing else. On the first outage, that mattered. As things then stood, the policy would have rejected the shop on all four clusters. So the fix had to go out first, and it did.</p><blockquote>The refusal step is shown in the Devil Agent’s transcript above. It rotates the token a second time, the shop survives, so it copies the old manifest into a new deployment instead. The cluster rejects it and names the policy that stopped it.</blockquote><p>The 3rd agent, Milton, refuses to accept one of the changes: a production memory limit cut that staging has not yet run and has not proven to be safe. We stopped that run and production rollout as a result. Note that each policy is stored <em>with the outage that caused it, the cases it was tested against, and who approved it.</em> Humans can find it all later.</p><h3>Could you not do this with Git?</h3><p>ConfigHub is a new control system solution for managing all of your configuration as data along with value added workflows and controls. The system inputs and outputs OCI images making it easy to add to your GitOps.</p><p>Why not just use Git for all this? One reason is how we handle compliance next to the unique per-node configs and promotion flows.</p><p>Two of the three preventions are Kubernetes admission policies. Git can hold those.</p><p>The third cannot be done from a repository. Before a policy ships we run it against every object the fleet is currently running, to see what it would reject. A repository holds what the clusters were told to run. It does not hold what they are running.</p><p>The same is true of the gate. Each agent has credentials that can change configuration and cannot change a workflow or a trigger. Git has branch protection, which does this for people opening pull requests. These agents never open one.Then there is the question of how we record the changes. Six months after an outage, somebody may ask why a policy exists. In this demo the answer is stored beside the policy: the outage that caused it, the cases it was tested against, and the name of whoever approved it. Commit history can be made to yield some of that, with discipline and careful messages. Here it is simply another thing that the ConfigHub store keeps.</p><p>That is work ConfigHub is doing in this story. It holds the configuration as data, publishes each release as an OCI artifact for Sveltos to apply, and keeps the approvals and the reasons attached to the things they belong to.</p><h3>Try it yourself</h3><p>You can reproduce this on your own machine using kind clusters standing in for production. It needs a Mac, Docker with room for six kind clusters, a ConfigHub organization where you are an admin, and Claude Code.</p><p>• Clone confighub/sveltos-confighub and run bash demo/standup.sh — check.</p><p>• Stand up the fleet, about seven minutes, then onboard it into ConfigHub. Twenty minutes in all before the first outage.</p><p>• Run the three outages in order, from demo/scenarios/.</p><p><strong>Write your own outage:</strong> Our three outages are available as <strong><em>prompt files</em></strong> in demo/prompts/, and the Devil Agent’s standing instructions are in demo/agents/devil.md. Write your own, point the Devil Agent at the fleet, and see whether the Angel Agent finds it and whether Milton lets the fix through. Our three preset failures are just a starting set :-)</p><p>To use this on a Sveltos fleet you already run: install the plugin and follow the onboarding guide instead. The same pattern works for platform components and GPU stacks: examples/gpu-operator in the same repo takes an NVIDIA GPU operator fleet through it and upgrades the driver on staging before production.</p><p><strong>Where to read more</strong></p><p>Everything below is in the same repository, github.com/confighub/sveltos-confighub.</p><ul><li><strong>The code and how to run it</strong>: demo/README.md</li><li><strong>A guide for your own AI agent to read</strong>, covering how to answer questions about this post and how to reproduce the examples: blog/2026–11-ai-chaos-in-production/agent-guide.md</li><li><strong>The whole story rewritten at a slower ‘walking pace’</strong>: blog/2026–11-ai-chaos-in-production/intro.md</li><li><strong>Glossary: </strong>Space, unit, variant, change order, release, trigger, live status: demo/GLOSSARY.md</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=178205600338" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is my configuration repo a database?]]></title>
            <link>https://medium.com/@monadic/is-my-configuration-repo-a-database-9df165182b25?source=rss-c1c94bbe89c2------2</link>
            <guid isPermaLink="false">https://medium.com/p/9df165182b25</guid>
            <category><![CDATA[gitops]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[configuration]]></category>
            <category><![CDATA[confighub]]></category>
            <dc:creator><![CDATA[alexis richardson]]></dc:creator>
            <pubDate>Fri, 02 Oct 2026 10:14:51 GMT</pubDate>
            <atom:updated>2026-10-02T10:14:51.435Z</atom:updated>
            <content:encoded><![CDATA[<p>A customer told me: “In GitOps, the number one failure mode is when people start treating Git like it is a database when it is a file system”. This is an extremely insightful comment because we <em>need a reliable system of record for desired state</em>, and those are normally databases.</p><p>So we must ask: <em>should we also treat Git and its leading implementations as a database?</em></p><p>This customer runs large repos each with hundreds of users. For them the Git-database problem is so obvious that it doesn’t need spelling out. But if we don’t name and describe problems they never get solved. I am here to tell you that:</p><blockquote>Treating Git as a Database leads to problems! How you represent, store and manage configuration is critical for software operations. Especially when you introduce generative AI and agents at any scale.</blockquote><p>In this blog I’ll ask seven questions to help you see if you’re treating Git as a database, explain why each is a real business problem, and leave you with some tools to check it for yourself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*PiWRRIlod0wcg15atWktAQ.png" /><figcaption>US treasury 1920s</figcaption></figure><h3>What Git likes and What Git doesn’t like</h3><p>Let’s start with how we use Git today due to <em>Config as Code.</em></p><p>We observe two elements:</p><ol><li>In config-as-code, we represent configuration using source code and then we generate configuration data from that code</li><li>We store and manage it all as text in files in folders in Git code repos</li></ol><p>We have covered the generation of configuration from templates <a href="https://confighub.com/blog/abstractions-as-views-over-configuration-instead-of-just-generators-of-configuration-8ec6ea8aa746">in other posts</a>. Let’s look at the storage element here: Git is a distributed versioned file store organised into folders like an operating system. The transaction model is optimized for teams working on different parts of one code base. Git is good at ordered, attributed, reviewable history of text. And we need ordering, attribution and review in a GitOps system of record.</p><p>The trouble starts when a repository becomes the operational store for a fleet. It then takes on the jobs of a database: keys, indexing, constraints, queries and transactions, potentially in bulk and at machine speed. Increasingly there may be machine data involved too. Operations may involve transactions across many data fields and objects at once. Git lacks the machinery to support this in a proper way that is fast, safe, scalable, reliable.</p><p>Without this machinery, modern teams have to create workarounds. This has happened before. Homebrew, CocoaPods, Go modules and Nixpkgs all started with a Git index and moved off it. They ‘hit a wall’ when too many users need to do something “ungitly” that Git doesn’t want to do.</p><p>Git has problems with managing configuration too. As well as the problem of how to store config, a complicating factor is <em>the representation of config as code</em>. Templating is fine, and so is Helm. What is missing is a store that can manage the mutation and fanout of one config code statement into multiple derived config data objects, without losing critical information. The rendered result may need to be addressed, queried or diffed. Today it is lost in CI and clusters.</p><p>OK now on to the fun part — the seven questions! I have divided them into three sections: Deployment issues, Promotions and Rollouts.</p><h3>How do we map and approve deployments</h3><h3>1. Am I using filenames as primary keys?</h3><p>Open your Git repos which store configuration. Do you have paths like this?</p><blockquote>clusters/&lt;org&gt;-&lt;env&gt;-&lt;region&gt;.yaml</blockquote><p>Databases have keys, with important uniqueness and referential properties. That filename is acting as a composite key. Your ‘config store in Git’ has directories instead of tables, and they have no type, no uniqueness check, no referential integrity. Ask: what happens when I rename a cluster? Or several clusters? Or a region? Or if someone adds a new app team?</p><p>A rename is now “a delete plus an insert”. The same fields are repeated inside every cluster and renaming one cluster touches about 20 files. This is because your ‘data’ is represented as text filenames.</p><p>Another issue is adding a value, and error handling:</p><ul><li>In a database a new enum value failure is explicit and seen by consumers</li><li>In a folder scheme the lookup can miss, skip a layer, and report nothing</li></ul><p>For example, suppose someone restructures <em>org/&lt;name&gt;.yaml</em> into <em>org/&lt;name&gt;/&lt;env&gt;.yaml</em> and some old lines stay in place. Suppose <em>ignoreMissingValueFiles: true. </em>Now some lookups resolve to nothing.</p><p>Clusters really do differ, and that is fine. In a database, we are equipped to handle variety and can describe variant clusters, apps, regions, teams, just as we represent bank accounts, cash, and customers in a business database.</p><p>Imagine describing a bank system as a set of files and folders, repos and orgs. You would have a sprawl of complex fields, text, documents and relations between them all. This breaks security and reliability. I don’t want to leave my money in a bank that doesn’t use a database, and I don’t think that modern (agentic) operations should be used without a database either.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9eoGzJHLEmyuwoQeBNy3gQ.jpeg" /><figcaption>danske bank 1960s</figcaption></figure><h3>2. Am I approving input queries, but deploying output values?</h3><p>With agentic applications, every enterprise wants compliance and controls. <em>That means approvals are cool again. Yay! </em>But when you approve a PR, what exactly are you approving? Your review may happen before rendering. If so then what downstream values will get deployed, that depends on your review and approval of the upstream query?</p><p>For example: In traditional devops with config as code tools in Git, you may <em>review a values change</em>. What ships is a rendered manifest. But the render happens after review and before apply. That creates rendering and application risks that are not transparent or approved.</p><p>We have seen customers doing this across their whole estate. For example: over a thousand templated sources and no rendered manifests stored anywhere. Every approval was an <em>“approval of an input”.</em></p><p>Argo CD has started on this with Source Hydrator, a clever workaround which is in beta. This renders manifests before committing them back to Git repos. That may fix what one Application renders to. It does not tell you how many Applications re-render when you change a shared file.</p><h3>When is a promotion actually ‘done’</h3><h3>3. In my promotion flow, are transactional changes implemented as find-and-replace scripts?</h3><p>A promotion is a sequence of approved changes to a system <a href="https://confighub.com/blog/gitops-in-the-age-of-ai-4-977d800175f4">as explained here</a>. Normally we would think of a promotion as a state change: this artifact, verified here, is now admitted there. In a database, that would be a sequence of ACID transactions.</p><p>In a file store, promotions involve updates to text files, via editing, and potentially some remapping of files into new folders. You might see a text edit repeated once per target group. Keeping a record that this happened, and can be reversed, or replicated, requires extra machinery. That is why CICD tools exist even when customers adopt GitOps successfully. Eg. <a href="https://docs.kargo.io/user-guide/core-concepts">Kargo</a> already treats promotion as an object with an identity, stages and verification.</p><p>In one estate we looked at, a version string was repeated a median of seven times inside one definition, and over forty times in the worst. A partial replace is valid YAML, passes lint, and leaves the fleet on two versions with no record that anyone chose that.</p><p>Ask to see where it says this version was promoted to production, when, and who admitted it? Is it a script whose job is to ‘do the edit safely’? That’s a warning sign.</p><h3>4. Can I consistently distinguish “in progress” changes from “abandoned”?</h3><p>Do you know which of your charts are mid-promotion right now, and which ones stalled? Do you need to ask a colleague or check their text notes? This matters. A rollout that is 80% done and one we abandoned weeks ago can leave identical Git repositories. There is no start date, no target, no completion criterion and no expiry. Only version strings.</p><p>We have found that customers may be running two or more versions at once. An evil example: three versions spread across staging and production rollout waves. Tracing problems and their causes becomes very slow and painstaking at that point, unless you are willing to unleash AI on your live systems to get ‘a whole picture’. But why not fix the config management instead?</p><h3>Tracing connections between files and rollouts</h3><h3>5. I edit one value. Do I use grep to calculate how many dependent values will change?</h3><p>This is a classic blast radius problem which we have discussed in other blogs. In traditional devops we have Git files for config as code, templates and CI that then feeds into a set of clusters. The “Don’t Repeat Yourself” approach will mean that changes are handled in source and then rendered out through a tree of dependent manifests that each represent a real config.</p><p>What if <em>one edited value </em>produces as many changed values as our templating fans out to? Before you merge a change that may impact some or all of our whole fleet, can we say:</p><ol><li><em>how many clusters does this touch</em></li><li><em>and how many values on each?</em></li></ol><p>You see here the tension between <em>one-to-many generation of config</em>, from one DRY source to many targets, and the traditional GitOps model of <em>one-one reconciliation</em> between the live target system and the desired state ie. explicit or “Write Every Time” config.</p><p>In one example we saw: a one-line change to the global values file re-renders almost every Application in the fleet. The same change in one cluster’s file re-rendered a little over a hundred. The two diffs look the same, but one has a wildly larger blast radius..</p><p>Don’t forget: “density” multiplies the effect. Density is the number of impacted values per target. Blast radius is then: <em>fan-out times density.</em> This equals the number of targets a change reaches, times the number of values it changes on each.</p><p>In our example: The global file held about a hundred and twenty values. A single cluster’s file held more, nearly two hundred. Rewriting the global file could change over ten million live values. Rewriting a cluster’s file could change thirty thousand.</p><p>As we have seen, there is always a workaround. But is it safe to rely on patches, workarounds and so on when we are entering a new era of AI led automation and high scale?</p><p>In our example: The team built a blast-radius control: a hard CI check that caps a pull request at &lt;N cluster files. The effect is to protect the per-cluster files; but these have the smallest reach and are owned by teams. The global file remains ungated without a natural owner, We are still trapped by “teams own paths in Git”, without blast control.</p><h3>6. Do my controls reduce the rate of change, or make all changes safe?</h3><p>In the Git world we set up all sorts of operational controls which help us work around its quirks. These can create a false sense of security.</p><p>List your controls and ask: <em>Which of these makes a change safer, as opposed to rarer?</em></p><ul><li>Files per pull request</li><li>Freeze windows</li><li>Patch cycles per year</li><li>Bot concurrency</li><li>Rollout waves</li><li>Limits on re-rendering</li></ul><p>If every one of them limits how often you change things and none limits what one change does, you are rationing change. This is bad because cost per change still grows with fleet size. And, the change rate is rising because agents now write config. <em>Total work is fleet size * change rate.</em></p><p>I.e: adding one more cluster costs its own work multiplied by every change made after it.</p><p><em>A system whose unit cost and unit risk both grow with scale has a ceiling: it is called your budget :-) And this is yet another reason why people use databases for business operations.</em></p><h3>7. Does it always matter if a live resource has a different value from Git?</h3><p>Last one! This is pretty subtle. GitOps says: make your live state reconcile with your desired state. But a large system could have many such loops reconciling at once. When is it ok to wait for convergence and when should you intervene? Using databases can give us more control.</p><p><em>Drift</em> is the difference between the repo and the running system. That means a repo cannot show drift on its own; you need a live system to compare with.</p><p>Instead, your Git repo can show “what you allow”. In order to connect this up to live operations and management, once again we layer on workaround tools to capture decisions and explain them. For example: protecting a hand-edited production value; pruning resources from the cluster and desired state correctly; and of course letting automation loose: not just auto-scalers, but also AI. Put those rules in a database and link them to change management!</p><p>We have covered some of these topics on earlier blogs for example:</p><ul><li><a href="https://confighub.com/blog/fixing-production-without-break-glass">Fixing production without break glass</a></li><li><a href="https://confighub.com/blog/building-imperative-operational-tools-on-a-declarative-platform-cec24f8efe05">Building imperative operational tools on a declarative platform</a></li><li><a href="https://confighub.com/blog/gitops-in-the-age-of-ai-5-9b4573c92765">GitOps in the age of AI part 5 — HELM</a></li></ul><h3>Does using AI and Machine-first Operations make this worse?</h3><p>I think using AI to automate operations means that workarounds will break and create more complexity. To take a simple example: Git is not able to handle bulk ops at machine speed, and this isn’t fixable without really serious surgery. The world of “Git extensions” and “Post-Git stores” is upon us. ALL machines will speak Git in perpetuity, I suspect, but Git will not be the lingua franca of agentic operations, and “GitOps” will extend to new Git-like stores as expected.</p><p>Looking at the seven questions again: Each of these was survivable when people wrote config at human rates. Agents raise the rate of change, the reach of whoever holds the credentials, and the number of changes in flight at once. Human review can barely keep up. Example: A dependency bot supports unlimited generation, has a mandatory human merge, and a cap of three open pull requests. Unlimited machine writes, executed in groups of three at a time, but throttled if a person must check each one.</p><p><strong>The seven questions again in short</strong></p><ol><li>Am I using filenames as primary keys?</li><li>Am I approving the query, but deploying the result?</li><li>Am I using find-and-replace as my transaction?</li><li>Am I using someone’s memory as my status column?</li><li>Am I using grep as my query engine?</li><li>Am I using rate limits as my constraints?</li><li>Am I using defaults as my drift policy?</li></ol><p>Today, each of these is a database job being done by Git. Each one was a reasonable fix when people wrote config by hand, a few changes a day, and someone on the team remembered how it all fitted together. That is no longer how config gets written.</p><p>An agent needs three things a folder of files cannot give it. It needs to ask “which clusters run this version” and get an answer from a query, not a grep. It needs a missing reference or a missing value to be an error, not a silence. And it needs the reach of a change to be a number it can check before it acts, not a person who will look later.</p><p><strong>What you can do this week</strong></p><p>Turn off silent resolution. Set ignoreMissingValueFiles: false in CI only, render, and read the failures. You will find your dead layers in an afternoon. Then create the missing files or delete the dead lines, one chart at a time.</p><p>Put reach on the pull request. You probably already declare which paths affect which applications, for cache invalidation. Use the same data to count the targets each change touches, and post the number as a comment. Do not gate on it yet. Just stop your riskiest change looking the same as your safest one.</p><p>Keep the loop, swap the store, add the proof. Argo and Flux keep reconciling, and Helm keeps authoring. What changes is where the rendered result is held and what must be true before it moves. Once reach is a number, a quota can become a limit on what a change does, and cost per change follows what changed rather than the size of the fleet.</p><p><strong>Find out where you stand</strong></p><ol><li><strong>See the answers first.</strong> The<a href="https://github.com/confighub/examples/tree/main/gitops/argo/intermediate-git-as-database"> example fleet</a> is a small Argo CD repo with one planted instance of each of the seven, and an answer sheet. Two minutes.</li><li><strong>Ask your own repo.</strong> Paste<a href="https://github.com/confighub/examples/blob/main/prompts/config-repo-seven-questions.md"> the seven-question prompt</a> into Claude Code, Cursor, Codex or any agent with shell access to a clone of your config repo. It writes and runs its own scripts, reports only what they returned, and says NOT OBSERVED where the repo cannot answer. For Argo CD and Helm repos,<a href="https://github.com/confighub/examples/tree/main/prompts/scout"> scout</a> runs the same seven questions as fixed checks in one read-only script, so you can repeat them exactly.</li><li><strong>Talk to us.</strong> If the answers worry you, we would like to go through them with you (<a href="/service/mailto:alexis@confighub.com">email</a>)</li></ol><p>Next time on this theme: a cost model so you can run the numbers yourself.</p><p>Keep Git for source code development, and give your fleet a GitOps-ready config database. Let us help make your software operations simpler, faster and safer — especially for agentic work.</p><p>–alexis</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VInLvJno9OS55k-CE9xl4g.png" /><figcaption>a graph database (neo4j)</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9df165182b25" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Four roads to config hell]]></title>
            <link>https://medium.com/@monadic/four-roads-to-config-hell-c6671a787e4c?source=rss-c1c94bbe89c2------2</link>
            <guid isPermaLink="false">https://medium.com/p/c6671a787e4c</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[finops]]></category>
            <category><![CDATA[gitops]]></category>
            <category><![CDATA[confighub]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[alexis richardson]]></dc:creator>
            <pubDate>Thu, 17 Sep 2026 11:01:01 GMT</pubDate>
            <atom:updated>2026-09-17T11:01:01.745Z</atom:updated>
            <content:encoded><![CDATA[<p>When we started ConfigHub, we talked about getting organisations “out of config hell”. And we soon found people who wanted help.</p><p>You know the symptoms: outages, data loss, business downtime, dev and ops fighting to fix endless YAML and Helm. Then the diagnosis: <em>we have lost control, costs are growing, performance is stuck, we can’t see what is going on, we are fighting misconfiguration and not delivering, do we need AI?</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*bklWzrzFQALwv2pI71PupA.jpeg" /><figcaption>the all-time genius of mr brazeal</figcaption></figure><p>Our goal today is to give you four measurable “pain points” that you can easily recognise. These are causal factors that lead to bad outages, delays and losses. The idea is to help you quantify and prevent these from losing you time, data and money. We provide a free AI prompt to assist diagnosis.</p><h3>Config Drift</h3><p>Our first example is familiar: “configuration drift”. This is when “systems or cloud environments gradually move away from their intended, documented state”. That quote is from the security company <a href="http://wiz.com/">Wiz.com</a> who have a helpful <a href="https://www.wiz.io/academy/cloud-security/configuration-drift">deep dive page on the topic</a>. Wiz go on to say that drift is to blame for “outages, security vulnerabilities, compliance failures, and unexpected costs” and caused by “manual updates, software patches, or environmental factors”.</p><p>There are tools that help you ‘fix’ drift, indeed one of the core principles of GitOps is for an agent to detect configuration drift between a runtime component and its desired state, and then converge or “reconcile” back to the correct state. But sometimes the problem isn’t so simple: you may have many such clusters and targets and what counts as “correct state” may be volatile or opaque.</p><h3>Main config pain points</h3><p>Drift isn’t the only way config can go wrong and bite you. Here is a table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OumnWqoCXrbxST6ejiKSlQ.png" /><figcaption>four major pain points of configuration</figcaption></figure><p>We shall explain these more below. But first note that <em>everyone has these problems.</em> Nobody sets out to build a bad configuration system. Careful people try to make good decisions. But for a bunch of <em>reasons</em>, most organisations can soon find themselves struggling.</p><p>One of the main reasons is that our tools don’t look after config properly. As we have argued in other blog posts, fundamentally configuration is data: and it is data about the state your systems are meant to be in, so if tools throw it away you lose the state, its provenance and history.</p><p><strong><em>Imagine if a bank handled your money by using YAML files and CICD pipelines to execute transactions. Yet this is how we operate IT.</em></strong></p><p><em>A warning: Take care with AI and configuration management. If you have lost transparency and control then AI-led changes could do more harm than good. “Why not just add AI?” can speed you towards further trouble. By the time those Helm errors are in production, it may be too late.</em></p><h3>Config Sprawl</h3><p>Our second pain point is “configuration sprawl”.<a href="https://www.techtarget.com/enterprise-software/feature/Configuration-sprawl-is-hidden-software-debt"> TechTarget</a> describe it as hidden software debt, growing through “fields, workflows, roles and settings that become harder to govern over time”.</p><p>Sprawl is what you get when configuration has spread across enough systems, files and formats that nobody can say with confidence which copy of a value is the one actually in force for a given live issue. Therefore, the cost turns up as lost time rather than as an outage, in the hours (or days!) spent looking for the right value and in the changes made against the wrong copy of it.</p><p>Consolidation is the standard approach to solving this. Maybe you put the shared parts in one chart, you layer environment values over the top, you keep a single source of truth. But as we have argued elsewhere, generating all your config from a template can lead to trouble at scale. Maybe the business adds a region or a team or a new class of cluster. The schema quietly becomes insufficient, then someone adds per-cluster overrides; but nobody looks after them.</p><h3>Blast Radius</h3><p>Third is “blast radius”, a term from safety engineering that infrastructure people borrowed. AWS makes an equivalent argument in their<a href="https://docs.aws.amazon.com/wellarchitected/latest/reducing-scope-of-impact-with-cell-based-architecture/reducing-scope-of-impact-with-cell-based-architecture.html"> Well-Architected guidance on reducing the scope of impact</a> and in their<a href="https://wa.aws.amazon.com/wellarchitected/2020-07-02T19-33-23/wat.question.REL_10.en.html"> fault isolation material</a>. The idea is to keep a failure limited to a fixed number of components rather than letting it spread “unbounded” or “too far”. A lot can go wrong when one small change is made. Just look at Helm charts with production based hooks.</p><p>The industry evolved canaries and approval gates to help. The concept: roll out in ‘waves’. And/or cap the number of files a single pull request may touch eg. “no more than five cluster files at a time”. But the rule does not distinguish between two very different operational cases:</p><ul><li>Change one line in one cluster’s own settings and you change one cluster.</li><li>Change one line in a file that every cluster shares and you change five hundred</li></ul><p>This is “reach”, and it comes from <em>fanout</em>: how many clusters pick that file up. The number exists, but it is not written down anywhere in the repository.</p><blockquote>Example: This can really bite you: one engineer changes some cluster metadata. Another engineer changes a network policy. Neither looks like anything in review. Together they were enough to upgrade monitoring across the whole fleet, and nobody caught it because there was nothing in either pull request to catch.</blockquote><blockquote>What would have caught it is a number. How many clusters does this change? Put that next to the diff.</blockquote><h3>Config Limbo</h3><p>Our fourth factor doesn’t have a ‘settled’ industry name, so we call it “configuration limbo”. A related phenomenon is ‘forgotten feature flags’ (<a href="https://martinfowler.com/articles/feature-toggles.html">eg here</a>).</p><p>Limbo is: changes that were neither finished nor abandoned. Maybe a rollout that reached most of the fleet and stopped. Maybe a version bump went to staging, and never left. Most teams we talk to recognise this straight away. But they are hard to track, trace and prevent.</p><blockquote>An example pain point:</blockquote><blockquote>Most charts depend on shared library charts, usually pinned to a range rather than a single version, with no lock file committed. That is often deliberate, because pinning exactly means a pull request per chart every time a shared function is fixed.</blockquote><blockquote>The pain shows up during a staged rollout. If a new library version is published halfway through, the early waves rendered against the old library version and the later waves against the new library version. Every cluster reports the same chart version. The manifests differ. Git does not record the wave split. <strong>OUCH</strong>.</blockquote><p>Some tooling exists. Promotion systems, rollout controllers and release dashboards aim to sequence and track a change on its way through a ‘pipeline’. But what if a rollout that is halfway done looks like one that was deliberately stopped two quarters ago? Go and find the owners.</p><h3>Measuring your Config</h3><p>Point your favourite AI tool at a clone of your own repositories and ask it four questions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rCLvFLzHg7TL39ceLGJmJg.png" /><figcaption>using your AI tools to investigate config pain</figcaption></figure><p>Your AI will also want to know, if at all possible: what a loaded engineer hour costs you, how many patch cycles you run today against how many your security policy asks for, how long one target takes in a cycle, and how many of last year’s incidents started with a configuration change.</p><blockquote>TRY IT NOW: There is a longer version, ten questions with the scripts an agent should write to answer them, in our public open source repo: <a href="https://github.com/confighub/examples/tree/main/prompts">examples/prompts</a></blockquote><h3>Measuring your Cost</h3><p>Imagine an example. Take a fleet of five hundred clusters and a hundred and forty charts, twenty platform engineers at $110 an hour, monthly patching that you manage a quarter of the time, and six incidents last year that began with a configuration change.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KsSXbcshiJXU3fqdgfQ5Yg.png" /><figcaption>a basic cost estimate for bad config in a fleet</figcaption></figure><p>Something over a million a year. Substitute your own numbers — and feel free to <a href="/service/mailto:alexis@confighub.com">email me</a> if you need help on how to structure estimates here and make use of the right tools.</p><p>Note how the two largest line items are not the cost of anything going wrong. They occur when nobody can work out what a change will touch. We put ‘top down’ numbers on that in a prior blog post:<a href="https://confighub.com/blog/the-hidden-cost-of-config-errors-95ee1aeeb130"> The Hidden Cost of Configuration Errors</a>. More on cost analysis in future blogs.</p><h3>Conclusion</h3><p>As we said above this is about transparency and control.</p><p>Each pain point implies a question:</p><ul><li>What will this change touch</li><li>What differs here, and why</li><li>Where did this rollout get to</li><li>What changed out there, and who decided it</li></ul><p>Each question is about a relationship, between:</p><ul><li>A source config definition and the targets it reached</li><li>A live value that differs, and the origin where that difference was agreed</li><li>A decision and the edits that carried it out</li><li>What was declared and what is actually running</li></ul><p>You wouldn’t run a bank as a set of CI jobs on YAML and Helm. You’d have customer accounts in a database. When money moves between accounts, that’s a transaction on the database. That’s basically a ledger for customers, money and relationships between accounts. That’s what we need to get out of config hell — a single, central, transparent point of control. You keep your GitOps loop: Argo and Flux keep reconciling, developers keep authoring, and agents keep raising the rate of change; and all this needs policy and approvals.</p><h3>Where to start</h3><p>If you want to review your own private repos to see if you may be on the “road to config hell” then you can. Just start your favourite AI tool and give it a free prompt from our public open source repo: <a href="https://github.com/confighub/examples/tree/main/prompts">examples/prompts</a></p><p><a href="https://confighub.com/sign-up">Sign Up for ConfigHub</a> and try the <a href="https://docs.confighub.com/get-started/setup/">Tutorial</a>.</p><p>Feel free to <a href="/service/mailto:hello@confighub.com">contact us directly</a>.</p><p>— alexis</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/447/1*5tOASWKpLkPtLXCYujjQHQ.jpeg" /><figcaption>couldn’t leave this out :-)</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c6671a787e4c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Intelligence is the fourth commodity]]></title>
            <link>https://medium.com/@monadic/intelligence-is-the-fourth-commodity-d446154ff285?source=rss-c1c94bbe89c2------2</link>
            <guid isPermaLink="false">https://medium.com/p/d446154ff285</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[confighub]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[gitops]]></category>
            <dc:creator><![CDATA[alexis richardson]]></dc:creator>
            <pubDate>Thu, 03 Sep 2026 11:11:00 GMT</pubDate>
            <atom:updated>2026-09-03T11:11:00.858Z</atom:updated>
            <content:encoded><![CDATA[<p>This blog post is about enterprises consuming (artificial) intelligence as a commodity. I have a load of cool product stuff in the pipeline, but wanted to step up and do a “macro view” piece. This question matters:</p><blockquote>What happens when everyone needs to spend bet-the-company money on AI?</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/435/1*OEdJ8Y05YUyc4UW_2TZC7w.jpeg" /></figure><p>Twenty years ago <a href="https://press.aboutamazon.com/2006/3/amazon-web-services-launches">AWS launched the S3 cloud storage service</a> and thus began the industry race to commoditise on-demand computing. Today, every company’s standard infrastructure line items are indeed commodities: compute, storage and network.</p><p>Three commodities: all bought from a handful of suppliers. Feature parity is expected. Instead CxOs ask: How much are we spending? What are we locked into? What happens when they put the price up? Can anyone constrain our use? For vendors it’s: how fast can we fund and build?</p><h3>AI Inference is must-have infrastructure</h3><p><em>There is a fourth line item now, and it is </em><a href="https://cloud.google.com/discover/what-is-ai-inference?hl=en"><em>AI inference</em></a>. In some companies it is already bigger than storage. The same CxOs are asking the same questions: How much? How fast? How soon? That is what makes it a commodity, rather than any claim that models are becoming alike. A commodity is something you purchase by the unit, from somebody, and have to keep buying.</p><p>So everything that happened to the first three commodities starts happening to it. When AI was a feature you bought to solve a problem using one vendor was a preference. Now, when it is a commodity you run on,<em> using one vendor is a dependency, and enterprises do not accept single dependencies for anything they treat as infrastructure</em>.</p><p>The exposures:</p><ul><li>Prices can and do move — see eg. memory in the last year</li><li>Feature lock-in and deprecation, because your supplier can retire the model you depend on with a few weeks notice, and the replacement does not behave exactly the same way.</li><li>Capacity: every LOB wants the best GPUs at the same time.</li><li>Terms of use change in surprising ways. Looking at you, big tech.</li><li>Jurisdiction, which decides where your data may go and whose law reaches it.</li></ul><p>So the interesting question is not whether large companies will use more than one source of inference. They will, the same way they ended up with more than one of everything else.</p><p>The question is <em>what that actually looks like</em>.</p><h3>Five Ways to Diversify Intelligence</h3><p>How will the C-team place their bets?</p><p><strong>Build your own, on open weights.</strong> The drivers are cost and data. At low volume renting is obviously right. At high volume using foundation models is like using professional services: if it is not adding 4x or more to the top line then it is eroding margin and you’re in trouble. Unless you build your own model from one of <a href="https://onyx.app/open-llm-leaderboard">increasingly many powerful ‘open’ options</a>. This also keeps high value proprietary data inside your company, which can be a compliance red line.</p><p><strong>Regional.</strong> The driver is sovereignty. A national model rather than one belonging to another country, and in a growing number of places that is a rule rather than a preference. The risk: what happens if country A stops exports of technology B to countries C but not D?</p><p><strong>Vertical.</strong> The driver is “fitness”. The big one here is Security: which bleeding edge model will find and fix bugs the fastest? Remember we are in “<a href="https://www.confighub.com/blog/gitops-in-the-age-of-ai-2-48e13026f434">Mythos time</a>” and it is a race for survival. Or look at models trained on case law, or on markets, or fund management, or on insurance claims. The latter will be bought by LOBs and not by central IT or platform.</p><p><strong>Small and large.</strong> The driver is unit economics. Most work does not need the biggest model. Routing the cheap work to something small is where the bill actually falls. Smaller models may also mean less hardware, and less corporate ‘ceremony’ in general. Good for: customer-facing.</p><p><strong>More than one supplier.</strong> The oldest reason of all. Service continuity and negotiation flexibility. Nobody argues with it once the line item is big enough to notice.</p><p>Each of those is sensible on its own and so, frequently, each one of them is arranged separately and not coordinated with the others. In a global organization, then, enterprise IT does not end up choosing a strategy. It ends up running four or five arrangements at once.</p><h3>How and where do I run the AI workloads?</h3><p>Once you have a few diversified options, what’s next? Let’s do some work. <em>But where?</em></p><p>The hipster answer is that agent workloads need something new. Fresh substrate, purpose built, unencumbered by the last decade. Greenfield GPU stacks, SaaS dev/test, VM private cloud… And for some companies that could be the right answer: eg guerilla teams moving super fast.</p><p>For most enterprises though: They spent the last fifteen years installing Kubernetes and virtual machines and cloud storage before that. It is paid for, it is staffed, it has contracts behind it and people who know how it behaves at three in the morning. They want AI to be the same</p><p><em>Which means AI lands on the Kubernetes you already have.</em></p><p><strong>AI in the software lifecycle.</strong> Making your existing applications and your existing operational tooling AI-enabled. This is the largest by count and the least discussed, because it looks like ordinary work: <em>get rid of CVEs, speed up continuous delivery, automate fault resolution.</em></p><p><strong>Agents at scale, single tenant.</strong> Deploy something agentic on the clusters you already run and you have thousands of agents in your own cloud, without worrying about where the compute is, or having to sign new contracts for it. It is worth noting that this is small compute rather than GPU, which is why it can go on what you have.</p><p><strong>New GPU stacks.</strong> Let’s use Neocloud and Nvidia patterns for standard inference solutions. Guess what, a lot of these use Helm, Kubernetes, GitOps and can be part of the enterprise platform engineering systems — now rebranded “<a href="https://blogs.vmware.com/cloud-foundation/2026/06/16/platform-engineering-2-0-the-evolution-the-ai-era-demands/">Agentic Platform 2.0</a>” or something :-)</p><p>All of these are variant use cases for Kubernetes. That means we can package and manage them as configurations. Enter ConfigHub: a central control plane and storage system of record for configuration and any operational or agentic facts that are represented as configuration data.</p><h3>Let us count the configurations</h3><p>How many choices did you make?</p><ul><li>AI Inference models</li><li>Suppliers and OSS</li><li>Hardware resources: GPUs, network and data</li><li>Use cases</li><li>Data access</li><li>Compliance</li><li>… more …</li></ul><p>Every combination needs to be told which model, which version, in which region, through which endpoint, with what fallback, for which workload, in which environment, under which contract, at which price, with which data allowed to reach it. And the configuration surface is growing.</p><p>Then you have management and operations, such as:</p><ol><li>Which model version is production calling right now. Not which one is in the code. Which one is being called, this afternoon, and in which user team or company region.</li><li>Who approved the switch to it, and when, and what happened to quality when it moved.</li><li>Can you roll one region back and leave the others alone.</li><li>And when a supplier gives ninety days notice on a model you depend on, how long does it take to find everything affected. Like a CVE patch day but worse…</li></ol><h3>What to do about it</h3><p>Treat it as configuration, because that is what it is, and put it somewhere you can query.</p><p>That is what we built ConfigHub for. Your configuration lives as data with a history, so which model is being called where is a question with an answer. Changing it is a reviewed change, so somebody approved this exact version and you can see who. Environments keep what they need to keep, so a regional or vertical difference is a declared exception rather than a fork. And you can roll one back without touching the others.</p><p>None of that is completely specific to AI. It is what a system of record does. It just happens that the fourth commodity is arriving faster than the first three did, on infrastructure that is already full, and it is going to be configured by more people, in more places, than any of them were.</p><p>Next: <a href="/service/mailto: alexis@confighub.com">email me</a>.</p><p>Or: <a href="https://docs.confighub.com/get-started/tutorial/">take the tutorial!</a> It only needs fifteen minutes.</p><p>Or: <a href="https://confighub.com/blog">read more blogs</a>!</p><p>– alexis</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9IXFN2Ij6H5JX_pRm8ncNA.jpeg" /><figcaption>so long!</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d446154ff285" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simpler, Faster and Safer Platform Engineering with ConfigHub and Kubara]]></title>
            <link>https://medium.com/@monadic/simpler-faster-and-safer-platform-engineering-with-confighub-and-kubara-9db869bd9816?source=rss-c1c94bbe89c2------2</link>
            <guid isPermaLink="false">https://medium.com/p/9db869bd9816</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[confighub]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[gitops]]></category>
            <dc:creator><![CDATA[alexis richardson]]></dc:creator>
            <pubDate>Tue, 18 Aug 2026 18:01:01 GMT</pubDate>
            <atom:updated>2026-08-18T18:01:01.518Z</atom:updated>
            <content:encoded><![CDATA[<p>Every team running Kubernetes ends up building the same thing underneath their apps: ingress, certificates, secrets, monitoring, a dashboard. App teams do not want to build that stuff. They want the Heroku dream: “just run my app for me”.</p><p>Enterprises call this pattern the<em> internal developer platform or IDP</em>. This is usually a privately hosted, self service ‘vending machine’ dashboard for: clusters, an addon manager, identity, wiring and data access, setting policies, and developer tooling with CI/CD, app deployment and more.</p><p>The common parts are often exposed through an IDP <em>catalog</em> of parametrised and predetermined configuration patterns. Many catalogs are based on Helm charts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*LWyWRs35UgpG3LvjI5EK0g.jpeg" /><figcaption>build me the platform I need</figcaption></figure><p><em>NOTE: if you reading this and have your own platform, I think you might like to add ConfigHub to track and manage configurations. With that in place you can easily add more operational features (eg using AI). This blog should show how.</em></p><h3>Meet kubara — the platform foundation</h3><p>If you have not met <a href="https://kubara.io/">kubara</a>, it is a framework for Kubernetes that provides addon management, wiring deployment and some Day-2 Operations support. Essentially a mini-IDP that focuses on the <em>catalog-to-addons story</em>.</p><p>As a user and platform owner, you describe your clusters and the capabilities you want. It selects components from a <a href="https://docs.kubara.io/latest-stable/6_components/components_overview/">curated catalog</a>, pins their versions, works out the order they need to go in, and generates the platform files. Artem Lajko, a founding member of kubara, and part of the team building it, has written both<a href="https://itnext.io/what-is-kubara-a-gitops-first-approach-to-reusable-kubernetes-platforms-3f91a060b39c"> an introduction</a> and<a href="https://itnext.io/build-a-local-kubernetes-platform-with-kubara-kind-and-argo-cd-b83aaa36b65b"> a walkthrough on a local kind cluster</a>. If kubara is new to you, read those first.</p><p>Kubara uses the word <em>catalog</em> as follows:</p><ul><li>a kubara catalog packages reusable platform architecture</li><li>including service definitions, Helm charts, Terraform modules, Kustomize files, CRDs, scripts, and templates</li><li>from which concrete platform state can be generated</li><li>as config manifests in standard formats (OCI, yaml)</li></ul><p>In general I find kubara to be a kind of catalog to addons factory; a model that is 100% worth understanding if you build platforms. Especially when using Helm charts and custom values. <a href="https://itnext.io/stop-rebuilding-your-kubernetes-platform-how-kubara-catalogs-make-architecture-reusable-e13a9ba97e3a">A good introduction blog is here</a>.</p><h3>We Added ConfigHub to kubara</h3><p>Now let’s combine ConfigHub and kubara. ConfigHub sits downstream of the kubara CLI and stores platform-ready kubara configs, which we deploy as OCI images using Argo. Simple. Everything is on GitHub at<a href="https://github.com/confighub/kubara-confighub"> </a><a href="http://github.com/confighub/kubara-confighub">github.com/confighub/kubara-confighub</a> with more back story too.</p><p>We have built two things for you.</p><ol><li>Tools to create your own kubara with ConfigHub</li><li>A complete reference implementation that you can run, with several services and example apps</li></ol><p>As mentioned, the integration flow is very simple. Kubara has a CLI which, when given a catalog input, will generate all your configs and then commit them to Git. After this Argo CD reconciles with the live systems. In our integration ConfigHub imports configs from Git, creates multiple custom, verified, live-ready variants, and outputs them as OCI for Argo to deploy.</p><p>Using kubara’s CLI:</p><ol><li>Select and configure services from the kubara project’s public general catalog</li><li>Add organization-specific platform definitions through your own external catalog and documented values overrides.</li><li>The generated platform can provide shared capabilities that applications consume.</li></ol><p>In the normal kubara workflow you will declare clusters and desired platform services in a <em>config.yaml</em> file and pin the catalog version(s), and then let kubara render the effective catalog into reusable platform-components/ and cluster-specific platform-configs/. We’ll show examples of application deployment below.</p><p><em>NOTE: In the standard kubara ‘standalone’ workflow, the kubara CLI generates configs and commits them to Git; then Argo CD reconciles with the live systems. But in our ConfigHub integration, we do it a little differently. This time the kubara CLI still generates configs into Git, and then ConfigHub imports these configs from Git, and stores them in our configuration database. ConfigHub then releases updates via OCI and ArgoCD.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*YNshGapHvMT6Ntg72filBQ.png" /><figcaption>overview of kubara and confighub</figcaption></figure><h3>Why add ConfigHub?</h3><p>Today ConfigHub stores everything that kubara generated as <em>configurations you can read, check, approve and promote</em>, and it <em>owns the release that each cluster pulls.</em></p><p>This enables operational controls: Which exact version of each component is on which cluster right now? Who approved this before it reached production? Can I roll one production cluster back without touching its twin? Is anything left running that nobody owns?</p><p>In the future we would like to continue adding operational capabilities that simplify human and agentic applications that require configuration and context. Promotions, staging, rollouts and fleets are a current focus — in combination with AI and compliance. Interested? <a href="/service/mailto:hello@confighub.com">Get in touch</a>.</p><p><em>We chose kubara because </em><a href="https://confighub.com/blog/simpler-faster-and-safer-fleets-with-confighub-and-sveltos-d74b9808ccee"><em>like Sveltos</em></a><em> it is an elegant </em><a href="https://kubara.io/#what-is-kubara"><em>GitOps-based design</em></a><em> that aligns with our vision. You should definitely consider it if you have Helm charts and want them to behave :-) Or if you have built your own IDP try using these patterns with ConfigHub.</em></p><h3>Show me the reference implementation</h3><p>This is all set up for you. Four clusters.</p><ul><li>One ‘kubara hub’ = our dev cluster</li><li>Three ‘kubara spokes’ = 1x staging cluster and two production clusters</li></ul><p>In this reference implementation, each cluster runs an Argo CD and a small ‘argobot’ that tells Argo to sync as soon as a release is published. cub cluster up installs Argo CD on each cluster, and ConfigHub maintains its configuration from then on.</p><p>A note for Flux users: ConfigHub publishes reviewed releases as OCI images, and we have<a href="https://confighub.github.io/helm-expt/site/d/data/serverless-oci-gitops-proof/summary.html"> a verified test run</a> where Flux reconciled one of those digests and the workload came up. So your own Flux should take this too.</p><p>As well as Argo, our kubara reference platform has some platform services:</p><ul><li>cert-manager for certificates</li><li>Traefik for ingress</li><li>External Secrets</li><li>Metrics Server</li><li>a Prometheus and Grafana stack for monitoring</li><li>a Homer landing page</li></ul><p>The dev hub runs all of these. The spokes run cert-manager and Traefik. Sixteen component instances across the fleet.</p><p>We also run two apps:</p><ol><li>hx-web is a small nginx with an Ingress and a Certificate</li><li>Cubbychat is a three-tier <a href="https://docs.confighub.com/get-started/tutorial/">tutorial</a> app with Postgres, a backend and a frontend</li></ol><p>Both are reachable over HTTPS through the platform’s ingress, with a certificate the platform issued, on all four clusters. As noted this is all set up for you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*mmZ5qNmrv2XpLv1y2JixDg.png" /><figcaption>platform services + core + apps in confighub</figcaption></figure><p>In ConfigHub that comes to 53 Spaces, 63 managed Units, and 25 links recording what each app needs from the platform. On the clusters there are 35 Argo Applications and no ApplicationSets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*d8QfELtRSzXnijNZEWTJsA.png" /><figcaption>a table of variant configs</figcaption></figure><h3>HOW we integrated this</h3><p>We selected seven platform services exposed by <a href="https://docs.kubara.io/latest-stable/6_components/components_overview/">kubara’s versioned general catalog</a> and declared the required clusters and service configuration in <em>config.yaml</em>. Kubara validated that intent against the effective catalog and generated reusable Helm-based platform components plus the cluster-specific configuration. We reviewed the generated result and the dependencies between the selected services before handing it to ConfigHub.</p><p>Two of them are about metrics, because metrics comes in two kinds: Metrics Server for the live numbers that autoscaling reads, and the Prometheus stack for dashboards and history.</p><p><em>We made an important implementation change. </em>We dropped kubara’s own Argo CD deployment because ConfigHub already runs one on every cluster. That means the platform contract in config.yaml still declares what the platform should contain, Kubara validates and renders it with the effective catalog, and ConfigHub delivers the adapted output using the generated bill of materials used by this integration. So the implementation is an “adapted kubara platform”. We checked correctness by verifying parity of the two versions: adapted and unadapted.</p><p>This worked well — we only ran into a couple of minor issues, like how to wire up secrets management. kubara and ConfigHub are both young projects so this was not surprising. But it was very pleasing that all solutions came easily.</p><h3>Trying out Two Applications</h3><p>We have two apps for demos. <strong>Cubbychat</strong>, the three-tier chat <a href="https://docs.confighub.com/get-started/tutorial/">application from the ConfigHub Tutorial</a>, is the realistic one: a Postgres database, a backend and a frontend. <strong>hx-web</strong> is a plain nginx, the smallest thing that can prove an end-to-end web flow. Both use the shared platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*pr8062OB-PBpp6aBhLXDbg.png" /><figcaption>hello cubby chat ai bot</figcaption></figure><p>The pattern is identical for each application. You bring your app, add an ingress to put it on the network and a certificate for TLS, and the platform does the rest. cert-manager and Traefik are shared platform capabilities, generated from the kubara catalog: the app team does not install them, maintain them, or think about them much. Like in an IDP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*zE2iM4uG9OKMwmA2E5_PkA.png" /><figcaption>cubbychat has dev, staging and 2x prod config variants — all derived from base</figcaption></figure><blockquote><strong>Above:</strong> Cubbychat in ConfigHub: one base, four deployments, each Healthy and Synced with its own release. From here you click into diffs, version history, approvals and the rest. And when you want the reconciler’s view, ConfigHub units are wired to their Argo applications, so you flip straight to the Argo GUI.</blockquote><p>Then we added a second application: hx-web. This has an end to end web flow, and notice that cert-manager and Traefik are <em>shared platform capabilities</em> generated from the kubara catalog. Three tiers, a route, a cert and serving over HTTPS on all four clusters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*GGJZj2pl0-2HJFx4oBnNPw.png" /><figcaption>hx-web has additional configs</figcaption></figure><blockquote><strong>Above:</strong> The hx-web app; and its platform binding of the ingress and certificate. Note: in Cubbychat the binding and the app are tightly coupled. But in some use cases you want infra bindings kept as separate configs and version history. The app is on releases 8 and 9; the binding on 2 and 3. Workload changes and network changes may move at different speeds.</blockquote><p>Then we changed hx-web once, from two replicas to three, and promoted it; then we rolled one production cluster back. This is one place where ConfigHub earns its keep: <a href="https://confighub.com/blog/simpler-faster-and-safer-changes-with-confighub-ba1bc7328aef">safe, fast changes</a>. Staging kept its own sandbox setting straight through the promotion. The rollback restored one production cluster to an exact earlier revision while its twin stayed on the new release.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*U4VwfZIGCVJLSySwpAMHVg.png" /><figcaption>rollback can be controlled per-config and per-cluster</figcaption></figure><p>To see the whole estate at once, we generated a report for this post (Claude did the legwork): the platform plus the apps as a grid, components down the side, clusters across the top. Every cell keeps four things apart: what should be there, what ConfigHub released, what Argo says it synced, and what Kubernetes reports as ready. Those are four different facts, and most outages live in the gaps between them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*dwuMmVQh4-du_sOcbRFVjg.png" /><figcaption>Claude makes it easy to draw additional ad hoc fleet reports</figcaption></figure><h3>Approvals — a ConfigHub feature</h3><p>We might like AI changes to be approved by a manager, and have the record bound to the revision that was there when they approved it. ConfigHub supports this as a first class option. Approvals can act as release gates for example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*kuRDQZtd2mMl5M2KcFGsgA.png" /><figcaption>blocking and approving example — here shown via confighub CLI ‘cub’ and terminal</figcaption></figure><p>Often approvals are part of a larger set of “jobs to be done” around an application component. Maybe you also have to check that applications have sync’d correctly too. Maybe agents are proposing changes at the same time, or need to check settings from past changes. All of these are potentially important machine-verifiable context for human, manager and agentic decisions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*thtjd1BsQGnfGajSYWxXtg.png" /><figcaption>approvals and other jobs are aggregated in the confighub GUI</figcaption></figure><h3>Try the platform, try the apps</h3><p>I would encourage you to deploy the platform and run apps. Maybe vibe code an app or two and deploy them onto the mini-IDP. Point your agent at the ConfigHub docs as well as your own code, and let it propose the changes. Or if you like you can ‘rehearse ‘the whole thing on your laptop with npm, and no cluster, no account. Just use this little verifier tool:</p><p><strong>npm run kubara-adoption:self-test</strong></p><p>The tool reads the kubara config, compiles the generated platform into packages, imports them, and hands off the application releases, against local stand-ins throughout. Success means the tooling you would later run for real should still work, end to end, including refusing bad input.</p><p>We also have a <em>live reference platform running inside ConfigHub. </em>This is four clusters and both apps, and you are welcome to come and look around it: the components and their versions, the promotion history, the approval rule on production, and the grid above with real values in it.</p><p><a href="https://confighub.com/#:~:text=Contact%20Us-,Sign%20Up,-Take%20Back%20Control">Sign up</a>, and/or <a href="/service/mailto:alexis@confighub.com">get in touch with me</a>, and we will show you the live one before you build your own.</p><p>Finally when you want to build your own kubara platform properly and using ConfigHub, <a href="https://github.com/confighub/kubara-confighub">follow the instructions in the repo</a>.</p><p><em>Build the platform once. Keep the output as the record. Then run your apps, and your agents, on top of the record instead of on top of a rebuild.</em></p><p>– alexis</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*FCANXyL-bHcQAM9a8XfLbA.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9db869bd9816" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simpler, Faster and Safer Fleets with ConfigHub and Sveltos]]></title>
            <link>https://medium.com/@monadic/simpler-faster-and-safer-fleets-with-confighub-and-sveltos-d74b9808ccee?source=rss-c1c94bbe89c2------2</link>
            <guid isPermaLink="false">https://medium.com/p/d74b9808ccee</guid>
            <category><![CDATA[confighub]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[gitops]]></category>
            <dc:creator><![CDATA[alexis richardson]]></dc:creator>
            <pubDate>Thu, 13 Aug 2026 20:01:01 GMT</pubDate>
            <atom:updated>2026-08-15T11:56:44.874Z</atom:updated>
            <content:encoded><![CDATA[<p>Every team that runs more than a handful of Kubernetes clusters ends up inventing the same thing. You do not want a change landing everywhere at once, so it goes to one cluster first, then a few more, then the rest. Canary, then a ring (or ‘wave’), then the fleet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/514/1*K_WYbF77lyN9yYM5hM0wOg.jpeg" /></figure><p>The odd part is how much machinery that simple idea usually needs. Rollout groups, stage definitions, a pipeline with its own syntax, somewhere to write down which clusters are in which wave (or ‘ring’), and something to keep that list updated as clusters come and go.</p><p>We have spent a few weeks on this with <a href="https://projectsveltos.io/"><strong>Sveltos</strong></a>, and the result is very cool.</p><p>ConfigHub gives your Sveltos fleet an editable backend, enabling simpler, faster and safer changes. Change your cluster groups? Update the database. Filter and patch? Run a query. It is all declarative and GitOps friendly.</p><h3>What is Sveltos?</h3><p>If you have not come across it, <a href="https://projectsveltos.io/">Sveltos</a> is an open source Kubernetes add-on controller. It runs on one management cluster and looks after the others. <a href="https://github.com/confighub/sveltos-confighub">Our integration is on GitHub here</a>.</p><p>You register all your clusters in Sveltos and give them labels, the same way you label anything else in Kubernetes:</p><ul><li>environment=staging</li><li>region=eu</li><li>rollout=pilot</li></ul><p>Then you write a <em>ClusterProfile</em>, which says which labels to match and what the matching clusters should run.</p><p>Sveltos works out which clusters the selector covers, installs the thing on each one, and keeps watching. If someone changes it on a cluster, Sveltos puts it back. (And yes, Claude can help with all this)</p><p>In its native design <em>you never name the clusters</em>. You describe them, and any cluster matching the description picks up everything that matches. That is a good design, it is why Sveltos scales to hundreds of clusters, and it is the first thing anyone likes about it.</p><p>Updates are straightforward. Sveltos leverages OCI and can use Flux or Argo too. In today’s post (and <a href="https://github.com/confighub/sveltos-confighub">GitHub example</a>) we’ve gone for a minimal set where ConfigHub publishes changes to an OCI gateway. Let me know if you are interested in other configurations!</p><h3>Two models → One</h3><p>Before the examples, it is worth saying how the two tools think, because they overlap in exactly one place and that creates a design problem.</p><p>Sveltos thinks in <em>sets:</em></p><ul><li>A management cluster</li><li>Workload clusters registered with labels</li><li>And a profile carrying a label query</li><li>The profile matches a set of clusters, installs the add-on on all of them, and reconciles them forever</li><li>Scope is decided at delivery time, by the controller</li></ul><p>ConfigHub thinks in variants which are database <em>records</em>:</p><ul><li>Configurations are <em>records with revision history and approval gates</em></li><li>These records have a data model: <em>variants, in a relationship graph</em></li><li>Variants are derived (revised) from a base variant</li><li>Revisions are declared (declarative) departures from the parent variant</li><li>Scope is a property of the record, decided at review time and stored</li></ul><p>Therefore:</p><blockquote>Both models can answer “which clusters get this change”. Without integration, you could end up expressing the same change set decision twice, in two places, with only one revision able to ‘answer questions’ afterwards.</blockquote><p><strong>The way we join them has three parts</strong></p><p><em>ConfigHub owns what configuration is and whether a change may be approved.</em> The content, the review, and the approval are bound to one exact revision.</p><p><em>OCI is the boundary from ConfigHub to Sveltos.</em> An approved revision is published as an <em>image</em> on ConfigHub’s gateway. The handoff is a digest: immutable, addressable, verifiable from either side. Neither tool has to understand the other’s internals.</p><p><em>Sveltos owns delivery and live runtime convergence.</em> It fetches that image, applies the reviewed profile to the cluster it names, and keeps it there.</p><blockquote>The integration may be summarised as: <strong>one ConfigHub record per Sveltos cluster</strong>. Each record carries a selector that matches exactly one cluster, so the selector stops being a Sveltos fan-out mechanism and becomes a Sveltos address.</blockquote><p>What does this give up? Matching many clusters with one profile is a Sveltos feature that this design does not need, because all cluster configs are explicit in ConfigHub. Sveltos takes care of fleet orchestration: it keeps every addressed cluster converged and repairs drift without anyone asking, continuously. Governance and authorisation derives from config changes.</p><p><em>Analogy: we deliver change across the fleet.</em></p><p><em>Think of a record office and a delivery service. The office issues the authorisation and says who it is for. The delivery service takes the sealed package to the address on it and makes sure it stays there. The only way it goes wrong is if the delivery service starts deciding who the recipients are.</em></p><h3>One variant per cluster</h3><p>You may wonder if this is efficient. The obvious objection is arithmetic. Fifty clusters, fifty records? Yes, and that is the point: fifty independently approvable, independently rollbackable, independently answerable records. Remember we want to be explicit about all changes because of AI.</p><p>What makes it cheap is that they are not fifty copies.<em> They are variants.</em></p><ul><li>One base variant holds what <em>every cluster shares</em>.</li><li>Each live deployed cluster corresponds to a deployment variant of that base and carries local changes: its name, the selector line that addresses it, and whatever that cluster genuinely does differently.</li></ul><p>In our fleet example today, the pilot cluster’s config variant departs from the base variant in three fields, and one of them is the address. This is the whole declaration, from the committed example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ECkU9F4YO_C8v6XPGUejbg.png" /></figure><h4>Promotion in a Fleet</h4><p>Promotion is then the interesting part. You make a change once, on the base, and it flows outward to each cluster’s record, while each record keeps what makes it itself. <em>Nothing is ‘clobbered’</em> or quietly overwritten by a careless upstream change in the platform, such as an expedited patch.</p><p>And a roll out “wave” is not fifty separate manual updates. It is one <em>bulk operation over a set</em>: select the records for this wave with a label query, upgrade that set in one command, approve that set in one command.</p><p>ConfigHub applies the bulk update query and records an approval per record, each bound to the exact revision that record held at the time. One operation, N recorded approvals, the mapping intact.</p><p><em>Bulk action and per-cluster truth are not in tension.</em></p><blockquote>For fifty genuinely identical clusters where none of that matters to you, a delivery-time label query is the right tool and you should use Sveltos exactly as it is designed. The moment one cluster needs to be held back, or rolled back, or someone asks what production A specifically is running today, the mapping has to live somewhere you can query.</blockquote><h3>How they fit together</h3><p>Let’s go to ConfigHub and look at a running Sveltos fleet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*It_QeO3lrWZZV6IJhXN14g.png" /><figcaption>how the configs are related. note that ‘base’ is not deployed</figcaption></figure><p>This fleet snapshot is from our initial set up in ConfigHub, showing the base variant on the left, and one deployed variant per cluster; then on the right, four workload clusters and the deployed management variant below. Each is on its second revision after one reviewed change to the base.</p><p>What about updating to live? This happens when we hit “release” in ConfigHub, and OCI images are published, like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j0pmlAU7NlkG1qsR-8nQEQ.png" /><figcaption>OCI based updates</figcaption></figure><p>Here is the flow:</p><ol><li>Config comes from ConfigHub, where each cluster’s config variant is stored, checked, and held at an approval gate.</li><li>A named person approves one exact revision, and ConfigHub publishes it as an OCI image on its gateway. Each variant publishes to its own OCI gateway address.</li><li>Sveltos, on the management cluster, fetches that image and sends the reviewed profile to the one cluster it addresses.</li><li>Sveltos then keeps that cluster aligned and repairs drift.</li></ol><p>We have notes:</p><blockquote>NOTE that we did this without an added GitOps controller and without an additonal intermediate registry. Promotion does not change the wiring at all. Publishing the approved release moves the tag, and the addressed cluster follows, so approval alone moves the fleet.</blockquote><p>NOTE: Reading a release from the gateway needs an add-on controller that decompresses gzipped layers, so every run used <em>projectsveltos/addon-controller:v1.13.0-ch</em> while that fix waits for a release. The same profile on stock v1.13.0 fails, because it tries to read the gzip bytes as YAML.</p><p>BOOTSTRAP NOTE: The management cluster’s own deployed config variant records the <em>bootstrap profiles</em> that tell Sveltos where to fetch from. Its first revision cannot arrive through the gateway, because it is what enables gateway fetching, so it is applied <em>once at cluster setup</em>, and ConfigHub governs every revision after that via the same approval gates.</p><h3>The Integration between Sveltos and ConfigHub</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lfL5NcVGjef3EDl7ATy7qQ.png" /><figcaption>records (variants) and update flows</figcaption></figure><p>Our <a href="https://github.com/confighub/sveltos-confighub">GitHub integration</a> features several use cases as “chapters”. They all have: one base variant, one deployed variant per cluster, one approval.</p><ol><li><a href="https://github.com/confighub/sveltos-confighub/blob/main/examples/sveltos/kyverno-fleet/README.md">Kyverno across the fleet</a> installs admission policy through reviewed per-cluster variants, each behind an approval gate.</li><li>Kyverno canary: using the same example as (1), the pilot cluster’s revision is approved and delivered first, and widening the rollout means approving the revision for the next cluster’s variant.</li><li><a href="https://github.com/confighub/sveltos-confighub/blob/main/examples/sveltos/env-rollout/README.md">Environment rollout</a> promotes one reviewed values change from pilot to staging to production, wave by wave.</li><li><a href="https://github.com/confighub/sveltos-confighub/blob/main/examples/sveltos/cve-patch/README.md">CVE patching</a>: one reviewed version bump with digest-bound provenance, closed by a coverage audit that proves no cluster was missed. No vulnerability scanning is claimed.</li><li><a href="https://github.com/confighub/sveltos-confighub/blob/main/examples/sveltos/bulk-ops/README.md">Bulk operations</a>: one reviewed edit promoted to every cluster’s variant in one pass, closed by a zero-drift audit.</li></ol><p>Let’s take a look in stages.</p><h3>Canaries made simple</h3><p>We roll out the Kyverno open source software across the fleet.</p><p>Imagine two staging clusters, and two variants i.e. one per cluster, and both variants of the same base. Underneath is Kyverno 3.8.1, with three replicas of the admission controller.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z0JRKMNV2GFG_3z50wFHGA.png" /><figcaption>YAML for this example</figcaption></figure><p>See below: the history of changes to the second cluster’s variant. We cloned from the base, armed its approval, then departed in exactly three fields including the clusterRefs entry naming its cluster, then approved.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O2CPL7-ns-wkBkMQaJqR6A.png" /><figcaption>kyverno roll out</figcaption></figure><blockquote>NOTE: Admission policy is a good thing to be careful with! Since it decides what a cluster will and will not accept. So both variants sit behind an approval gate. ConfigHub held the pilot cluster’s variant revision, until someone approved that exact revision, published the now approved revision to the gateway, and Sveltos installed Kyverno on the pilot cluster. An unapproved record publishes nothing. So the second cluster got nothing: no release, no namespace, no trace. It is inert.</blockquote><p><strong>How to widen the rollout with one small config change</strong></p><p>Widening the rollout is simple: now we <em>approve the second cluster’s revision</em>. There is no selector to edit and no list to maintain, and there is no moment where one approval quietly covers a set of machines. Each ConfigHub approval names one cluster and one revision.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BKhfRPVh39Ll7ezBkY-n8A.gif" /></figure><p>Above: Read the revision activity from bottom to top: cloned from the base with the approval gate armed, departed in exactly three fields (its name, the line that addresses its cluster, its removal behaviour), then inheriting the reviewed change. The approval binds to this record’s exact revision.</p><p>Post approval it is then published with a different digest. And then Sveltos installs Kyverno on the second cluster while the first remains untouched.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fT3NnJVdCEbGvKg07EDc2w.png" /></figure><p>Above: two deployed variants, shown after both waves. Each ships to a Target (~cluster name). Base variants are undeployed and have no Target.</p><p>Summary:</p><blockquote>We did the whole rollout. Two clusters, two config variants defining them, then ConfigHub revision approvals, and (for release) we published two OCI digests. Then we set the admission controller to one replica on both clusters, and Sveltos put both back to three.</blockquote><h3>Why this is better than a pipeline</h3><p>We think this is a <em>simpler, faster and safer approach</em> than many alternatives.</p><p><strong>The config change approval is recorded upstream</strong>: as we revise the one variant that authorises the change for that one cluster. Not on a deployment somewhere downstream, and not on a ticket describing one. Nothing reached the second cluster until a named person approved the exact revision that let it in. Approving yesterday’s revision authorises nothing about today’s.</p><p><strong>Rollback has the same shape as rollout:</strong> Every cluster’s variant has its own history, so you can put one cluster back on its previous revision and leave its twin where it is. Fleet tools that treat the fleet as one unit cannot say that.</p><p><strong>And nobody had to build a rollout engine:</strong> Look at the diagram again: there is no component in the middle doing waves. Sveltos already delivers to addressed clusters and keeps them reconciled. ConfigHub already stores reviewed records with approval, history and variants. The rollout is what happens when you put those two things next to each other.</p><h3>The same move, three sizes bigger</h3><p>Once every cluster corresponds to a config variant, the other fleet jobs turn out to be the same move at different speeds. <em>Our example has all three.</em></p><h4>Environments</h4><p>One reviewed values change, made once on the base: Kyverno’s background controller went from one replica to two. It flowed outward to four per-cluster records in three waves: pilot, then staging, then both production clusters. The production wave is worth looking at closely. One label query selected exactly the two production records, one command upgraded that set, one command approved it, and the receipt records two approvals, one per record, each bound to that record’s own revision.</p><p>Each record then published its own release, and prod-a and prod-b converged from different digests at their own gateway addresses. The digests differ by construction, not by luck: each record’s bytes differ at least in the line that addresses its cluster, so their releases can never be the same artifact. The per-cluster matrix shows which cluster ran which revision at each checkpoint, and the observed column comes from the run going and looking at the clusters.</p><h4>Patch day</h4><p>One reviewed version bump, Kyverno 3.8.1 to 3.8.2, with the patched chart’s provenance checked against the reviewed digest before anything was stored. It promotes outward through every cluster’s variant, the reviewed values from the previous chapter survive the bump on every cluster, and it closes with a coverage audit naming every cluster and confirming each one runs the patched version.</p><p><em>No vulnerability scanning is claimed anywhere: this is governed delivery of a version bump, not an advisory check.</em></p><h4>Everywhere at once</h4><p>One reviewed edit raised a replica count and promoted to every cluster’s variant in a single pass, closing with a zero-drift audit: no armed gates left, no record changed out of band, and drift injected on every cluster repaired.</p><blockquote>ConfigHub publishes and never connects to your clusters, which is deliberate and is why it holds no credentials for them. So it does not tell you what is running.</blockquote><blockquote>Sveltos tells you, because it reconciles continuously and records what it did per cluster. This can then be reported to ConfigHub for additionial operations.</blockquote><h3>Why ConfigHub</h3><p>I said it was simpler, faster and safer. That comes from making all configuration explicit. Once we have done that, we have a central place to gather observed facts from observability tools and to control operations.</p><p>I highly recommend reading Brian’s post “<a href="https://confighub.com/blog/there-should-be-an-app-for-that-1f4d23286a9e">There should be an app for that</a>”. This tells you how easy it is to add more of your own functionality to the baseline fleet ops we have described. Just use the data in ConfigHub. If you can’t get the right data into ConfigHub, email me.</p><p>Some <em>fleet apps</em> you could build:</p><ul><li>Cumulative failure budget that halts a rollout when failures cross a threshold</li><li>Automated verification step between stages</li><li>Timeout on a stalled wave</li><li>Red button halts and reverses across the whole fleet</li></ul><p>None of those need a new engine underneath. They are queries and writes over data you already hold, which is why they are cheap to build here and expensive to build anywhere else.</p><p>ConfigHub is the <strong>data substrate</strong>:</p><blockquote>We are the part that decides whether any of it is safe, if the rollout definition is a reviewed configuration, and where the approval binds to exact bytes.</blockquote><h3>Run it yourself</h3><p>Every offline check runs with no account, no cluster and no network, and the repository has no npm dependencies:</p><blockquote><strong><em>git clone </em></strong><a href="https://github.com/confighub/sveltos-confighub"><strong><em>https://github.com/confighub/sveltos-confighub</em></strong></a></blockquote><blockquote><strong><em>cd sveltos-confighub</em></strong></blockquote><blockquote><strong><em>npm run verify</em></strong></blockquote><p>Node 22 or newer, python3 with pyyaml, and tar. The command runs the verifier and the self-test for every chapter, so a failure tells you which one.</p><p>The five-cluster fleet <em>rehearsal</em> builds its own clusters and needs no ConfigHub account to check it works:</p><blockquote><strong><em>HELM_EXPT_ALLOW_LIVE_SVELTOS_REHEARSAL=1 npm run</em></strong></blockquote><blockquote><strong><em>sveltos-fleet-rehearsal:run</em></strong></blockquote><p>After the set up phase, the ‘governed’ chapters will need the <strong>cub CLI</strong> and one <strong>ConfigHub authenticated context</strong>, so that each runner checks its preconditions and, if it stops, does so with a named reason.</p><h3>Conclusion</h3><p>We believe the following is true if you are using ConfigHub.</p><p><strong>Fleets do not need a rollout language.</strong></p><p>They need a declarative data substrate and <strong>GitOps type fleet management</strong> of which <strong>Sveltos is a leading example</strong>.</p><p><strong>A label query is used to select cluster variants in bulk at review time.</strong> Acting on the set enables a bulk approval applied per cluster, and individually recorded.</p><blockquote>Widening a rollout means approving the next cluster variant’s revision, and that brings review, approval and history with it, because it is configuration like everything else.</blockquote><p><strong>Try it yourself:</strong> If you run a fleet, the repository is open, the checks run on your laptop, and I would like to hear from you if it falls over, please. You can easily run the example with a few kind clusters or in the cloud.</p><p>Finally, a companion example repository: <a href="https://github.com/confighub/kubara-confighub">kubara-confighub</a>, tells the same story one cluster at a time, where the platform itself is the thing being governed. Blog post on Kubara is coming soon!</p><p>Onwards.</p><p>– alexis</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/481/1*XWnrHsqI48QEV5SpYqY1LA.jpeg" /><figcaption>get it done</figcaption></figure><blockquote>Final note: At time of writing, ConfigHub cannot yet show live per-cluster status from Sveltos — the “what landed” flow back from the fleet — and the recordings used a patched addon controller (v1.13.0-ch) because the gateway serves gzipped layers and the fix is merged upstream but not yet released. When the Sveltos live-status reporter lands and when the fix ships in a Sveltos release, we will refresh this post. Both issues are <a href="https://github.com/confighub/sveltos-confighub/issues">tracked in the open GH repository</a>.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d74b9808ccee" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simpler, Faster and Safer Changes with ConfigHub]]></title>
            <link>https://medium.com/@monadic/simpler-faster-and-safer-changes-with-confighub-ba1bc7328aef?source=rss-c1c94bbe89c2------2</link>
            <guid isPermaLink="false">https://medium.com/p/ba1bc7328aef</guid>
            <category><![CDATA[confighub]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[gitops]]></category>
            <dc:creator><![CDATA[alexis richardson]]></dc:creator>
            <pubDate>Tue, 04 Aug 2026 09:54:26 GMT</pubDate>
            <atom:updated>2026-08-04T10:14:17.209Z</atom:updated>
            <content:encoded><![CDATA[<p>This post is about how ConfigHub is SIMPLER than other approaches.</p><blockquote>Today’s post is very “hands on”.</blockquote><blockquote>This is because we are <a href="https://confighub.com/blog/announcing-confighub-ungated-public-preview">open to everyone now</a>! Read Jesper’s note about<a href="https://confighub.com/blog/evolving-confighub"> what changed and why</a>. And please try our <a href="https://docs.confighub.com/get-started/tutorial/">updated ConfigHub tutorial</a>.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/448/1*MvsPU3n9VBoMzspYYcxb-g.png" /><figcaption>simplicity is all</figcaption></figure><h3>Why ConfigHub is cool</h3><p>Imagine a company running an AI app called Cubby Chat. There is a production app and a dev app for testing. Both are built from a common base, and each is deployed to its own cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5RXtThmwKuv7epaOJggwUg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ExYkCf_qqfkDpe-HfLuYNA.png" /></figure><p>The platform team looks after the base configurations for all the apps. The app teams change things as they go, which is normal, and both of those happen at the same time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mBJ35vaRxHAv9-hbuX-cZw.png" /><figcaption>tutorial commands</figcaption></figure><h3>What just happened</h3><p>The change to the base flowed out to dev and to prod.</p><ul><li><strong>Prod</strong> took all of the change</li><li><strong>Dev</strong> took the new replica count <em>but kept its own chat title</em>, because dev had changed that field itself and ConfigHub had recorded that</li><li><em>Promote</em> merged everything else, and left that one field alone in Dev.</li></ul><blockquote>SIMPLER:</blockquote><blockquote>Gone is all the logic for re-generating config for every single environment or change. This effectively <strong>simplifies your source code</strong>, and <strong>organises your configuration data</strong>. Now the platform team can upgrade the base without ‘clobbering’ the app team’s changes!</blockquote><p><em>We also have a more intuitive data flow through the graph of configurations:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y5lHMmlev5NkecGRcgr95w.png" /><figcaption>changes flow logically</figcaption></figure><h3>The model: Components, variants and releases</h3><p>A <strong><em>component</em></strong> is the configuration for a logical piece of software. Cubby Chat is a component, and so are its backend, frontend and database, depending on how you want to group them.</p><p>A <strong><em>variant</em> </strong>is a stored instance of that component. Some are <em>base variants</em> that exist to be copied from, which is what cubbychat-base is. Others hold the configuration for one live deployment on one cluster, and we call those <em>deployment variants</em>. Here they are cubbychat-dev and cubbychat-prod.</p><p>A <strong><em>release</em></strong> publishes a variant’s configuration as an OCI image, which Argo or Flux then pulls and applies. So the deployment variant is what you edit, and the release is how it gets to the cluster.</p><blockquote>Those two things are separate on purpose. You can change the base, promote it to as many variants as you like, look at what you have, and release your changes when you are ready. Nothing reaches a cluster until you publish.</blockquote><h3>The mission: Simpler, faster and safer changes</h3><p><em>Why are we here? </em>Today’s tools are powerful but too complicated. Changing apps like Cubby Chat should be simple, fast and safe! Without using all of the extra machinery we normally build around apps. Instead, a simpler model using a <a href="https://confighub.com/blog/validating-scoring-and-gating-kubernetes-configuration-after-every-change-a11fbab14110">config store with automatic data validation</a>.</p><p><em>We did not need</em> an exclusion list, or a set of rules, or a do-not-overwrite annotation. We did not need a separate values file for dev. We did not need someone on the team who remembers which environments are special and why. Instead, fields have owners. Someone changed the title in dev, ConfigHub recorded it, and ‘promote’ left it alone.</p><p><em>We also did not have to fight</em> the rest of the stack. Git merges text, so the last writer wins and the loser leaves no trace. Helm rebuilds its output from the chart, so an edit made afterwards is gone at the next upgrade. Kustomize lets you express the difference, but once it is rendered nobody knows who owns a field. Argo and Flux faithfully apply whatever they are handed, including over the top of your change. None of these tools are wrong, and none of them record who touched what, which is why every team ends up maintaining that exclusion list by hand.</p><h3>The tutorial: explained</h3><p>Start with a fresh install and nothing running. The next steps should take 15mins or so.</p><p><strong>cub cluster up — name dev</strong> builds a kind cluster on your laptop, installs Argo CD in it, and wires it to ConfigHub. One command, and you have somewhere to deploy.</p><p><strong>cub variant upload</strong> installs the component. Cubby Chat arrives as a bundle of ordinary Kubernetes YAML, with no templates and nothing to interpolate. It becomes the base, four units, one per file, with the namespace left blank because the base does not yet know where it will run.</p><p><strong>cub variant create dev cubbychat-base — target dev/target — namespace cubbychat </strong>creates the deployment variant for the dev cluster. It copies the base, fills in the namespace, points at the cluster and writes the Argo Application. Nothing is running yet.</p><p><strong>cub release publish cubbychat-dev</strong> publishes that configuration as an OCI image, Argo pulls it, and the app comes up.</p><blockquote><strong>This is ordinary GitOps, in the pull style</strong>. ConfigHub publishes releases and your existing tools fetch them, which means ConfigHub holds no credentials to your infrastructure at all.</blockquote><h3>The big picture: Operations on the config graph</h3><p><em>Our technology aims to simplify and automate operations, using a declarative model where configuration is data. Day to day, that turns into a loop: change the base, promote to dev and prod, and release when you are ready.</em></p><p>The graph underneath is doing important work. When ConfigHub copies a configuration, it records where the copy came from and which version it took, so dev and prod both know they came from the base. ‘Promote’ takes that record and reads it forwards, meaning: <em>catch up with the base.</em> Dev keeps its own title in the example, as explained. Everything else is built from basic verbs: read a configuration, copy it, change it, and keep a record of what happened. A change to a value will affect copies of that value. And a new cluster is one more copy. There is no separate ‘folder’ system for overrides.</p><p>This all lets us <a href="https://confighub.com/blog/the-hidden-cost-of-config-errors-95ee1aeeb130">calculate a change’s blast radius</a>: Because every copy records where it came from, the store can work out what a change will reach before you send it, across every environment that has copied from it (<em>cf.</em> <a href="https://confighub.com/blog/gitops-in-the-age-of-ai-2-48e13026f434">more</a>).</p><p><em>That stuff is more advanced: for when we add agents, and when we scale up. What we have not shown you today is how to order operations. E.g. update order, copy order, promote prod after dev. That is an operational decision about how your team works, not a fact about your configuration, so here you make it by typing the two ‘promotes’ in that order. More on that soon.</em></p><h3>Where this fits: simplifying operations</h3><p>ConfigHub fits most easily into any GitOps flow that has adopted OCI. The operational loop will be shorter and simpler as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qdEBx0kDsPxI_Jd0-VH1Dw.png" /><figcaption>simpler operational loop with confighub</figcaption></figure><p><em>You can check the endnotes(*) if you want to see a delivery image with Git/CI .</em></p><p>In today’s example we did four “jobs” in ConfigHub: deciding config changes, reviewing and promoting, and checking they worked. There are more jobs to do — for future blog posts :-)</p><p>Also today our example was 100% manual. In future I will write about adding more automation, and when an agent is the one proposing the changes, and tie this back to the themes of my GitOps meets AI posts. Eg. release gating is being reworked, so you cannot block a publish on a policy check today. Delete protection on production units is in the tutorial.</p><h3>Try the example with ConfigHub now!</h3><p><a href="https://docs.confighub.com/get-started/setup/">Please take a minute to sign up</a> to try our developer SaaS. All you need is Docker and a laptop.</p><p>Install the open source CLI and <a href="https://docs.confighub.com/get-started/tutorial/">try the tutorial</a>. Try out what we did here today: Change the title on the base, promote it to both, and watch dev keep its own.</p><p>Let’s keep it simple!</p><p>– alexis</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/1*Av_Qjxt4KTqKX0QxGeuVdA.jpeg" /></figure><p><strong>A note on Git and CI</strong></p><p>Source code lives in Git. You may want to <em>generate your configuration from source code</em>, or you may want to <em>have AI write it</em>, or <em>produce it using special config generator tools like Cuelang, Pkl or Helm</em>.</p><p>I talked about the pros and cons of “generative” config, and Generative GitOps, <a href="https://confighub.com/blog/gitops-in-the-age-of-ai-c73d0bd506ba">in the earlier series of posts</a>.</p><p><strong>Whatever approach you use, please note that you want your <em>configuration data</em> to be centralized in one store, if you want a scalable, safe, fast and fully agentic operational GitOps platform.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*70q1Uja0hI9PII75Aw8qwA.png" /><figcaption>source code lives in git and config lives in confighub</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba1bc7328aef" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GitOps in the age of AI — 5]]></title>
            <link>https://medium.com/@monadic/gitops-in-the-age-of-ai-5-9b4573c92765?source=rss-c1c94bbe89c2------2</link>
            <guid isPermaLink="false">https://medium.com/p/9b4573c92765</guid>
            <dc:creator><![CDATA[alexis richardson]]></dc:creator>
            <pubDate>Wed, 29 Jul 2026 19:31:01 GMT</pubDate>
            <atom:updated>2026-07-30T13:07:04.560Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Part 5 of a series.</em></p><p>When you add an app to your phone, you install it from the app store. Then you make it yours: your friends, your contacts, your settings. When the app updates, you do not lose any of that, because your data is kept separately from the app itself. The update replaces the app and leaves your data alone. This is also true for any customisation and configuration data.</p><p>Installing an app with Helm does not work this way. Your customisations and the package are the same thing, so an upgrade can quietly overwrite them, and it can be hard to say where your configuration actually lives at all, and what it is. We want to fix that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*EAKen5qatpAvbGcol8Oy1A.gif" /><figcaption>helm oh helm</figcaption></figure><h3>Quick Intro: HELM</h3><p>Helm renders your configuration at deploy time and <em>does not keep the result</em>. But that result is what your cluster actually runs.</p><p>Much operational pain follows from this. You cannot look up what was deployed last month, see what a change will touch before it ships, or make a change that survives the next chart upgrade.</p><p>The fix is to <em>render once</em>, keep the output in a <em>store</em>, and <em>make changes there</em>. Charts and values files stay exactly as they are, and so does Argo or Flux.</p><p>People say they want to keep Helm charts. What we want to replace is the operational model around Helm, which today is <em>different for every chart and every team</em>. Yikes!</p><blockquote>What we need: One way to install, change, upgrade, promote and roll back, whatever the chart.</blockquote><p><em>Rendering early is not a new idea</em>. The Rendered Manifests Pattern, sourceHydrator, Kargo and Gitless GitOps all do it. They keep the output as files, however files hold no durable relationships between environments and may be replaced by the next render. We’ll fix some of this.</p><p><em>We use OCI a lot</em>. Our store is “OCI in and OCI out” so it fits any standard container pipeline <em>without a migration</em>.</p><p><em>Rendered objects are not the whole story</em>. In Helm, Hooks, CRD ordering, setup jobs and prerequisites are imperative, depend on the context of the render, and are not safely repeatable. Running an imperative sequence once, at install, is tolerable. Running one on every upgrade is not, because nobody can say what state you are in if it stops halfway. What we can do here is make explicit upfront choices for each supported configuration, recording use cases (or ‘playbooks’), and automate that for you (and AI).</p><p><em>This gets us a ‘99% solution’</em>. Most of what a chart does can be settled once, when the package is built, and checked then. The number of settings left open at install time should be small, though never zero, and that small set should carry its own schema saying how it may be set. Doing this chart by chart is real work, and it is exactly the kind of work an AI can do well, and record. So we can give you a <em>catalog of working charts and examples</em>.</p><blockquote>Agents make this urgent. A person fills the gaps with memory and colleagues. An agent has neither, so it proposes changes against configuration it has never seen.</blockquote><p><strong>Try it today.</strong> Pick a chart. Run helm template for two environments, keep the outputs and diff the results. Do you see any surprises in the diff? We shall talk about why and how to address those below. And in our next post we’ll get super practical on the ConfigHub Tutorial.</p><p>Onwards!</p><h3>WHERE is my real config?</h3><p>Imagine a week with Helm. Your setup is solid. Charts are pinned by version, values files per environment, every change flows through a PR, with Argo or Flux reconciling in the cluster.</p><p><strong>Monday.</strong> Dependency update. Your ingress chart goes from 4.9 to 4.10. The PR is a one line version bump, approved and merged. What actually changed on the clusters was 900 lines of rendered YAML: new probes, labels, a sidecar. You can produce that diff if you have set it up, with helm-diff or by rendering both versions in CI. But it is computed at review time from the chart and the values, so it is only as good as the re-render, and it disappears when the job finishes. 900 lines a lot to review and record what changed.</p><p><strong>Tuesday.</strong> Security audit. “What exactly was running in prod-eu on the 14th of last month?” There is no place to look that up. Someone checks out the commit from the 14th, re-renders the chart, with the Helm version the tooling had <em>then</em>, not now, and submits the output with the comment: “sorry but this is our best reconstruction”.</p><p><strong>Wednesday.</strong> You raise a shared timeout in the base values. Tested, reviewed, merged. Three weeks later you learn prod-eu never got it: its values file has overridden that key since a migration two years ago, the override silently shadowed your fix, and nothing anywhere would have told you before shipping. The “did EU actually get it?” ticket thread runs to forty messages.</p><p><strong>Thursday, 2:14am.</strong> There is an incident. The on-call engineer finds it: a resource limit. They edit the live Deployment; the service recovers, and everyone sleeps. At the 9am standup the graphs show a second dip at 6:40: the sync ran, the fix was reverted back to the broken value, and the record of what on-call knew at 2am is buried in a comment.</p><p><strong>Friday.</strong> You need to set a field the chart does not expose. You try a post-renderer, and locally it works. Then you go to deploy and find that Argo CD does not support post-renderers, so now you are either rendering the chart through Kustomize or writing a config management plugin. You use an ApplicationSet, so that decision changes how configuration is rendered for every environment, not just this one. And you still have to work out how to compare what is running now with what the new pipeline will produce. All of this to set one field.</p><p>You made no mistakes this week! But it was painful. The configuration your clusters actually run, the rendered, final, real thing, <strong>exists nowhere</strong>. On Monday nobody could review it. On Tuesday nobody could look it up. On Wednesday nobody could ask what a change would touch. On Thursday we lost a key change. On Friday nobody could edit it directly. It is rebuilt from source, in memory, at every sync, trusted on faith, and thrown away.</p><p>You wouldn’t ship code this way. Nobody compiles on the production host, at deploy time, with whatever compiler is lying around, and discards the binary. You want as much traceability as possible, and cause and effect on the record, especially in the <em>Mythos era of racing CVEs and patches</em>. For some critical configuration data, we are still shipping source to production and compiling it there again and again.</p><p>This is what we want: <em>build once and keep the output artifact, then configure and deploy the artifact.</em></p><h3>WHY are we here?</h3><p>Start with how the tools work today.</p><p>You write a chart and some values files. That is the source. It is small and reusable: one set of templates, one small file of differences per environment. Programmers call this style DRY, Don’t Repeat Yourself.</p><p>To deploy, a tool runs helm template, or Kustomize builds an overlay. That step is the render. It combines templates and values and produces the full set of Kubernetes objects for one environment, every field written out. Call that output WET, Write Every Time. The WET is what the cluster actually runs.</p><p>Now here is the important part. In standard GitOps, the render runs inside the deployment tool, at sync time. Argo’s repo-server and Flux’s controllers render it, cache the result, apply it, and drop that cache the next time they render. The output is not kept anywhere you can use. Git holds the source. The cluster holds the result. The rendered configuration, the thing in between, the thing you would want to review, look up, and change, is not stored anywhere.</p><p>Every pain in the week above is caused by that missing information. Monday: you cannot review a diff of files that were never saved. Tuesday: you cannot look up an old version of files that were never saved. Wednesday: you cannot ask questions of files that were never saved. Thursday and Friday: you cannot write changes into files that were never saved, so changes go to the cluster, or into patches, and the next render overwrites them.</p><p>The industry has introduced tools to address some of these issues. Many share the same conviction: the render belongs before delivery, not inside it. CI rendering, the Rendered Manifests Pattern from the Argo community, is the original. Argo CD is adding a sourceHydrator natively, where the input field is called drySource and hydrateTo can put the rendered output on a review branch first, which is the closest thing anyone has to a review gate on real objects. Kargo renders as part of promotion between stages. Flux consumes OCI artifacts directly. Gitless GitOps renders in CI and applies signed OCI, with no git in the delivery path at all. All of them save literal, fully evaluated “WET” configs; ie. better than rendering at sync time.</p><p>They differ in where the render runs and where its output goes. They agree on what the output is: files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cPkzjlMGoGZHTS-V_8sFzg.png" /><figcaption>current rendering strategies</figcaption></figure><p>Note these problems:</p><ol><li>Files hold every final value but none of the relationships between values, so the everyday questions have no direct source: which environments share this value, which override it on purpose, what would this change reach, who may change this field.</li><li>To answer, you need the values files, tribal knowledge, and good use of grep. Perhaps this is a reason teams keep rebuilding the same promotion, review and audit machinery in CI and around the files.</li><li>And a saved render is still a build product, so the next render replaces it, your edits included. That is the cause of the tracking issues on Thursday and Friday. A stored render is a cache of the source, and a cache is not a place you can safely make changes.</li></ol><p><strong><em>What you want is the opposite arrangement.</em></strong></p><ul><li>Store the rendered output once, then treat the stored copy as the real thing: changes are made to it directly, recorded one by one, and a chart upgrade arrives as one more change, merged with yours, instead of a rebuild that replaces everything.</li><li>Keep the source next to it, chart, version, values, so every stored object can say where it came from.</li></ul><p>In short:</p><blockquote><strong>Build once.<br></strong> <strong>Make the output artifact the record, in a config data store you can query.<br></strong> <strong>Now change configuration and deployment by operating on that config data store.</strong></blockquote><p>In previous posts we have called this the store: a database holding a graph of config variants. More on graphs in later posts. Here we focus on Helm.</p><h3>Back to the Future</h3><p>Let’s try our “Helm week” again, but this time using that config data store. Same team and charts.</p><p><strong>Monday.</strong> Your ingress chart goes from 4.9 to 4.10. The version bump arrives as a computed change against the record: 900 lines of difference, present before anything ships, and present as data rather than as a wall of text. That is the difference that matters, because a person scrolling 900 lines catches nothing, while a check that knows what a probe or a security context should look like catches the three lines that matter. What a person reviews is the summary and the exceptions, for example in this blog post showing changes <a href="https://confighub.com/blog/validating-scoring-and-gating-kubernetes-configuration-after-every-change-a11fbab14110">validated</a> by policy in ConfigHub.</p><p><strong>Tuesday.</strong> “What was running in prod-eu on the 14th” is a lookup. The record holds every revision of the real thing; the auditor gets an answer, not a reconstruction.</p><p><strong>Wednesday.</strong> Before the timeout change moves anywhere, you can see where it lands: these environments take it, prod-eu does not, because its override is recorded as a deliberate exception with a reason attached. You find that out before shipping, not three weeks later. No forty-message ticket thread.</p><p><strong>Thursday, 2:14am.</strong> The on-call engineer’s fix is a change to the record and it has their name on it, applied from there to the cluster. The morning sync has nothing to revert, because the record already agrees. When the chart’s own fix arrives next week, it lands on the base and merges forward into the on-call engineer’s change.</p><p><strong>Friday.</strong> The field the chart never exposed is just a field in the record. You can just edit it, in your environment’s variant. It survives the next upgrade, because upgrades merge forward instead of pasting over you. No fork, no patch, no need to roll a saving throw today :-)</p><blockquote>How did we get rid of the pain? We kept the rendered, literal “WET” config durably, as data in the store, and made the store the authoritative write path. Because everything, the config, the changes, the environment structure, and the sources, lives in one place, the same data serves GitOps delivery, fleet operations, and AI tools. That is what we mean by a config database.</blockquote><h3>WHO is going to do this work?</h3><p>I want to say why the config data store is also an essential foundation for using AI agents.</p><p>In 2026 it is likely the above ops work will be done by an agent and not only by a human. This is important because the tracking and information gaps that made that week <em>painful for humans</em> are real deal breakers if you want agents to behave correctly. Let’s take a quick look.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BBtUy2divdQnzZ6AtLmyMg.png" /><figcaption>human or agent?</figcaption></figure><p>A person fills those gaps with memory, colleagues, and a thread from three weeks ago. An <em>agent lacks all that context</em>. Therefore it proposes changes against configuration it has never seen, at a rate no reviewer can absorb. That is a real problem.</p><p><em>However with the record in place</em>, in the store, every cell in the third column now becomes a <em>query against stored data</em>, and <a href="https://confighub.com/blog/gitops-in-the-age-of-ai-4-977d800175f4">Part 4’s machinery applies</a>: the change is proposed against the real objects, its footprint is computed before it moves, and a field can carry an owner, so two agents changing the same value is arbitrated rather than last write wins.</p><h3>HOW does the store add value?</h3><p>And in particular how does the store help with HELM pain points?</p><p><em>Do we need to “fix Helm”?</em> If you have been doing this a while, you have watched a procession of new tools arrive to fix Helm: new template languages, new overlay systems, new DSLs, most of them now unmaintained. Each asked you to <em>rewrite what already worked.</em></p><p>ConfigHub’s approach is a different kind of proposal. The difference is structural: all those tools competed with the chart, and our proposal does not touch the chart. Templates, values files, the chart ecosystem, your golden path, all unchanged. <em>What changes is what happens to the output after the render.</em></p><p>First, what the store is. A database that holds your rendered Kubernetes configuration as structured data, and that inputs and outputs OCI images. Each environment’s configuration lives in it as a set of <em>versioned units</em>. These are the actual objects: every field value is there; not hidden in templates. Alongside them you can retain metadata and comments, for example the chart source, version and values that produced them. Also the history of every change with who made it, relationships between environments: which one is a variant of which, etc.</p><p>You read and change config data through a CLI and an API, and it publishes plain YAML out the other side, as OCI images (or files), for the Argo or Flux you already run. The cluster side of your setup does not know it exists. You can insert it into any OCI GitOps flow.</p><p>Where is the added value? It is more than the render-first family we set out in the table above. Those approaches save the right thing and save it as text, so the relationships are gone and the next render still wins. The store <em>keeps the same content as data</em>, with the <em>relationships recorded,</em> so a footprint is a query and your changes merge forward.</p><blockquote>If you already render before delivery, this is the same idea carried to its logical conclusion.</blockquote><p>Note that this is not “another way to write charts”. There are extra steps: render, import, and then the environment structure if you want it. What matters is that they happen once <em>per chart version</em> rather than once per deploy, and they replace more complex steps you are already doing. For example: the post-renderer, the config management plugin, managing the patches that have to survive every upgrade.</p><p>Helm’s model is: the chart is the truth, and every install or upgrade rebuilds the output from it. The store inverts that: the config output is the truth, the record, and the chart becomes an input, one source of changes among several. That inversion is exactly why your edit survives an upgrade here and gets lost in Helm.</p><blockquote>And it is not another deployment tool. It does not talk to your clusters. Argo, Flux, or kubectl still do the applying; the store is where the configuration they apply is kept, changed, and accounted for.</blockquote><p>It also does not require a migration. If you already run Git to CI to OCI to Argo or Flux, the store can sit inside that flow and publish the same objects straight through, with its own digest and a provenance annotation, before you make a single variant. You can check that by comparing what went in with what came out.</p><p>So here is what “keep using Helm” looks like</p><ul><li>Authoring is unchanged: charts, values files, the community ecosystem, all as before, and chart authors do nothing differently.</li><li>So here is the boundary. Values stay where they are and do what they are for. Anything the chart exposes, and you <em>set routinely</em> belongs in values. Keep doing that if you like!</li><li>The <em>store is for what values cannot do</em>: a field the chart does not expose, an edit that has to survive the next upgrade, a difference in one environment that you want recorded with a reason, and anything an agent or a 2am fix wrote.</li><li>Both layers stay visible, what came from the chart and its values, and what was changed after the render, so you can always tell which is which, and a change that collides with a chart value is something the store can flag rather than quietly resolve.</li><li>Upgrades run through the store’s merge rather than Helm’s rebuild.</li><li>The render becomes an import step rather than the deploy path.</li></ul><p>If your team lives in helm commands all day, this is a real change.</p><blockquote>Helm is kept as the way configuration is <strong><em>written</em></strong>.</blockquote><blockquote>It stops being the way configuration is <strong><em>run</em></strong>.</blockquote><h3>Walk it through with Redis</h3><p>We present an example which also uses some of our tools. Please do as much of this as you like: the first part needs nothing but the tools you already have, the second needs no account, and only the last one asks you to sign in to ConfigHub.</p><h4>With tools you already have</h4><p><strong>1. Render it yourself and keep the output.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b778D-QCS5kOnLdSHyu2OA.png" /><figcaption>rendering redis chart</figcaption></figure><p>That file is the configuration your cluster would actually run. Read it. If you deploy Redis to more than one environment, render each of them and diff the outputs against each other; that diff is usually how your environments really differ, and it is where a long-forgotten override shows up.</p><p><strong>2. Read the Secrets before anything is applied.</strong> The Redis render includes a Secret with generated password material. On your machine you can see it. Through a sync-time render you cannot, which is how a chart-generated credential ends up in a cluster without anyone having looked at it. This is also how you inspect what an agent’s values actually produced, rather than what it said they would.</p><p><strong>3. Deploy the files, not the chart.</strong> Commit them to a branch or push them to a registry, and point the Argo or Flux you already run at them. Any OCI push tool will do for now, and step 7 below will replace this with one command. Good work! you have now built the Rendered Manifests Pattern by hand, and it is a real improvement: reviews show the actual change, and an auditor has something to look up.</p><p><strong>4. Try to break it.</strong> Edit one field in the rendered file, ideally a field that the chart does not expose at all, and then apply it. <em>It works.</em> Then upgrade the chart and re-render. Your edit is gone, because the render only knows the chart and the values, and your edit was in neither. Keeping it would mean a three-way merge between the old render, the new render and your copy, by hand, for every file, every environment, every upgrade.</p><p>Ok, next!</p><h4>With our open source tools, and no account</h4><p>Bear with me, because Helm is complex and I want to skip a lot of details for now.</p><p>I want to show you some tooling that does not break or lose config data due to re-rendering. Remember that our strategy is: Helm is kept as the way configuration is <em>written</em>, but it stops being the way configuration is <em>run</em>. Now, I also want to give you a way to see this <em>without </em>signing up for our server. You are also welcome to sign up for the free trial on our SaaS edition.</p><blockquote>I have chosen to use Brian’s package installer which is designed to only use explicit config. This means we can see what we are installing at each step, and update, patch and customize installations without re-rendering. It does not have templates or a programming language. We can use this tool to capture the output of Helm in a reusable form. That is we can take a chart and create config files, which the installer can then deliver as a package. The idea is to produce exactly the same outputs as Helm! But in a way that we can manage without re-rendering.</blockquote><blockquote>To save you having to learn anything, I have prepared some test packages, all pre-rendered.</blockquote><p><strong>5. Take a render someone already did.</strong> For demo purposes I shall <a href="https://confighub.github.io/helm-expt/site/charts/index.html">publish a small public catalog of charts already rendered, reviewed and recorded properly</a>. Pulling one writes the files locally and does not run Helm, because the files are the package.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1W-vzpUcnfrVNZ8PoTS5Iw.png" /><figcaption>same redis new package!</figcaption></figure><p><strong>6. Read what it wrote, then check it against Helm.</strong> The objects land in out/manifests, and if the preset separates secret material it goes to out/secrets rather than being mixed in with everything else. Run helm template and compare object for object; across the catalog it currently matches on every measured row.</p><p><strong>7. Publish the rendered objects as an artifact, in the same command.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NxX8bxnbHMQ8Buug7jA77w.png" /><figcaption>release OCI</figcaption></figure><p>That writes the exact non-secret objects you just read, records which package and which base produced them, then pulls the artifact back and checks its object-set digest before it returns. Point Argo or Flux at it and your controller receives precisely the objects you inspected. This is what we do: check an artifact in, work on it, and check an artifact out.</p><p><strong>8. Choose a setting and watch it survive one upgrade.</strong> Set an image or a value at pull time, then pull a newer chart version. Your choice is carried forward rather than reset to the chart default. Win!</p><blockquote>At this point you can see everything, prove parity with Helm, publish a checked artifact your controller consumes, and keep the settings you chose.</blockquote><p>What you still cannot do is keep an arbitrary edit, one to a field no chart value exposes, and nothing yet knows how your environments relate. For this you will need a config data store.</p><h4>Sign Up for ConfigHub</h4><p>In simple terms this will let you store config, data, policies, access and relationships. Create new versions, patch, upgrade and do bulk operations. It is OCI in and <a href="https://docs.confighub.com/background/concepts/oci/">OCI out</a>: hand it an artifact of rendered objects, work on them, and get a checked artifact back, which is how it fits a flow you already run without a migration. Like a content management system for configurations.</p><p><strong>9. Put the files under management.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zsWlwORFF76gAHGfKmvP_g.png" /><figcaption>easy peasy</figcaption></figure><p><strong>10. Edit a field the chart never exposed.</strong> Not a value, the object itself. Plan it, upload it.</p><p><strong>11. Upgrade the chart, and keep your edit.</strong> Pull the newer version with the merge flag and the upgrade arrives as a change against your record rather than a rebuild over the top of it.</p><p>See also <a href="https://confighub.github.io/helm-expt/site/try.html">Redis walkthrough in detail</a>.</p><p><strong>Summary</strong></p><p>We upgraded in place! We did not lose anything. This is great news for platform and app teams who need to tweak configurations.</p><p>Starting from the public Redis 25.5.3 package, fourteen objects were imported and a change was recorded after the render, dropping the replica StatefulSet from three to two. The chart was then upgraded to 27.0.0, a major version that also moves Redis itself from 8.6.3 to 8.8.0. Before anything ran, the plan said what would happen: nothing was added, thirteen objects were changed, nothing was deleted, and there was no reset of the replica count. And after the upgrade, the replica count was still two.</p><p>The rest of the run is the part a Helm user cannot do at all today.</p><ol><li>Two environments were shown as being in the path, development first and staging behind it. Both were promoted.</li><li>A release will be published, a portable OCI built and pulled back with the files matching, and two Argo CD clusters reconcile the same digest with Redis answering on both.</li><li>Then the exact pre-upgrade revisions are then restored under a change set named for the rollback, a rollback package can be published, and both clusters returned to 25.5.3 with two replicas, healthy. Every step will be a recorded, versioned change.</li></ol><h3>Every chart is different: so we need a catalog of best practice</h3><p>I said Helm was complicated. Hooks, CRDs, oh my! How can we make these behave? The short answer is that there is NO programmatic transformation from a general Helm chart to another authoring format. That may be why all the “Helm replacements” did not take off. People actually use all the funky extra features in Helm even when it may not be best practice. And so:</p><p><em>All charts can be managed this way; but the best way is not identical for every chart. Values layer differently, templates conditionally produce different objects, some fields are structures and some are just knobs. One way to deal with this is to have a small number of core “base” use cases, each with their own setup.</em></p><p>The — base default you saw in step 5 is one of those: a supported way to run one chart version, with the chart, version, values and choices that produced it recorded alongside the objects, and the review evidence published.</p><p>A base also records what the chart does at the edges. Charts install CRDs, run setup jobs, generate Secrets, and expect things to already exist, and none of that is silently executed here: each behaviour is a recorded choice per base, and nothing is described as automatic until it has actually run and been recorded. In some use cases we might not use the CRDs at all.</p><p>Follow that one step further and something bigger appears: <em>best-practice editions of your charts, the same chart, held as data, specialised to a policy set, PCI, air-gapped, dev-loop, shareable, and maintained as the base evolves.</em></p><h3>Try it</h3><p><strong>If you have not done the walkthrough, do it.</strong> Steps 1 to 4 take an hour with tools you already have, and step 4 is worth the hour on its own, whatever you decide about the rest. Steps 5 to 8 need no account. The site has the copy-paste version for all of it. TK_SITE_LINK.</p><p><strong>If you want to try it on something of your own, here is the interesting one.</strong> Take a chart and a values file that an AI wrote for you, and run it through the same path: render before applying, read what it actually produced, fix what it got wrong, and package the reviewed result as a checked artifact with the fixes recorded. We did try this out, with an NGINX values file. We found six problems in one file: an API key sitting in plain text in the Deployment, an unpinned image, non-root execution turned off, privilege escalation turned on, the Service quietly changed to a LoadBalancer, and a writable root filesystem.</p><h3>Where this leaves us</h3><blockquote>Your phone does not lose your contacts when the app updates.</blockquote><blockquote>Your clusters should not lose your configuration due to how you use Helm charts.</blockquote><p>Helm and CI are kept for what they are good at: authoring and building. Operating moves to the config data record: the ConfigHub store. Charts continue to be authored by package owners. CI keeps building and rendering. But we keep the output: it stops being a throwaway compile on the production path and becomes the record. If you remember one thing: build once, make the output the record, change everything from there.</p><p>Next in the series: the <em>ConfigHub Tutorial</em>. After that: the store. Where the record actually lives, what it holds besides config, secrets included, and why an export to Git and OCI means you can always walk away.</p><p>– alexis</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/356/1*zdFbrHbWeqm5hLTdCT32uQ.gif" /></figure><h3>Postscript: other tools</h3><p><strong>Kubara, the same shape at platform scope.</strong> Everything above was argued at the scale of a chart. Kubara (kubara.io) shows a whole platform: an open-source CLI that generates Helm and Terraform from embedded best-practice templates plus your config, hands the durable files to a GitOps repo, and bootstraps Argo to apply them. Generation up front, durable output, a reconciler applying literal files, and an opinionated edition of dozens of community charts in one move. Adding a database such as ConfigHub brings the fleet-scale questions within reach: who changed what after generation, what does a re-generation overwrite, what is the blast radius of a platform-version bump across many such platforms? The answer is the same as before: keep the WET as data, with its structure, its owners, and its history.</p><p><strong>Rendered Manifests and more.</strong> Prior art, gladly acknowledged. The Rendered Manifests Pattern and sourceHydrator come from the Argo community, who also supply the best one-liner for the disease, rendering at sync time is like running apt-get install in a container’s startup script, and the repo-server arithmetic for why render-once pays. “Gitless GitOps” is the same instinct. The variant model here was shaped by the creator of Kustomize, to fix the pains that overlays exposed. The pattern is converging from several directions. What this series adds is what the stored result is: not files, a record, as data, with its structure recovered.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9b4573c92765" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GitOps in the age of AI — 4]]></title>
            <link>https://medium.com/@monadic/gitops-in-the-age-of-ai-4-977d800175f4?source=rss-c1c94bbe89c2------2</link>
            <guid isPermaLink="false">https://medium.com/p/977d800175f4</guid>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[gitops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[confighub]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[alexis richardson]]></dc:creator>
            <pubDate>Thu, 16 Jul 2026 19:01:01 GMT</pubDate>
            <atom:updated>2026-07-22T08:28:59.369Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Part 4 of a series. Parts 2 and 3 were theory. Now we apply that to a use case.</em></p><p>Set up the right stack, make sure it is running, check that it has the right add-on components, data and service integrations. Make a change. Patch the system. Replace the container image. Set up dev, stage and production. How are these related to your base system, to each other? What about fleets, how are your apps set up? Roll out a change in stages, maybe you have to check some tests and gates before promotion to production? Now do this “progressively”. You have lots of different use cases and your ordering of changes is dynamic, continuous. And now the boss team has arrived, bringing Mythos, and you have to patch all your Helm, yesterday!</p><p><em>This is when we can ask: do you have a configuration management system that can understand all this and automate it, using AI where possible, and provide dashboards for every job. In today’s post we shall look at a practical example of this, with generative GitOps: </em><strong><em>Promotions</em></strong><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HfinZgGzYC7waC7l_xB-yA.avif" /><figcaption>heath robinson imagines CICD promotions</figcaption></figure><h3>Promotions as Code or as Promotions Data?</h3><p>Promotions, patches, image rollouts, config updates are versions of the same job: advance a change through your environments until it is running everywhere it should be. Mature GitOps teams already run these through one process: change the config in git, promote it with CI, add policy etc.</p><p>But look inside a CI promotion pipeline and, unless its well crafted to a simple case, you may find bespoke, imperative CODE. This is fragile. It can end in a broken state, half promoted, half not. It makes rollbacks difficult. Break glass? Another custom script? Every team writes this machinery, every team maintains it, and rarely is it part of the configuration data itself.</p><p>To manage this safely, CI owners will often simplify. They may take a snapshot and pass that ‘through’ the pipeline (or workflow). Maybe a githash names the state of a whole repository. It does not identify and describe the changes to declared config: which values moved, what they touch downstream, who may approve them, what proves they landed on each target. All of that lives in people’s heads and in the pipeline scripts.</p><p>Can we do better? Yes, if we can see changes as things, rippling out across a data set. This data set represents your operational desired state — your configs, your facts from the running systems, and the record of every change. A change can be a real thing, with an identity, moving through your environments. It can move through policy gates, leaving a record at each step, one operation each time. <strong>An operation on configuration DATA.</strong></p><h3>Traditional GitOps did not include promotions</h3><p>A note on the GitOps state of the art for this, and what we hope to do differently <em>with the goal of simpler promotions; and usage by other CD tools.</em></p><p>GitOps was never originally conceived to solve promotion by itself. The core reconciliation contract is target-local and does not specify how one release advances between targets.</p><p>I want to talk about why, and then talk about what can change today.</p><p>The argument goes like this: GitOps is a loop. One desired state, one environment, continuously reconciled. Then <em>dev, stage and prod</em> are three loops between three desired states and running environments. But a promotion is not a loop. It is an operation: the act of making and moving a change from one ‘dev’ loop, to the ‘stage’ loop, and then to the ‘prod’ loop.</p><p>The GitOps principles said nothing about it. Deal with it how you like! And this gave us flexibility: we could organise dev, stage and prod repositories how we wanted and not be tied down to some CICD/promotions vendor.</p><blockquote>But people wanted promotions!</blockquote><blockquote>If you breathe, eat and sleep CICD then switch to GitOps, you will ask “give me back my pipelines!”. And the thing is they never left. GitOps moved the deploy step out and left promotions behind.</blockquote><blockquote>That created an ad hoc mix of declarative GitOps loops, and imperative job implementations. Not always with happy results: Promotion gets assembled from parts. Environment folders here. Cherry-picked commits there. CI jobs with approval steps holding it together. It adds up to a scrappy work-around for doing operations on configuration data, and is too unpredictable for adding AI safely.</blockquote><p><strong>Let’s “name the pain”: promotion pipelines in CI are imperative, fragile and brittle. They can easily end in broken states. They make rollbacks difficult. Can you break glass? It’s all bespoke, custom, imperative code. Maybe some humans can understand it and “get by”, but AI will barf.</strong></p><p><strong>Agents!<em> Proposing changes by the dozen</em>. What used to be contextual joins become a bottleneck. And user questions shift from “how do we promote?” to “how do we let an AI help without losing control?” And the answer is not “add more things, add AI config, make it more complex”.</strong></p><p><strong>We have to simplify.</strong></p><blockquote>How can we do this without creating a <a href="https://socks-studio.com/2016/11/07/william-heath-robinsons-wacky-inventions/">“Heath Robinson”</a> assembly of repos, CI jobs, workflows, GUIs and magic buttons? We need a proper <strong>model for config operations</strong>, an extension to GitOps that can explain how config changes.</blockquote><h3>Prior Art and New Art</h3><p>There is a ton of great prior art. Octopus, Harness, Kargo, and the vendors around Argo and Flux. Please see my footnote(*). <em>We want to work with you.</em></p><p>A few things are new. We have a declarative model that treats configuration as data and adds relationships, schemas and expicit config management of variants in a <em>graph</em>; which also includes relevant context for dealing with generation of configuraton and transforms for infra and app operations. This lets us <a href="https://confighub.com/blog/gitops-in-the-age-of-ai-c73d0bd506ba">expand what GitOps means</a> and make space for pipelines and promotions as data. In this post we write down an illustrative MVP for that.</p><p>We think this model is more suited to the machinations of AI tools and will enable combining deterministic control with non-deterministic AI outputs.</p><blockquote>In the past operations have been seen mostly as some kind of step by step workflow. This mental model is everywhere: promotions, approvals, cost optimisation, incident management. WHY MUST IT BE THIS WAY? When using workflows, passing objects between processes, to get a job or “actions” done, then your implementations are either very simple or ephemeral; or will involve ever-widening transaction and event models. Could we instead, for some workloads, implement operations as updates on a shared database, ie. “apps”. With the simpler mental model and reliable failure modes of, say, Vercel or Lovable.</blockquote><h3>The model, in plain words</h3><p>I shall describe a conceptual model: more ‘specification’, and less ‘implementation’ which is out of scope for this blog post.</p><p>A <strong>change</strong> (config revision) is a sealed (immutable) set of edits with an identity. Say, “image: 2.3 to 2.4”, sealed as change CS-42. Once sealed, it never alters. If the content changes, that is a new change(***footnote).</p><p>Your <strong>environments</strong> are variants of one base configuration: dev, stage, and prod, related and connected. In one example, prod is also three regional variants of one environment. We track what each one inherits and how extensions and override customisations work. This is a config graph and if you use Helm or Kustomize today then the graph is what your values files and overlays already imply, made explicit and queryable. My colleague Brian Grant has <a href="https://confighub.com/blog/configuration-belongs-in-a-database-be6d31f46d3d">written about this in depth</a>.</p><p>A <strong>promotion</strong> is one ‘governed’ revision: move this change to the next environment, but before it lands, gates check it; and after it lands, it is recorded.</p><p><strong>Operational Context. </strong>Observed status, gate info, live checks and other ‘facts’ flow back from the running clusters. Is it reconciled? Is it healthy? Those facts, plus approvals, are what open the next gate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z5aKri1fTk-mNzg1cpL97g.png" /><figcaption>promotion sequence and checks</figcaption></figure><p><em>One change, moving left to right. Evidence flows back to the ConfigHub store, right to left. Reconciliation both ways.</em></p><p>The pipeline itself is data. At its most distilled, it is an ordered list of environments and gates that the user declares: dev, then stage, then prod. And, each can be a set of targets: several prod regions, a canary group, a blue/green pair. The pipeline is not a CI system: it runs nothing, it only constrains what may move, to where and when.</p><blockquote>This is a consequence of <strong>config as data</strong> in our model. Two properties matter most. The promotion machinery becomes <strong>declarative</strong>: an object you read and edit, not scripts you maintain. And it lives in the <strong>configuration store</strong> itself, as a first class citizen, next to the config it moves, instead of outside in CI.</blockquote><p>Two more useful things follow.</p><ol><li>First, it is additive. Declaring a pipeline changes nothing about your variants or their edges. It adds one small object beside them. You can change it, or remove it, without re-plumbing anything else.</li><li>Second, it is visible. You can view and edit it like any other configuration, in the GUI or from the CLI. Humans can read it. Agents can read it. So “where is my change” becomes a question the system answers, rather than a Slack thread.</li></ol><p>Let’s look at three examples. Maybe these are jobs you already have.</p><h3>Example one: an image rollout</h3><p>First, the order itself. Someone on the platform team declared it, once:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*44qHdycvphGEV4zZoE_LDA.png" /><figcaption>promotion sequence</figcaption></figure><p><em>Illustrative syntax of environments, an order, and checks at each gate.</em></p><p>That is the pipeline object, showing sequential promotion. So it is declarative i.e. the promotion sequence is stated but not executed as a job script. A simple interface but powerful capabilities exposed, where for illustrative reasons I have simplified and idealised a couple of things, eg. syntax, likely names, and showing small pieces of configuration, each written once, and edited when the process changes.</p><blockquote><strong>Notice what isn’t part of the pipeline</strong>:</blockquote><blockquote>The promotion sequence is not a property of the underlying config graph, which expresses logical config dependencies between base config, dev, stage and prod. It is not inferred from the config variants, or from a repo layout, or from labels and naming conventions.</blockquote><blockquote>It is declared: promotion order is a team’s process decision, not a property of the configuration’s structure. You change the process, you own the data, you edit the object.</blockquote><blockquote>And because the declaration went in as a <strong><em>governed GitOps change like any other</em></strong> “who decided prod needs an approval, and when” is itself on record.</blockquote><p>Second question: where does the change come from? Nobody should be typing image tags into a box. CI builds checkout-service 2.4, and a registry event proposes the sealed changeset automatically, through the API. It lands in an inbox of available changes for this sequence:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XomM0oFvuW7abtf6urpwjA.png" /><figcaption><em>changes arrive and you (or a rule) decide what moves.</em></figcaption></figure><p><strong><em>What this makes easy? </em></strong><em>Because an image reference is just a configuration field, image and config changes are the same kind of thing, so “this image with that config” needs no special bundling: you select which sealed changesets travel together, either alone, both, or neither, in one governed promotion. And unlike promoting a pinned snapshot, the combination is applied as changes against what each target currently runs: every field keeps its owner, the footprint is planned per target, and deliberate overrides can be protected inside the combined change. Want the older config back instead? A revert is just another changeset.</em></p><p>Now the rollout:</p><ul><li>checkout-service 2.4 is ready. The change: <em>one field, image, 2.3 to 2.4</em></li><li>in our simplified promotion sequence it is already live in dev</li><li>you promote it to stage</li><li>the system shows what will change before it does: one field, one target</li><li>it applies, writes a receipt (= the record of what was done) and waits</li><li>facts come back from the stage cluster: <em>reconciled</em>, <em>running</em>, <em>checks green</em></li><li>the stage half of the prod gate is satisfied; it waits for approval</li><li>a human approves, and the change may move to prod</li></ul><p>Prod is three regions, and this is a way for the config management graph to do some more work. Before anything ships, the fan-out is computed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bo1yrnvOk3IQElDFUVsQxA.png" /><figcaption>a map of the promotion fan-out</figcaption></figure><p><em>This map should come back before the change goes anywhere.</em></p><p>Now let’s review, or imagine an agent doing this: prod-eu keeps its deliberate override( → Note 1). No one had to remember the exception, because it is recorded as an edge in the graph and surfaced when the change is planned. Two records (‘receipts’) were noted, one per region that applied.</p><p>Then: when prod-ap’s health facts come back bad an hour later, that is not a “mystery outage to try to explain”, it is a recorded fact, it blocks the pipeline, and you can roll back ( → Note 2)</p><p>Notes</p><ol><li>You can express the override in Helm today, a pinned value in prod-eu’s values file. What Helm cannot do is surface it at the moment a change is planned, attach an owner and a reason to it, or protect it from being silently flattened by the next edit to the layering. <strong>Context matters.</strong></li><li>Standard tools should already tell you a rollout is unhealthy; that part is not new. What is new is that the signal is tied to this exact change and this attempt, it automatically blocks the rest of the sequence, and it lands in the same record as the change, so “what happened to CS-42 in prod-ap” has one answer (and you can roll back). <strong>Again: Context.</strong></li></ol><blockquote>What this replaces: the Heath Robinson mess of the environment branches, the cherry-pick PRs, and the “did eu get the override reapplied?” thread, all of which come from tying imperative processes together with declarative GitOps loops.</blockquote><blockquote>In plain terms. We arein a model were configuration and context are stored as data, with relationships expressed as a graph. In this: the change itself has an identity and stays sealed; the promotion is planned against each target’s current configuration and not applied without checking first; also a value can have an owner that checked before the write; and the evidence (recorded context) of what happened is tied to the exact change and attempt, in the same record. Deployment products may capture some or all of this information.</blockquote><blockquote><strong>We are doing it in one place as declarative GitOps </strong><a href="https://confighub.com/blog/gitops-in-the-age-of-ai-2-48e13026f434"><strong>in the ‘generative’ style</strong></a><strong>.</strong></blockquote><h3>Example two: Friday’s CVE, with an agent</h3><p><a href="https://confighub.com/blog/gitops-in-the-age-of-ai-3-4d80429b2b24">In Part 3</a> we posed the 4pm on a Friday test. A critical CVE arrives in a base image, is patched and needs to be verified and deployed fleet-wide ASAP. Here is the machinery…</p><p>CI produces the patched image. This is not one service’s pipeline: the vulnerable base image is baked into many services across the estate. So an agent queries the desired state (config graph): which deployments run the vulnerable image, which configuration produces each one, who owns them. It proposes a single sealed change: replace the image digest, everywhere it applies.</p><blockquote>Notice how this differs from example one. That was one service’s own release, moving through its own sequence. The trigger here is a security advisory, not a build event. So the first job is discovery, which is why the graph gets queried.</blockquote><blockquote>An agent asks: which deployments run images built on the vulnerable base, which configuration produces each one, who owns them. That is <a href="https://confighub.com/blog/gitops-in-the-age-of-ai-3-4d80429b2b24">Part 3’s</a> question one, answered as a query. CI produces the patched image, and the agent proposes a single sealed change: replace the digest everywhere it was found.</blockquote><blockquote>Could each service instead pick this up through its own normal sequence, like example one? For a single service, yes. What the fleet needs on a deadline is the campaign: one discovery, one approval, one rollout, one“are we done”. Fifty independent sequences could give you fifty local answers and no completion.</blockquote><p>The system computes the footprint before anything moves. Every affected deployment is listed, along with the deliberate exclusions, each exclusion carrying its owner and its reason.</p><p>A human approves it once. Then the change rolls out through a pipeline whose stages are groups of targets, declared the same way as dev, stage and prod. Some deployment practices call these ‘rings’; others say waves. Note that in example one, prod was one pipeline element containing three parallel targets, in example two rings are several such stages in sequence.</p><p><em>The example below should illustrate the shape, though in a real system there would be more machinery both intra- and inter- cluster.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oyI9k5C4fo2C3oQaIFoaBA.png" /><figcaption>rings are pipelines for fleets</figcaption></figure><p>Each gate opens on detailed evidence that a machine can read. Not “the cluster looks fine”, but the exact new image digest, confirmed as reconciled and healthy in the previous ring. Then later if ring two stalls, the rollout holds. Note where the AI sits, because this is our generative GitOps safety plan: the agent proposed the change and started it moving, and it <em>cannot certify its own success</em>. The evidence is coming from the running clusters, independently, and we can store it in our graph of facts and config. Finally, when done: the proof that this work was completed is a query “what was affected, what changed, who approved it, where it landed, what was verified”.</p><blockquote>What this replaces: the war-room spreadsheet, the per-cluster archaeology, “we think it’s everywhere.”. No more config hell!</blockquote><h3>Example three: two agents, one value</h3><p>This is more of a ‘what if’ case: to see when AI is safe day to day.</p><p>A cost-tuning agent writes a change to the checkout service’s replica count (or maybe to the autoscaler parameters). The same afternoon, a second agent, working a performance ticket, writes to the same field. With writes sequential in time, does Git spot the concurrency? Not quite: with file/text based change operations in Git, whichever agent merges last will win cleanly, and the conflict info will be lost.</p><p>As set out <a href="https://confighub.com/blog/gitops-in-the-age-of-ai-3-4d80429b2b24">in previous posts</a> and above, in our model <a href="https://docs.confighub.com/background/config-as-data/">configuration is data</a> and lives in a graph along with operational facts; so there are fields(**) with owners and we can do this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mx2MkK2x37Q71D7pIJYdfQ.png" /><figcaption>two agents try to make two changes</figcaption></figure><p>Both writes were authorised at the resource level. The collision is named and held. The owner decides and full records are kept.</p><p>Git records what changed in the repository; it does not arbitrate. Two PRs to the same line are resolved by whoever merges last, CODEOWNERS gates files, and at dozens of agent changes a day nobody else is reading the PRs anyway. History should record that a contention existed and how/why it was decided. Named ownership of the value means the collision may also be arbitrated automatically, by policy, with both attempts on record.</p><blockquote>This is a level of deployment control that <a href="https://confighub.com/blog/git-is-not-great-for-deployment-configuration-b1bbad7a5428">100% Git-based operations cannot easily express</a>. We can add value by extending the data model (domain and store), eg. ownership of a single value makes dozens of agent changes a day a routine matter. (and no information is lost; as we set out in <a href="https://confighub.com/blog/gitops-in-the-age-of-ai-2-48e13026f434">part 2</a> and <a href="https://confighub.com/blog/gitops-in-the-age-of-ai-3-4d80429b2b24">part 3</a>)</blockquote><h3>Summary: we did three jobs</h3><p>We showed how the promotions pipeline and policy checks could be expressed declaratively as an illustration of the model.</p><ol><li>An image rollout is a promotion through environments.</li><li>A patch is a promotion whose pipeline is rollout ‘rings’.</li><li>An everyday update is a promotion of one field.</li></ol><p>We have been proposing a more comprehensive and modern version of GitOps that accounts for generation of declarative configuration ‘as data’, graphing logical relationships like ‘base config to stage config’, a data API enabling operations to transform this, and the expression of context for greater control eg. for analysis or remediation. The model is machine friendly: agents propose and initiate, and contextual facts are recorded.</p><blockquote>By removing more of the imperative workflows people have been forced to use, we hope that CICD Promotions applications may be fully declarative.</blockquote><blockquote>This should lead to much higher certainty, lower error rates, and get platform delivery working “on Mythos time”. AI meets GitOps, via a shared toolchain.</blockquote><blockquote>No more Heath Robinson edge cases, no weird transaction breakages as workflow processes conflict at scale and under pressure of a high change volume.</blockquote><h3>How does this fit into my existing GitOps world?</h3><p>We built ConfigHub assuming users who are already managing infra and apps on K8s with OCI, Helm, Argo, Flux, and other GitOps-type tools.</p><p><strong>You keep Argo or Flux and use OCI for artefacts and updates. Your chosen GitOps tool then reconciles updates that are presented via an OCI endpoint, enabling their deployment into Kubernetes.</strong></p><p>Your configuration lives in ConfigHub as variants. We help you manage their lifecycle and relationships via a data API that enables bulk ops and machine automation (AI). You declare a promotion pipeline (sequential flow) by listing your variant configs in promotion order. There may be policy checks declared here too. Changes are ‘sealed’ changesets. A new image version does not need typing in by hand: a registry or CI event can propose the sealed changeset automatically, through the API. Its promotion stays governed like any other.</p><p>Promotion: your agent presses the virtual button, the system computes the footprint, checks the gates, applies, and collects the evidence.</p><p>The view might be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K7WQkvqF9ge-H-pzYL_94A.png" /><figcaption>a change mapped out</figcaption></figure><p><em>What is live right now and where a given change has got to. (TWO facts)</em></p><p>OK. One implementation question: what happens when a second change lands while the first is still mid-flight? The safe default is a queue: changes go through in order. You could say “just take the newest”, but this might silently drop the older changesets unless the newer one contains them. We think the system should surface this question through the UX (GUI, CLI, API) so that different users can answer it explicitly for their own cases.</p><p>What more might users want? Richer policy? Trigger automatic proposals to create enhanced changesets through our API? Automatic promotion after a soak period? Include a break-glass path, an “expedited” hotfix could skip the queue? All AI-led and ConfigHub-backed of course.</p><blockquote>In our world, operations act on data (config, context).</blockquote><blockquote>This takes us into an “apps” model. That is to say, when a human needs to do promotions, they will want a promotions GUI for things like “showing all my pipelines in colour, with actions I must take”.</blockquote><blockquote>That is: a custom data-backed app on top of the core functionality on the graph. And that is much better than a complex transactional evented workflow!</blockquote><h3>Where this leaves us</h3><p>None of this asks you to replace what already works. Git remains where your source code lives. Argo or Flux get simpler and apply the exact configuration that ConfigHub publishes. Desired and running state are kept in sync by GitOps; and your existing policies still run at the gates.</p><blockquote>What changes is that promotion becomes an ordinary, governed operation on configuration data, rather than a process assembled from branches and CI jobs. Patches, rollouts and updates turn out to be the same operation with different pipelines.</blockquote><p>And agents get exactly the role they are safe in. They propose changes and start them moving, under the same authority checks as anyone else. Whether a change succeeded is decided by evidence from the running systems, not by the agent that made it.</p><p>This is the foundation for generative GitOps: treat configuration as data; let humans and agents propose governed changes against the same graph; let independent observer tools return exact-version Ops Facts; and package each operational job as a focused ConfigHub App over shared governance.</p><p>Next in the series: Helm.</p><p>– alexis</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*mMMHsoustMde1xki47cEqQ.jpeg" /><figcaption>we welcome our sequential overlords</figcaption></figure><p><strong><em>Footnotes</em></strong></p><p><strong><em>(*) Why are we here talking about this when CICD products exist</em></strong></p><p>We are proposing a wider declarative model than is easily accomplished using Git, CI, Helm and live Kubernetes. It is a configuration and context store with relations explicitly modelled so that Generative GitOps can capture more operational information and enable simple operational applications like deployment, promotion, cost estimation, remediation.</p><p><em>We would love this to be adopted by GitOps enthusiasts and anyone who is scaling their operational lifecycle using AI with Kubernetes.</em></p><p>Environment promotion is a longstanding Continuous Delivery discipline. Big platform products such as Harness and Octopus model environments, lifecycles, approvals, and deployment processes; GitOps-native tools like Kargo model artifact promotion around continuous reconciliation. Kargo is cool! It gives exact artifact bundles an identity (Freight), links Stages into pipelines, runs Promotions, verifies Freight, and visualises progress.</p><p>GitOps in its original form is about how one environment converges on declared state. Dev, stage, and prod can each have a reconciliation loop, but GitOps does not specify how a tested change becomes eligible for the next loop. Promotion lives between the loops.</p><p><em>The model set out here is narrower at the workflow layer and deeper at the configuration layer: it promotes a sealed semantic change over related variant configs (data objects). In contrast, Kargo and other CICD tools pass ‘freight’ between processes, which can create some operational challenges.</em></p><p>In summary:</p><ul><li><strong>Octopus and Harness:</strong> broad deployment and release orchestration.</li><li><strong>Kargo:</strong> GitOps-native artifact discovery and promotion.</li><li><strong>ConfigHub:</strong> semantic, target-relative configuration transitions with context: authority, provenance, protected differences, materializations, and externally sourced evidence (‘facts’).</li></ul><p>(**) <em>This is not Kubernetes managed fields. Server-Side Apply does more than record ownership: a non-forced apply against another manager’s field is refused, real conflict detection, and credit where due. But it works per cluster, at apply time, and the “owner” is whichever manager last applied, not a declared policy. And in GitOps it is structurally blind to our case: both agents’ changes arrive through the same reconciler credential, one field manager, so no conflict is ever seen. Ownership here is declared in advance, checked at write time, before apply, fleet-wide, with collisions arbitrated and both attempts receipted.</em></p><p>(***) <em>A commit names a snapshot of a tree; a changeset names a semantic delta over typed fields with a declared footprint. CS-42 is one identity across every target, and is applied as a planned delta per target, three-way, not a snapshot.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=977d800175f4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GitOps in the age of AI — 3]]></title>
            <link>https://medium.com/@monadic/gitops-in-the-age-of-ai-3-4d80429b2b24?source=rss-c1c94bbe89c2------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d80429b2b24</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[gitops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[confighub]]></category>
            <dc:creator><![CDATA[alexis richardson]]></dc:creator>
            <pubDate>Tue, 14 Jul 2026 13:31:01 GMT</pubDate>
            <atom:updated>2026-07-15T15:03:17.996Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Part 3 of a series.</em></p><p><a href="https://confighub.com/blog/gitops-in-the-age-of-ai-2-48e13026f434">Last week I posted</a> about how we should update the traditional GitOps model to reflect new operational requirements from AI models and coding tools. Now we shall look at why this matters for applications and software operations. It is clear that for operational factors <em>AI is an amplifier.</em> AI is already amplifying existing risks which impact software operations. For example when IT is already overwhelmed, how can we safely increase the pace of change?</p><p><em>OBJECTIVE: we want to add AI and get measurably better at running software applications and operational “jobs to be done”. We are using GitOps and Kubernetes. What do we do?</em></p><h3>GitOps: from tools to applications</h3><p>In this post I want to look at our infrastructure and tooling for operating applications. We shall focus first on<em> best practice tools</em>. How well do today’s tools perform operationally, when “GitOps meets AI”? What needs to change? Is there a “stack”? What’s missing and how can that help even small scale users build the right applications which are safe for enterprise adoption.</p><p>Two quick <em>considerations</em>:</p><blockquote>Rather than fine-tuning an individual tool, often the problem is how to think holistically about the architecture, observability..; all components, not just some.</blockquote><blockquote>Popular tools are often stretched past their original design goals, with diminishing returns. Then people ask for better tools. For example, Kubernetes did not originally have a packaging tool. Helm filled that gap and soon expanded to configuration generation, using source code, templates, CI tools and (sometimes) live clusters.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*dm7vMWWM5p0dL5x4TGFstQ.jpeg" /><figcaption>consider phlebas © iain m. banks</figcaption></figure><h3>How GitOps loses information</h3><p>As a friend said to me this week: “CI and Helm really suck for configuration”. AI can make this worse. <a href="https://confighub.com/blog/gitops-in-the-age-of-ai-c73d0bd506ba">In part 1 we showed how simple changes plus Helm cause information loss made worse by AI</a>. To solve this we proposed an account of <a href="https://confighub.com/blog/gitops-in-the-age-of-ai-2-48e13026f434">“Generative” GitOps, for which I set out six GitOps principles</a>. The core ideas: <em>config is data, both-ways reconciliation, and agentic proof of change</em>.</p><p>A reader comment was: <em>your outage example is not really an AI problem. A person who forgets the 768Mi override causes the same incident. AI may scale the problem, but it does not create it. </em>YES. The problem is that we are losing information due to config generation and drift. For a decade, Helm, Kustomize, pipelines and CI have been throwing away the input-to-output relationships that we need to control complex systems. Long before agents arrived.</p><p><em>What AI does change is two things: it removes the compensating human, the person whose memory of the override and whose spare reconnaissance time papered over the gap, and it multiplies the volume of change flowing through the same information leaks.</em></p><p>Traditional GitOps loses information in two places.</p><ol><li>At generation: the relationship between input and output, which base produced which variant, what fans out where, is computed and then discarded.</li><li>At drift: someone or something changed the live system, with an identity and a reason, and the revert erases both without a record.</li></ol><p>Why do these “info leaks” matter? Because <em>lost information cannot be checked</em>. Once the fan-out map is discarded, no test can ask “what does this change touch.” Once the drift’s author is erased, no test can ask “was that allowed.” The checks do not disappear, they slide to after the fact, where they are called incidents.</p><h3>Ten tools, and why they matter</h3><p>So which tools should we look at? The ones fleet teams actually run. Almost every tool below exists as a <em>response</em> to lost information due to some leakage. A pattern can exist because the information leak hurt <em>someone</em>, even if it never hurt you :-)</p><ol><li><a href="https://akuity.io/blog/the-rendered-manifests-pattern"><strong>Rendered manifests</strong></a><strong>, </strong><a href="https://argo-cd.readthedocs.io/en/latest/user-guide/source-hydrator/"><strong>hydrators</strong></a><strong>, </strong><a href="https://dev.to/t-kikuc/introduction-to-gitless-gitops-a-new-oci-centric-and-secure-architecture-2pgi"><strong>Gitless GitOps</strong></a><strong>.</strong> Often promoted as current best practice. The GitOps industry converged on these after enough teams learned that the manifests reaching the cluster were not always the inputs that had been reviewed. Render once, freeze, sign, apply exactly that.</li><li><strong>CI and pipelines.</strong> Simply where much of the delivery automation pipeline lives; nobody chose CI for configuration, rendering just moved in.</li><li><strong>Helm diff.</strong> Obviously: see what a release would change before you apply it. You ran it the first time someone showed you.</li><li><strong>Argo and Flux previews.</strong> They came with the reconciler: server-side diffs and dry-runs, a prediction per cluster before anything lands.</li><li><strong>OPA, Kyverno, admission policies.</strong> Compliance asked, platform teams answered, and policy-as-code became table stakes.</li><li><strong>Kubernetes RBAC and Server-Side Apply.</strong> The platform ships them; you configure rather than adopt. SSA’s managed fields are an underrated field-level ownership map.</li><li><strong>Argo Rollouts and canary deployments.</strong> Progressive delivery is now received wisdom for several types of team…</li><li><strong>ResourceQuota dry runs.</strong> Ships with the platform and is usually switched on by the team that got paged.</li><li><strong>Secrets tooling: Sealed Secrets, SOPS, External Secrets.</strong> Git config is readable so act accordingly if you don’t want it to be read.</li><li><strong>Sveltos.</strong> For me one of the exciting new tools. Deliberate adoption by teams already managing fleets, the crowd that already thinks in one-to-many (see this <a href="https://itnext.io/gitops-for-15-000-clusters-what-large-scale-testing-with-vcluster-taught-us-41e4b0d43e0b">15,000 cluster example</a>!).</li></ol><p>Plus one entry at the end that is not a tool at all: <em>drift</em>. We will finish there.</p><p>STOP! “Just give me the punchline”. If you want to skip the ten tools deep dive then please jump down to “We need a data substrate..”.</p><h3>The tool analysis framework</h3><p><strong>Which principle does it already implement?</strong> Of the <a href="https://confighub.com/blog/gitops-in-the-age-of-ai-c73d0bd506ba">six Generative GitOps principles</a>, most teams are already running partial implementations without calling them that. You are further up this ladder than you think.</p><p><strong>Where do its checks run?</strong> This is the shift-left diagnostic, and it is how the leaks show up in practice. For each tool, ask when and where its checks execute: before apply or after, one target or the fleet, and what information the check would need that the substrate already threw away. A right check in the wrong place, or at the wrong time, is a leak, showing.</p><p><strong>What if we have an emergency at 4pm on a Friday?</strong> A critical CVE lands in a base image used across your estate; the CISO wants it patched fleet-wide, and proven patched, ASAP. Three questions define the weekend:</p><blockquote>which of our hundreds of deployments actually run this image, specifically which config, owned by whom — not just an image in a CVE repo</blockquote><blockquote>what breaks when we bump it;</blockquote><blockquote>how do we prove on <em>Mythos time*</em> that it reached everywhere.</blockquote><p>These questions are <em>asking what the leaks destroyed</em>: the first two need the fan-out map (leak one), the third needs a record of who changed what (leak two).</p><p><em>(*) Not by Monday any more! the same Mythos class of models that will help you patch is already helping someone else find what to exploit, so the gap between disclosure and weaponization is collapsing from weeks toward hours.</em></p><h3>The tools, one by one</h3><p><strong>Rendered manifests, hydrators, and of course: Gitless GitOps.</strong> The furthest the ecosystem has already gone, and if you run this pattern, keep running it.</p><p><em>Implements:</em> the <em>generative</em> principles that rendering should be done once.</p><p><em>Where the checks run:</em> at publish time, which is the right time, on the wrong object. Signing verifies the blob: what arrives is byte-for-byte what CI produced. It cannot check what changed inside, whose values moved, what they fan out to, whether the author was allowed, because the render that knew those things has already run and discarded them, and an OCI artifact can always be pulled and inspected, one blob at a time. What it cannot then answer is semantic questions: whose values moved, what they fan out to, where this artifact landed, without unpacking everything everywhere.</p><p><em>4pm on Friday:</em> the signatures help with question three, but the answers live in branches and blobs, so question one is still a script.</p><p><em>Note about Gitless: this splits the truth in two, humans authoring in git while controllers trust the registry, connected by a CI tool whose write discipline keeps them causally consistent. A risk factor is eg. two agents operating on resources with hidden overlaps between names, spaces and scopes. (The registry itself still earns its keep as a transport gate for Argo and Flux; the implementations are widespread and high quality.)</em></p><p><strong>CI and pipelines.</strong> The quote at the top of this post is about this.</p><p><em>Implements:</em> the mechanics of Generated, run on a schedule.</p><p><em>Where the checks run:</em> in the pipeline, before apply, which is the right time, but a pipeline checks its inputs and then vanishes. CI is a workflow, not a store: no memory between runs, no edges, no picture of the fleet. It produces the rendered result and forgets it, which is leak number one, run as a managed service. Any check that needs last week’s render, or the neighbouring environment’s, has nowhere to stand.</p><p><em>On Friday:</em> CI can rebuild anything and answer nothing; all three questions come back as scripts someone has to write.</p><p><strong>Helm diff.</strong></p><p><em>Implements:</em> a slice of the bounded principle, one target at a time.</p><p><em>Where the checks run:</em> before apply, which is correct, but limited to what one release can view. Diff compares a release against a target. Which of fifty environments inherit the value you are changing, and which overrode it on purpose, is not visible from inside any single release, because that relationship was discarded at render.</p><p><em>4pm on Friday:</em> question two is answered for the release in front of you; questions one and three have no answer.</p><p><strong>Argo and Flux previews.</strong> Both major reconcilers already compute a result before the apply step: Argo’s server-side diff sends a dry-run server-side apply to the Kubernetes API and compares the prediction against live state; Flux does dry-run and diff through the kustomize-controller and flux diff.</p><p><em>Implements:</em> the <em>Continuously reconciled, both ways</em> principle, plus the ‘local preview half’ of <em>Proven</em>.</p><p><em>Where the checks run:</em> before apply, per cluster, exactly where a reconciler should check. The local preview answers “<em>what changes on this cluster, against this cluster’s live state,”</em> and not “<em>which of these forty environments inherit this value, and which have overridden it.</em>” That second question needs a cross-environment data structure that understands variants and relationships.</p><p>We could try “just run the dry-run across all the clusters and collect the results,” but even joined up, this cannot tell a deliberate override from an accidental divergence, this cannot tell you which environments should have changed and did not, and cannot gate a promotion against the relationship, because the relationship that “these forty are instances of one thing” was never in the per-cluster view to begin with. A dry-run tells you what would change on a box. A graph of variant configurations tells you what is meant to be the same across boxes, and therefore what a change to the shared value actually means.</p><p><em>On Friday:</em> question two, answered per cluster; questions one and three not at all.</p><p><strong>OPA, Kyverno, and admission policies.</strong></p><p><em>Implements:</em> the validity half of the <em>Proven principle</em>, carried over unchanged: schema and policy. Making good use of your platform team’s rules.</p><p><em>Where the checks run:</em> before a change takes effect, in CI or at admission. This is good, but limited to local not fleet concerns. A policy engine evaluates the document you hand it, and the facts the governance half needs are not in the document. These might include the author’s declared scope, the ownership of each field, the fan-out of the change. Also therefore, it cannot answer across the fleet and be sure that a change stayed inside its declared blast radius, and cannot confirm the author had authority. To check one change against rules is not to query the fleet estate. (Exercise for the reader: try writing a crawler to report on policy changes across a non-homogeneous fleet. Not a simple task.)</p><p><em>On Friday:</em> it guards each change on the way in, and can ask the fleet nothing, so questions one and three go unanswered.</p><p><strong>Kubernetes RBAC and Server-Side Apply.</strong></p><p><em>Implements:</em> the Attributed principle, in a limited style where managed fields record which client touched which field.</p><p><em>Where the checks run:</em> at admission, the right place, with the wrong identity, and the ledger is written after ‘apply’, per cluster. The credential presenting a change belongs to the GitOps tool, whether it inherits Kubernetes RBAC like Flux or runs its own like Argo, so the actual author is invisible to the cluster and the managed-fields record mostly names the reconciler. And RBAC is all-or-nothing per resource: anyone who can update a Deployment can change any field in it. Deciding who may change one particular value has to happen where the desired state lives, outside the cluster. (Subresources like /status can help, but are tricky to use.) Right primitive, wrong place and time: the model relocates that record to the store and gives it what the cluster never keeps: history; this is written before apply, across the fleet, so it can be checked rather than merely read after an incident.</p><p><em>4pm on Friday:</em> 4pm on Friday: attribution exists, but split across three systems, git knows an author, Argo knows who synced, the cluster knows the reconciler, and none of them is joined to the field that changed. Question three’s audit is a join that nobody owns.</p><p><strong>Argo Rollouts and canary deployments.</strong></p><p><em>Implements:</em> the Bounded principle, in time.</p><p><em>Where the checks run:</em> during the rollout, after apply has begun, watching one target. That is the right design for its job: bound the blast radius in time, roll out gradually, watch, stop. What has no check anywhere is the other dimension: which of fifty environments does this change touch at all, and which are exempt. A canary bounds blast radius in time; the graph bounds it in space. Complementary axes, not competitors, and Bounded needs both. Keep your rollout strategy; the graph tells it where it is allowed to run.</p><p><em>On Friday:</em> each cluster’s image bump gets safer, and nothing says which clusters, or proves the set is complete.</p><p><strong>ResourceQuota dry runs.</strong></p><p><em>Implements:</em> an ops fact, read by hand.</p><p><em>Where the checks run:</em> at admission, per namespace, the latest possible moment. The signal is real, the prod-ap catch from Part 2’s example is exactly this signal doing its job, but read at apply it is a stalled rollout someone notices an hour later. Read before apply, as a fact in the graph, it is an answer that comes back with the change plan. Same check, moved left and widened.</p><p><em>On Friday:</em> question two, one namespace at a time, and only if someone remembers to run it.</p><p><strong>Secrets tooling.</strong></p><p><em>Implements:</em> Held as data, for the one field class that must never render: the reference is typed data, the value deliberately late-bound.</p><p><em>Where the checks run:</em> first, note these are two different models. Sealed Secrets and SOPS put the encrypted value itself in git; External Secrets keeps only a reference in git and resolves the value inside the cluster. Either way, resolution happens at the cluster, which for secrets is correct by design: the value must not shift left, ever. And inside one cluster, the obvious question is answerable: “which workloads reference this secret” is a kubectl query away.</p><p>The missing check is the fleet half, and it has nowhere to run: across the whole estate, which workloads reference this key, what breaks when I rotate it, and did every one of them actually pick up the new version. In the model, the value still never enters the store; the reference does, as typed data with an owner and an expected version by digest, so rotation becomes a ‘governed’ operation; then <em>who references the key</em> is a query across the fleet, <em>the fan-out is bounded before it ships</em>, and receipts <em>prove the new version was picked up everywhere</em>, by digest, without the value ever crossing the store. Proof without disclosure ;-)</p><p><em>On Friday:</em> rotation is the Friday test with a key instead of an image, and today it is the same weekend.</p><p><strong>Sveltos.</strong> Teams adopt it because they already think in fleets. Profiles select clusters by label and deploy add-ons in a fan-out style to cluster resources that match.</p><p><em>Implements:</em> the fan-out half of the graph, as cluster selection.</p><p><em>Checks:</em> a profile matching a label set is a one-to-many relationship made explicit. That is a thing most of this toolbox lacks. Selection is resolved before deployment: the right time. Sveltos then goes further than most here: DryRun mode produces ClusterReports, a genuine per-cluster preview of what a profile change would do before it does it, and ContinuousWithDriftDetection watches what it deployed and reconciles it back. Preview before, watch after: two checks in the right places. Then, the reports are about <em>per cluster</em> and <em>per profile</em>, not per change. There is no receipt tying an individual change to its author and its provenance, no view of who moved which value inside the profile’s payload, and drift detection restores the deployed state rather than asking whether the live change was authorized, so there is no governed reverse path that folds a legitimate live fix back into desired state. Sveltos ‘knows fan-out’ better than anything else on this list; what it does not carry is the per-change story travelling along it.</p><h3>We need a data substrate to retain information across tools</h3><p>Could we build a ‘best of breed stack’ to implement the six updated principles? Almost. The tools have information gaps (as described above) and are missing some kind of substrate to glue it all into one consistent system. We can see this if we map back to the <a href="https://confighub.com/blog/gitops-in-the-age-of-ai-2-48e13026f434">six principles</a>.</p><ul><li><strong>Generated</strong>. We can render-once in CI but the render lands as a <em>commit</em>, not as addressable data validated on entry. The next pipeline will re-render over it.</li><li><strong>Held as Data.</strong> A repo of rendered YAML manifest is ‘held as data’ in the same way that a folder of CSVs is a database: all of the content, none of the semantics. No edges, no variants, no write path with checks. You can read it. You cannot operate on it.</li><li><strong>Attributed.</strong> We could use git blame and managed fields. When the committer of record is the CI bot, blame answers “which pipeline ran” not “who.” Managed fields record a field-level manager (per cluster after apply) and the manager is usually the reconciler, because the tool’s credential masks the author. We end up with <em>post-hoc and per-cluster</em> information, but we need <em>pre-apply and fleet-wide</em>.</li><li><strong>Proven.</strong> Use OPA/Kyverno? This provides validity, schema and policy. Existing rules may be reused. The gap is <em>fleet governance</em>. A policy evaluates documents, for wider checks the required inputs are relationships that are not in that document: the author’s declared scope, the ownership of each field, the fan-out of the change. We also need <em>transitions</em>: was this change allowed, by this author, within these bounds. You cannot write that rule in Rego: the inputs live in the resource <em>graph</em>, not in the resources.</li><li><strong>Bounded.</strong> A helm diff shows the change on a target you pointed it at. Bounded is the other way round: it tells you which targets to point at, and which will refuse on purpose, and that needs the config relationships, which no diff holds.</li><li><strong>Continuously reconciled, both ways</strong> The reverse loop can be performed by hand, with no gate. It works until two agents do it at once ‘before the PR merges’. Doing both safely needs authority on the live write, ordering against the revert, and a receipt either way.</li></ul><h3>AI Applications operate on live systems</h3><p>A data substrate could capture and retain operational information. This especially matters if we have AI agents operating directly on live systems.</p><p>Think of the classic “Kubernetes dashboard”. According to the very early versions of GitOps, you should not use this to make direct updates to the system. Even if updates are needed to fix operational problems quickly, they may conflict with the single source of truth (desired state) represented in Git.</p><blockquote>Well, OK. But: an autoscaler, a controller default, and an agent’s security fix are all “the live system changed itself,”. Traditional GitOps gives you a menu for drift: revert it (selfHeal), flag it and wait (OutOfSync), or ignore the field entirely (ignoreDifferences, field management).</blockquote><p>We are going to need this, and more, if we want AI agents to help us operate safely and at scale. We just need to be careful: <em>Revert erases the context; ignore waves it through unattributed; flag leaves a human to reconstruct it. When all drift is wiped, who keeps the information? </em>That is ‘leak number two’, the mirror of the first: where generators like Helm can wipe critical config related info, the reconciler can erase facts on the way back.</p><p>The missing contribution here is not detecting drift, it is refusing to destroy what it tells you: attribute it, resolve it by authority, and either fold it back or revert it with a receipt. <em>When the live system changes on its own, do we know who or what changed it, whether they were allowed to, and can we decide per case whether it flows back or is reverted.</em></p><p>In our <a href="https://confighub.com/blog/gitops-in-the-age-of-ai-2-48e13026f434">Generative GitOps model</a>, live updates count as operational facts, and may record who changed a value or why. AI agents will be making decisions about live state ‘facts’ and acting on them. Already we see this with “FinOps” tools in AI platforms. None of this is solved without a suitable operational store that tracks these changes.</p><blockquote>This is another area where Git alone (or Git plus OCI) is not a very good fit. Note what is new there: the AI agent is the first actor that writes to all three surfaces, source, desired state, and the live system, at machine speed, which stretches the traditional GitOps loop.</blockquote><h3>Applications on an Operational Data Store</h3><p>In the new model, we are using AI agents to modify the whole system, aka Jobs to be Done:</p><blockquote>Preview a change</blockquote><blockquote>Gate a release</blockquote><blockquote>Check a policy</blockquote><blockquote>Verify a rollout</blockquote><blockquote>Rotate a secret</blockquote><blockquote>Update the dashboard</blockquote><p>These are applications that manage config and operational facts on a database (‘the graph’). They are no longer complex “workflows” with transactions, state, events and ordering.</p><p><em>Before, GitOps was how we managed K8s clusters and workloads. Applications were something else we could deploy after clusters and services, on top, maybe with Argo UI. In the future there will be one, simplified model for the whole stack: apps, services and infra.</em></p><p>Run the 4pm Friday weekend story again, but imagine we have a data graph of config and facts.</p><p><em>Which deployments run the image?</em> Your scanner has an inventory and asset databases and cluster queries will list where the image runs. What none of them can tell you is the other half: which configuration, owned by whom, produces each of those deployments, and through what path you would change it. On the graph that already exists, the image-to-deployment- to-config-to-owner chain is data, so the answer comes back ready to act on, not just to read.</p><p><em>What breaks when we bump it? </em>Blast radius computed along the graph edges before apply, deliberate overrides surfaced with their owners, quota facts read from the fleet.</p><p><em>Prove it landed? </em>Receipts, one per target, aggregated into the report the CISO actually wants.</p><p><strong><em>All completed in Mythos time, and signed off by the CISO before we go home on Friday.</em></strong></p><p>Because all we needed was an app. The fleet-change job, written as a short program over the graph. HOLD that thought. In the past, each job grew its own tool, its own product category even, because files gave it nothing to build on, so every job rebuilt its own storage, its own fleet awareness, its own audit trail, badly or not at all. Its own dashboard. Put the operational desired state graph underneath and each job stops being a product category and becomes an app: a small program over shared primitives, the edges, the gates, the facts, the receipts. That is what compose means here, concretely. The same Jobs To Be Done are now an app catalog.</p><p>Keep your rendered manifests, managed fields and validity rules. Keep Git, keep Argo and Flux, keep the policies and the rollout strategies. What you add is the graph store underneath them, and the store is what lets them finally compose so the jobs become apps over the store.</p><p>For decades the missing IT component was a human. The engineer who remembered which region overrides memory. The one who knew why that field drifted last March. The one with a spare afternoon to go and check. The information the tools threw away survived in people. That is what “it worked fine” meant: a person was quietly plugging the gaps. Agents end that. The fix is not: <em>a better human</em>. The information has to move from heads into the operational graph.</p><p><em>The model: config is data, both-ways reconciliation, and agentic proof of change.</em></p><p>– alexis</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HoNbgrRFaNfDZs29GstB0g.jpeg" /><figcaption>SP80 at 108 km/h</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d80429b2b24" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>