<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jeroen Soeters on Medium]]></title>
        <description><![CDATA[Stories by Jeroen Soeters on Medium]]></description>
        <link>https://medium.com/@jeroensoeters?source=rss-145657d24429------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*rXiGYfvfBppYjF_d</url>
            <title>Stories by Jeroen Soeters on Medium</title>
            <link>https://medium.com/@jeroensoeters?source=rss-145657d24429------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 20:10:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jeroensoeters/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Towards the dark factory: Autonomous, isolated AI agents in software engineering]]></title>
            <link>https://blog.platform.engineering/towards-the-dark-factory-autonomous-isolated-ai-agents-in-software-engineering-81f37aea0e24?source=rss-145657d24429------2</link>
            <guid isPermaLink="false">https://medium.com/p/81f37aea0e24</guid>
            <category><![CDATA[formae]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[dark-factories]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Jeroen Soeters]]></dc:creator>
            <pubDate>Thu, 30 Jul 2026 07:14:02 GMT</pubDate>
            <atom:updated>2026-07-30T07:14:02.257Z</atom:updated>
            <content:encoded><![CDATA[<p>How we built “clanker infrastructure” — autonomous, network-isolated AI agents that pick up tickets, design, implement, and ship PRs through a locked-down container factory provisioned with <strong>formae</strong> itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VGqg71jLBpiF7TIa.jpg" /></figure><p>Late 2025 was an inflection point for AI in software development. Before that I spent most of my time writing code in an editor (vim of course, as God intended). I used AI mostly for auto-complete or generating small patches with the awesome <a href="https://github.com/yetone/avante.nvim">avante</a> plugin. After the release of Opus 4.5 the workflow changed completely: I switched to Claude Code and never looked back.</p><h3>The human loop</h3><p>On a high level my workflow looks something like this. I pick up a new task and kick off a brainstorming session with Claude, using the <a href="https://github.com/obra/superpowers">superpowers</a> plugin. Claude writes a design spec, and I validate it across models by having Codex run an adversarial review of the design. I triage the findings and ask Claude to incorporate the ones worth acting on, and if the rework is substantive we rinse and repeat. Only then do I review the full spec myself. Once it holds up, Claude writes the implementation plan, and depending on complexity I either delegate the work to subagents or stay in the current session. Finished code goes through another cross-model review before I look at it myself, and then I either squash-merge the PR or send the agent back to fix what’s left.</p><p>This flow works well for substantive features that benefit from an interactive design session. But at a startup with a small team touching many different technologies, which an IaC tool does by definition, a lot of the work is death by a thousand cuts. Many tickets are tiny: change the severity of a log line, fix the JSON parsing of resource X, remove some temporary migration code. The design and implementation of these is trivial, and stepping through it with a coding agent feels mechanical, not something that should need a human.</p><p>Over the last few weeks I set aside a portion of my time to tackle this, and the result is what we now call our <strong>clanker infrastructure</strong>. Clankers are autonomous agents that do exactly the mechanical steps outlined above.</p><h3>Building the machine that builds the machine</h3><p>A clanker is built out of the same tools I use every day: Claude Code as the harness, a curated set of skills, various MCP servers, Codex for cross-model reviews. But instead of me orchestrating from the harness, clankers run on their own. They are triggered from our issue tracker and run headless in a container we control.</p><p>Every time a clanker does work we call this a turn. Proposing a design is a turn, opening a PR is a turn, answering a question is a turn. After each turn the clanker hands the baton back to a human. The container is disposable and nothing persists between turns; everything durable lives in our issue tracker and on the PR branch.</p><h3>The control plane</h3><p>We use <a href="https://linear.app/">Linear</a> as our issue tracker. Linear is also the surface for our autonomous agent infrastructure. One of its features lets you provision <em>app users</em> through an OAuth app: workspace members that are @mentionable and assignable like anyone else, but backed by an app instead of a person. “Clanker” is one of these, and you hand it work exactly the way you’d hand work to a teammate, by assigning a ticket to it. Behind that single name is a fleet: every turn runs in its own fresh clanker, and many can be working in parallel.</p><p>This is what our Linear board looks like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Xx2JLxw4Nu5hccP4.jpg" /></figure><p>When a ticket gets picked up from the backlog it first goes to “Design,” where the technical design gets worked out and a tech brief is produced and attached to the Linear issue. Only once that design is approved does the ticket move to “Implementation,” where the actual coding happens. Once the PR is up the ticket lands “In Review,” and once merged it moves to “Done.”</p><p>The column dictates <em>what</em> work Clanker will do. Assign Clanker in the “Design” column and it produces the tech brief; any blockers or decisions get escalated to the ticket owner, with Clanker commenting on the ticket and unassigning itself. Assign it in the “Implementation” column and it writes an implementation plan, delegates the coding to subagents, and opens a PR. From there you can also interact with Clanker through the PR, not just in Linear: request changes and it revises, approve and it merges the PR and moves the ticket to “Done.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*lFxisZz7_nsk18_bMFXPyQ.jpeg" /></figure><p>Another cool feature of Linear is agent sessions. Whenever an agent picks up work, it streams its thoughts and actions into a read-only thread on the issue, so you get a window into what’s happening inside the session.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hbXBDLroC-mropav.jpg" /></figure><h3>The Agent Bridge</h3><p>Between Linear and the agents lives a small always-on orchestration service we call the Agent Bridge. It is an ECS Fargate service, and it is what turns activity on a ticket into a running agent. When you assign Clanker to a ticket, or later interact with it on its PR, that event reaches the Agent Bridge, which figures out what kind of turn to run and spawns a fresh container to run it.</p><p>Before it does, it mints a short-lived GitHub token scoped to just the repositories that turn needs, so the agent can push a branch and open a PR without ever holding a broad, long-lived credential. When the container is done it hands the baton back and exits.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*p7l6VeSflP1YI0lgqM4JHg.png" /></figure><p>Each turn runs in its own container, spun up from a Docker image tailored to the Linear project the ticket belongs to, so the agent comes up with exactly what that turn needs and nothing more. Every container starts from a shared foundation: a global operating constitution, a CLAUDE.md that governs every turn in every project. It covers things like the turn model: one turn at a time, the auto-review pass, and the human gates it must never cross on its own. It also covers engineering practices: test-first development, matching the conventions of the surrounding code, YAGNI, and the like. And orientation: where our public and internal docs live and where to find things across our GitHub org. On top of that global base, each project layers in what <em>it</em> needs:</p><ul><li>the <strong>GitHub repo or repos</strong> associated with the Linear project;</li><li>a <strong>“local” </strong><strong>CLAUDE.md</strong> with repo-specific conventions, baked into the image;</li><li>the <strong>tools</strong> the work requires, like the AWS CLI for the AWS plugin or Deno for our public hub, which is built on the <a href="https://fresh.deno.dev/">Fresh</a> framework;</li><li>the <strong>access</strong> the container needs, such as credentials for a dedicated test CI account so the agent can debug failing tests there;</li><li>the <strong>MCP servers</strong> scoped to that project.</li></ul><p>The container arrives with exactly what it needs to do the job and nothing it doesn’t. A <strong>formae</strong> core ticket, a <strong>formae</strong> hub ticket, and an AWS plugin ticket each get a different image, the right tooling, and the right context, selected automatically.</p><p>These containers can potentially feed attacker-influenceable text, like issue descriptions and PR comments, through an LLM that holds live credentials. That is a prompt-injection risk, so the boundary is locked down. The containers have no route to the open internet. They sit in isolated subnets, and every outbound request goes through a firewall with an explicit FQDN allowlist: the handful of endpoints the work genuinely needs (GitHub, the Anthropic and Codex APIs, Linear, the Go module proxy, AWS) and nothing else. Everything else is dropped and logged. Credentials are per-turn and least-privilege: that scoped GitHub token, a minimal IAM role, and secrets that only ever come from Secrets Manager.</p><blockquote><em>A side effect of cutting off the open internet: a clanker is useless for research tickets. We’re fine with that. The clankers exist to execute well-specified work, not to wander an open solution space.</em></blockquote><h3>Cross-model reviews</h3><p>A model tends to be blind to the weaknesses in its own output, so before a design or a diff is accepted, a clanker hands it to a <em>different</em> model for an adversarial review, the same on a design turn as on an implementation turn. We use Codex for this today. The findings come back, the clanker triages them, incorporates the ones worth acting on and dismisses the rest. If the resulting change was substantial, it reviews again, and keeps going until the work comes back clean. All of this runs through a custom skill we call auto-review.</p><p>We put a lot of effort into the triage step. Codex flags everything that could theoretically be a problem, whether or not it matters here, so its findings need weighing rather than blind trust. The skill carries the context to do that. It takes into account the guarantees the codebase already provides, for example that <strong>formae</strong> and its plugins are built on the <a href="https://jeroensoeters.github.io/unlocking-concurrency-in-go/">actor model</a>, so a flagged race or shared-state hazard that can’t occur under the actor model gets discounted. It verifies whether a feature is even live yet, so backwards-compatibility and migration findings get set aside when there are no users or data to break. The reviewer surfaces everything; the skill decides what actually matters here.</p><p>Correctness and security are the exception. A finding there can’t be waved away with a bare “rejected”; it clears only with a real fix, a concrete argument that it doesn’t apply, or a human’s sign-off. The same clanker wrote the code and decides which findings to honor, so it can’t be allowed to quietly bury a serious one.</p><h3>Clankers all the way down</h3><p>With all of this in place, I kept finding myself in the same situation. I’d be deep in a Claude Code session on some feature, for example a customer demo, and realize I needed a small change somewhere else, support for a new resource type, or a bugfix in a common library. So I’d ask Claude to file the ticket (Linear has an excellent MCP, which we’ve wrapped in a set of custom skills), assign Clanker, review the design it produced, move it to “Implementation,” assign Clanker again, and approve the PR once I was confident it solved the problem.</p><p>That loop was itself mechanical, so it quickly became its own skill. clank-through takes an existing ticket and oversees a clanker end to end, walking it across the board and escalating to me only when something genuinely needs a decision. So there we are: agents delegating work to agents delegating work to agents 🤖.</p><h3>The dark factory</h3><p>This is our factory as it stands: work moves through the pipeline on its own, and I only step in when a clanker is genuinely blocked.</p><p>The type of work we hand to clankers isn’t really limited by size. A clanker can take on quite complex tasks, as long as the product outcome is clear (a clanker should never be making product decisions) and the technical design is either obvious, not important, or close enough to something that already exists for the clanker to pattern-match against. Beyond that you hit diminishing returns: working out a tricky design through issue comments is painful, and much easier in an interactive session. Once the design is settled I attach it to the issue and hand the implementation to a clanker, rather than driving it myself in Claude Code.</p><p>Concretely, my mental model of the dark factory is two production lines: product specification → technical design and technical design → working software. Work can be injected into either one. Where I inject it depends on whether I care about the code that will be produced and on the complexity of the design. Complexity here assumes no prior art: something can be highly complex, but if it has been done a million times before it is trivial for an LLM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*JAgZxrDIPjezx4g5ShHo6Q.png" /></figure><p>A true dark factory would run both lines lights-out: product spec in, working software out, across all four quadrants. For a complex, highly concurrent and distributed system like the one we are building, however, I don’t see this being realistic yet.</p><p>Of course, the entire clanker stack (the Agent Bridge, the containers it spawns, the firewall that boxes them in) is declared in Pkl and provisioned with <strong>formae</strong> itself on our own production AWS estate. The tool provisions the factory that builds the tool. And yes, <a href="https://github.com/platform-engineering-labs/formae"><strong>formae</strong> is open-source</a>, if you want to try it out for yourself.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=81f37aea0e24" width="1" height="1" alt=""><hr><p><a href="https://blog.platform.engineering/towards-the-dark-factory-autonomous-isolated-ai-agents-in-software-engineering-81f37aea0e24">Towards the dark factory: Autonomous, isolated AI agents in software engineering</a> was originally published in <a href="https://blog.platform.engineering/">Platform Engineering Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Targets, resources, resolvables and stacks: the formae infrastructure graph explained]]></title>
            <link>https://blog.platform.engineering/targets-resources-resolvables-and-stacks-the-formae-infrastructure-graph-explained-2d1ce0944bb1?source=rss-145657d24429------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d1ce0944bb1</guid>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[formae]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <dc:creator><![CDATA[Jeroen Soeters]]></dc:creator>
            <pubDate>Wed, 10 Jun 2026 08:33:07 GMT</pubDate>
            <atom:updated>2026-06-10T08:33:07.097Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8GmcBt5DI_kYBpRyS4iM-g.png" /></figure><p><strong>formae</strong> is built from a small set of composable concepts, and their power is in how they combine. The goal of this article is to make it easier to see the overall structure of a <strong>formae</strong> codebase.</p><p>In infrastructure codebases, the problems that matter most tend to surface at the joins: the points where resources managed by one plugin depend on resources managed by another. These cross-plugin dependencies are the ones mainstream IaC tools handle worst, pushing you toward separate states, separate apply runs, or a dedicated mechanism just to carry a value across the boundary. <strong>formae</strong> models all of it as a single graph.</p><p>That graph is built from just four concepts: resources, targets, resolvables, and stacks. We will look at each on its own first.</p><h3>Resources</h3><p>A resource is a single piece of infrastructure: an S3 bucket, a VPC, an ECS service, a Grafana dashboard. Each resource has a type, like AWS::S3::Bucket or GRAFANA::Core::Dashboard, and a set of properties that describe the state you want: a bucket has a name, a VPC has a CIDR block, a dashboard has a title and a folder.</p><h3>Targets</h3><p>A resource never exists in the abstract; it lives somewhere, and that somewhere is a <em>target</em>. A target is the connection <strong>formae</strong> uses to reach a place and act on it: to create and manage resources there, and read their state back. Every resource belongs to exactly one target. A target you mark discoverable is also one <strong>formae</strong> scans for resources that already exist.</p><p>A target has a label and a config that holds whatever <strong>formae</strong> needs to talk to that place.</p><p>What goes in the config depends on the plugin. An AWS target needs a region, and a Grafana target needs the URL of the Grafana instance. A Kubernetes target needs more: an auth strategy that knows how to reach and authenticate to the cluster, anything from a local kubeconfig to cloud-native auth for EKS, GKE, or AKS.</p><h3>Resolvables</h3><p>Most resources need to know something about another resource before they can be created. A subnet needs the ID of the VPC it sits in; an ECS service needs the ARN of its task definition. The catch is that those values do not exist when you write your Pkl. The VPC has no ID until AWS creates it, so there is nothing to paste in.</p><p>A <em>resolvable</em> lets you refer to the value anyway. Instead of a literal, you point at the resource and the property you want, written as vpc.res.vpcId: the id of the vpc resource, whatever it turns out to be. The .res accessor reaches into a resource&#39;s applied state, the version that exists once <strong>formae</strong> has created it.</p><p>Two things follow. First, <strong>formae</strong> resolves the value late, at apply time, not when you write the code: it creates the VPC, reads back the ID that was assigned, and only then creates the subnet with that ID in hand. Second, the reference tells <strong>formae</strong> the order to work in. Because the subnet points at the VPC, <strong>formae</strong> knows the VPC comes first, and that resources which do not point at each other can be created in parallel. You write down what needs what, and the order to build things in falls out of that.</p><p>The underlying algorithm that works out this ordering is more complicated than this, but we will cover this in a future article.</p><h3>Stacks</h3><p>A stack is a named group of related resources. Every resource belongs to exactly one stack, just as it belongs to exactly one target.</p><p>The grouping matters when you apply. Running formae apply --mode reconcile makes each stack in your Pkl match its definition exactly: a resource in your Pkl that does not exist yet is created, a resource that exists but is no longer in your Pkl is deleted, and any other difference becomes an update or a replace, depending on the property.</p><p>This is what makes a stack the scope of a reconcile. <strong>formae</strong> will create, change, and delete to make the running infrastructure match your Pkl, but only within the stacks you give it, so the way you group resources into stacks decides what a single reconcile can change or delete.</p><p>If you want to automatically clean up a set of resources, you set a time-to-live policy on a stack. If you want to automatically undo out-of-band changes, you set an auto-reconcile policy on a stack. Policies like these are set per stack, so they are another thing to weigh when designing stacks.</p><h3>Putting it together</h3><p>At first, this was the whole picture: pick a cloud, declare some resources, wire them together with resolvables, and group them into a stack. Here is a small AWS example: a network, and an EKS cluster to run workloads on.</p><pre>amends &quot;@formae/forma.pkl&quot;<br><br>import &quot;@formae/formae.pkl&quot;<br>import &quot;@aws/aws.pkl&quot;<br>import &quot;@aws/ec2/vpc.pkl&quot;<br>import &quot;@aws/ec2/subnet.pkl&quot;<br>import &quot;@aws/eks/ekscluster.pkl&quot;<br>local mainVpc = new vpc.VPC {<br>    label = &quot;main-vpc&quot;<br>    cidrBlock = &quot;10.0.0.0/16&quot;<br>    enableDnsHostnames = true<br>}<br>local publicSubnet = new subnet.Subnet {<br>    label = &quot;public-subnet&quot;<br>    vpcId = mainVpc.res.vpcId<br>    cidrBlock = &quot;10.0.1.0/24&quot;<br>    availabilityZone = &quot;us-east-1a&quot;<br>}<br>local appCluster = new ekscluster.Cluster {<br>    label = &quot;app-cluster&quot;<br>    name = &quot;app-cluster&quot;<br>    version = &quot;1.31&quot;<br>    resourcesVpcConfig = new ekscluster.ResourcesVpcConfig {<br>        subnetIds { publicSubnet.res.subnetId }<br>    }<br>    // a real cluster also needs an IAM role, security groups, and subnets<br>    // across two AZs; trimmed here to keep the example readable<br>}<br>forma {<br>    new formae.Stack { label = &quot;platform&quot; }<br>    new formae.Target {<br>        label = &quot;aws&quot;<br>        config = new aws.Config { region = &quot;us-east-1&quot; }<br>    }<br>    mainVpc<br>    publicSubnet<br>    appCluster<br>}</pre><p>That Pkl already describes a graph. Drawn out, it is a short chain:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/969/1*y8RRQBOxVbjMY7ydND39IA.png" /><figcaption>AWS / EKS stack: one target, resources resolving into each other</figcaption></figure><p>The target says where everything goes, an AWS account in us-east-1. If you only declare a single target, every resource automatically lands in it. The VPC, subnet, and cluster are the resources. The resolvables are the wiring: the subnet takes mainVpc.res.vpcId and the cluster takes publicSubnet.res.subnetId, so <strong>formae</strong> builds the VPC first, then the subnet, then the cluster. All three belong to one stack, platform, the unit you would reconcile or tear down as a whole.</p><p>When we first shipped <strong>formae</strong>, the only plugins were for cloud providers, so this is what most setups looked like: a target at the root with resources hanging off it. You could provision the EKS cluster, but anything you ran on it was beyond <strong>formae</strong>’s reach. As we added plugins for more than cloud providers, <strong>formae</strong> could manage what runs on the cluster too.</p><h3>Targets inside the graph</h3><p>In every example so far the target sat at the root. But look again at what a target is. It has a config, and a config is just data. We have already seen that data does not have to be a literal you write by hand; it can be a resolvable. So nothing stops a target&#39;s config from reading its values out of another resource.</p><p>This is where composition starts to shine: a target’s connection is built from a resource that lives in another target. Here is the Kubernetes target for the cluster from the previous section:</p><pre>import &quot;@k8s/k8s.pkl&quot; as k8s<br><br>local appK8s = new formae.Target {<br>    label = &quot;app-k8s&quot;<br>    config = new k8s.Config {<br>        kubernetesVersion = &quot;1.31&quot;<br>        auth = new k8s.EKSAuth {<br>            endpoint = appCluster.res.endpoint<br>            certificateAuthority = appCluster.res.certificateAuthorityData<br>            clusterName = appCluster.res.name<br>        }<br>    }<br>}</pre><p>appCluster is the EKS cluster from the AWS target. Its endpoint, CA, and name do not exist until <strong>formae</strong> has created it, so the Kubernetes target reads them through .res, exactly the way the subnet read the VPC id.</p><p>Because that config reads from the cluster, the ordering rule from the resolvables section now applies to the target as well: <strong>formae</strong> brings the Kubernetes target up only after the cluster it points at exists.</p><p>With the target defined, Kubernetes resources live inside it the same way the VPC and subnet live inside the AWS target. Here we deploy Grafana onto the cluster: a namespace, and a LoadBalancer Service in front of it.</p><pre>import &quot;@k8s/v1.31/core/Namespace.pkl&quot; as ns<br>import &quot;@k8s/v1.31/core/Service.pkl&quot; as svc<br><br>local appNs = new ns.Namespace {<br>    label = &quot;observability&quot;<br>    target = appK8s.res<br>    metadata = new ns.NamespaceMetadata { name = &quot;observability&quot; }<br>}<br>local grafanaSvc = new svc.Service {<br>    label = &quot;grafana&quot;<br>    target = appK8s.res<br>    metadata = new k8s.NamespacedObjectMeta {<br>        name = &quot;grafana&quot;<br>        namespace = appNs.res.name<br>    }<br>    // a LoadBalancer Service fronting the Grafana pods; spec omitted for brevity<br>}</pre><p>With more than one target in play, each resource has to say which one it belongs to: both set target = appK8s.res, so they land in the Kubernetes target rather than the AWS one. And the Service reads its namespace from appNs through the same .res mechanism.</p><p>That is the graph two targets deep: an AWS target with a cluster inside it, and a Kubernetes target hanging off the cluster with resources of its own.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a3K4NxmXoNLYVW3zfS_ggg.png" /><figcaption>Two targets: the Kubernetes target reads the EKS cluster</figcaption></figure><p>The recursion does not stop here. To create Grafana’s own resources, its folders and dashboards, we need a Grafana target. That target needs a URL: where the running Grafana can be reached. The LoadBalancer Service we just deployed has one, so the Grafana target reads it the same way the Kubernetes target read the cluster:</p><pre>import &quot;@grafana/grafana.pkl&quot;<br><br>local grafanaTarget = new formae.Target {<br>    label = &quot;grafana&quot;<br>    namespace = &quot;GRAFANA&quot;<br>    config = new grafana.Config {<br>        url = grafanaSvc.res.lbIngressUrl<br>    }<br>}</pre><p>Grafana’s own resources, a folder and a couple of dashboards, live in that target:</p><pre>import &quot;@grafana/core/folder.pkl&quot;<br>import &quot;@grafana/core/dashboard.pkl&quot;<br><br>local dashboards = new folder.Folder {<br>    target = grafanaTarget.res<br>    label = &quot;app-dashboards&quot;<br>    uid = &quot;app-dashboards&quot;<br>    title = &quot;App Dashboards&quot;<br>}<br>local latency = new dashboard.Dashboard {<br>    target = grafanaTarget.res<br>    label = &quot;latency&quot;<br>    uid = &quot;latency&quot;<br>    title = &quot;Latency&quot;<br>    folderUid = dashboards.res.uid<br>    configJson = read(&quot;dashboards/latency.json&quot;).text<br>}<br>local errors = new dashboard.Dashboard {<br>    target = grafanaTarget.res<br>    label = &quot;errors&quot;<br>    uid = &quot;errors&quot;<br>    title = &quot;Errors&quot;<br>    folderUid = dashboards.res.uid<br>    configJson = read(&quot;dashboards/errors.json&quot;).text<br>}</pre><p>So the pattern repeats, and then it repeats again. An AWS target holds a cluster; a Kubernetes target connects to that cluster and runs Grafana; a Grafana target connects to Grafana and holds the dashboards. [target] → resources → [target] → resources → [target] → resources, as far down as your infrastructure goes. None of the primitives changed: the same Target, the same resources, the same .res. Three plugins compose in one place, and every connection between them is a resolvable instead of a value copied across a boundary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yam8ND2_z5eAjh_J3EMwYA.png" /><figcaption>Three targets composing in one graph; stacks cut across the nesting</figcaption></figure><p>Notice too that stacks are orthogonal to the nesting: the diagram keeps the AWS and Kubernetes resources in a single platform stack, but we could have split them however we wanted. A resolvable can cross a stack boundary just as it crosses a target boundary, so resources in different stacks can still reference each other. A stack groups resources for reconcile; it does not have to follow the target boundaries.</p><h3>Changing the graph</h3><p>Every time you run formae apply, <strong>formae</strong> compares your Pkl to the actual infrastructure and works out the necessary operations: what to create, update, replace, or destroy. The edges of the graph decide the order of these operations. For example, in Kubernetes you cannot update a Service&#39;s loadBalancerClass; you have to delete the Service and recreate it. <strong>formae</strong> understands this: it deletes the existing Grafana resources and the target first, then recreates them once the new Service is up. Changing a property that doesn&#39;t require the Service to be deleted and recreated, for example adding a label, will not result in any downstream operations.</p><h3>The same target, any substrate</h3><p>A nested target is only as useful as the connection it reads from. This is why we put real thought into those connections. Take Grafana: a Grafana target needs a URL for the Grafana HTTP API, so whatever runs Grafana has to expose something that resolves to one. What that is depends on where Grafana runs. On Kubernetes, the Grafana Service is a LoadBalancer and exposes an lbIngressUrl. On ECS behind a load balancer, the ECS Service publishes a set of endpoints keyed by container and port. On Docker Compose, the stack publishes endpoints the same way. Each is a different resource with a different output, and part of a plugin&#39;s job is to make what it exposes resolvable into the URL the Grafana target expects.</p><p>The payoff is reuse. The same Grafana target, with the same folders and dashboards inside it, can sit on any of them: wire its url to whatever the substrate exposes and nothing else changes. We saw the Kubernetes form in the previous section, reading grafanaSvc.res.lbIngressUrl. On a Docker Compose stack, the URL comes from the stack&#39;s endpoints instead:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*p-hrw7celPAAcRv3KSpZdQ.png" /><figcaption>The same Grafana target on a Docker Compose stack</figcaption></figure><p>And on ECS, from the service’s endpoints, with a load balancer and target groups doing the routing behind it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vacI0Ra8B6LU7OR1e-9JcA.png" /><figcaption>The same Grafana target on ECS behind a load balancer</figcaption></figure><p>In all three the Grafana target and its dashboards are identical. Where Grafana runs becomes a wiring decision rather than a rewrite.</p><h3>How other tools handle this</h3><p>That chain from the cluster to Kubernetes to Grafana crosses two plugin boundaries, and in <strong>formae</strong> it stays inside one graph because a target’s connection is data, and that data can be a resolvable that resolves at apply time. Most tools draw a line right there. Terraform does not treat providers as nodes in its plan graph, so a Kubernetes provider cannot read a cluster built in the same run, and the usual fix is to split into two states and two applies. Pulumi does better within one program, where an explicit provider can take a cluster’s output as input, but carrying that across stacks or projects means routing it through a StackReference. Crossplane stays declarative, but connecting to a cluster it provisions goes through a Kubernetes Secret rather than a direct reference. In each case the handoff from one plugin to the next is where the extra work lives: a second state, a cross-stack reference, a connection secret.</p><p>In <strong>formae</strong> that boundary is an ordinary edge. The Grafana target reading service.res.endpoints.at(&quot;grafana:3000&quot;) is the same kind of reference as a subnet reading mainVpc.res.vpcId: no provider object to instantiate, no second apply, no secret passed by hand. Late binding is what enables this: the resolvable resolves once the resource it points at is ready, so a cross-plugin dependency lives in the data model like any other reference.</p><h3>Wrapping up</h3><p>Hopefully this article has given you a clearer picture of the <strong>formae</strong> infrastructure graph: enough to read an existing codebase, or design one yourself. Notice that none of the later sections introduced any new concepts. The same resources, targets, resolvables, and stacks that describe a single VPC also describe a cloud account running Kubernetes running Grafana. The concepts are few on purpose. What they compose into is anything but.</p><p>formae is open source and free to use for many teams. Check it out! Here are a few links:</p><ul><li><a href="https://docs.formae.io/"><strong>formae</strong> quick start</a></li><li><a href="https://github.com/platform-engineering-labs/formae"><strong>formae</strong> on GitHub</a></li><li><a href="https://docs.formae.io/en/latest/architecture/"><strong>formae</strong> architecture</a></li><li><a href="https://docs.formae.io/en/latest/core-concepts/label/"><strong>formae</strong> core concepts</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d1ce0944bb1" width="1" height="1" alt=""><hr><p><a href="https://blog.platform.engineering/targets-resources-resolvables-and-stacks-the-formae-infrastructure-graph-explained-2d1ce0944bb1">Targets, resources, resolvables and stacks: the formae infrastructure graph explained</a> was originally published in <a href="https://blog.platform.engineering/">Platform Engineering Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Go plugins, actors, and schema evolution: inside formae’s plugin SDK]]></title>
            <link>https://blog.platform.engineering/go-plugins-actors-and-schema-evolution-inside-formaes-plugin-sdk-353ccfd324ac?source=rss-145657d24429------2</link>
            <guid isPermaLink="false">https://medium.com/p/353ccfd324ac</guid>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[formae]]></category>
            <category><![CDATA[iac]]></category>
            <category><![CDATA[ergo]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[Jeroen Soeters]]></dc:creator>
            <pubDate>Mon, 01 Jun 2026 20:57:42 GMT</pubDate>
            <atom:updated>2026-06-01T20:57:42.966Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VDFnboXmGCjD9BwGo1Fqug.jpeg" /></figure><p>Like most infrastructure-as-code (IaC) tools, <strong>formae </strong>has a plugin architecture. The <strong>formae </strong>agent knows how to schedule, queue, order, rate-limit, execute, and retry plugin operations; the actual API interaction lives in what we call <strong>resource plugins</strong>. This post is a short history of how our plugin SDK has evolved, the constraints that forced each change, and where we’ve landed today.</p><h3>A deliberately tiny interface</h3><p>The <strong>formae </strong>resource plugin API is intentionally tiny. It has create/read/update/delete (CRUD) methods, a Status method for long-running (asynchronous) operations, a List method that our continuous discovery process uses, and a handful of configuration knobs. Here it is in full:</p><pre>type ResourcePlugin interface {<br>   <br>    // Configuration<br>    RateLimit() pkgmodel.RateLimitConfig<br>    DiscoveryFilters() []pkgmodel.MatchFilter<br>    LabelConfig() pkgmodel.LabelConfig<br>  <br>    // CRUD operations<br>    Create(ctx context.Context, request *resource.CreateRequest) (*resource.CreateResult, error)<br>    Read(ctx context.Context, request *resource.ReadRequest) (*resource.ReadResult, error)<br>    Update(ctx context.Context, request *resource.UpdateRequest) (*resource.UpdateResult, error)<br>    Delete(ctx context.Context, request *resource.DeleteRequest) (*resource.DeleteResult, error)<br>    <br>    // Async operation support<br>    Status(ctx context.Context, request *resource.StatusRequest) (*resource.StatusResult, error)<br>    <br>    // Discovery support<br>    List(ctx context.Context, request *resource.ListRequest) (*resource.ListResult, error)<br>}</pre><p>We keep this interface as small as we can on purpose. AI coding agents already write the vast majority of plugin code, and a reliable generated plugin comes from three things working together: a small surface with little room to deviate, an extensive <a href="https://github.com/platform-engineering-labs/formae/blob/main/pkg/plugin-conformance-tests/README.md">conformance harness</a> that turns “did it get it right?” into a runnable answer the agent can loop against, and <a href="https://docs.formae.io/en/latest/plugin-sdk/tutorial/">LLM-oriented documentation</a> that walks through a plugin one operation at a time, test-first. The interface is just the first leg.</p><p>That interface has stayed fixed ever since. What changed, three times over, is the machinery behind it: how a plugin is loaded, where it runs, and how it talks to the agent.</p><h3>First iteration: pkg/plugin and Go&#39;s native plugins</h3><p>The goal from day one was that a plugin developer should only have to implement the methods on ResourcePlugin, nothing more. We also didn&#39;t want to spawn a new process for every plugin <em>invocation</em>. The fastest way to get both was Go&#39;s built-in plugin mechanism.</p><p>Go’s plugin package loads compiled Go code at runtime via dynamic linking against shared objects: .so on Linux/FreeBSD, .dylib on macOS. (Notably, there&#39;s no Windows support.) A plugin looked like this:</p><pre>type AWS struct{}<br><br>// Compile-time check that we satisfy the contract<br>var _ plugin.ResourcePlugin = AWS{}<br><br>// Exported symbol the agent looks up by name<br>var Plugin = AWS{}<br><br>func (a AWS) Create(ctx context.Context, request *resource.CreateRequest) (*resource.CreateResult, error) {<br>    // AWS provisioning code<br>}</pre><p>Building it:</p><pre>go build -C plugins/aws -buildmode=plugin -o aws.so</pre><p>The plugin source is ordinary Go in package main (even though no main() ever runs). The toolchain produces an ELF (Mach-O on macOS) shared library with Go&#39;s runtime metadata baked in: type information, GC pointers, and so on. That metadata is exactly why these aren&#39;t just plain C-style shared libraries.</p><p>On the agent side, loading looked like this:</p><pre>dl, err := goplugin.Open(&quot;aws.so&quot;)<br>if err != nil {<br>    return err<br>}<br><br>// Look up the exported &quot;Plugin&quot; symbol<br>sym, err := dl.Lookup(&quot;Plugin&quot;)<br>if err != nil {<br>    return err<br>}<br>plug, ok := sym.(plugin.ResourcePlugin)<br>if !ok {<br>    return fmt.Errorf(&quot;aws.so does not implement ResourcePlugin&quot;)<br>}<br><br>// Now we can use the plugin<br>plug.Create(ctx, req)</pre><p>The plugin ran in-process inside the agent, and the developer only had to implement ResourcePlugin. For a while, this was great.</p><h3>Where it broke down</h3><p>This is where most people who try Go’s plugin package get hurt. The mechanism only works if the agent and the plugin are built in near-perfect lockstep, and that requirement shows up in two ways.</p><p>The first is the <strong>toolchain</strong>. The plugin and the agent must be built with the exact same Go version; even a patch-level mismatch is refused at load time. While every plugin lived in the <strong>formae</strong> monorepo this was a non-issue, since a single Makefile pinned one toolchain for every build. But it would clearly become a problem once anyone started building plugins on their own machines and their own schedules.</p><p>The second, and the one we couldn’t engineer around, is <strong>dependencies</strong>. Any package imported by <em>both</em> the host and the plugin must be at the exact same version, built with the exact same flags (down to things like -race and -tags). This one isn&#39;t fixable on our side: we can&#39;t control, or safely pin, the transitive dependencies that the many SDKs we wrap drag in. The moment two of them conflict, the plugin won&#39;t load, full stop.</p><p>We hit all of this while plugins were still in the monorepo, before we’d shipped a public SDK at all. It was enough to make the verdict obvious: a public plugin SDK built on Go’s plugin package was a dead end, and we needed a fundamentally different approach.</p><h3>Second iteration: lift the plugin into its own process with Ergo</h3><p>The <strong>formae </strong>agent is built on the <a href="https://ergo.services/">Ergo</a> actor framework, which brings an Erlang-style concurrency model to Go. (If the actor model is new to you, I wrote a separate <a href="https://blog.platform.engineering/unlocking-concurrency-in-go-67a530807616">primer on the actor model and Ergo</a>.) For this post, you only need one idea: the actor model’s unit of concurrency is a <strong>process</strong>. Not an OS process, but a lightweight process managed by a VM. In Erlang that VM is the BEAM; in Ergo it’s the <strong>Ergo node</strong>. Processes talk to each other by sending messages, and a process can spawn new processes.</p><p>The <strong>formae </strong>agent hosts an Ergo node, and everything the agent does is carried out by one or more Ergo processes, including plugin operations like the Create call above. The actor that orchestrates plugin operations is the <strong>PluginOperator</strong>.</p><p>So we asked: what if we lift the PluginOperator out into a <em>separate</em> OS process running its own Ergo node (a “plugin process”), and let it talk to the agent over the message protocol the agent already speaks everywhere internally: Ergo messages?</p><p>This isn’t a process per invocation. It’s a persistent process that the agent communicates with by message-passing. And because each plugin is now its own binary, the dependency-hell problem simply disappears: there’s no shared linking anymore. The same separation decouples licensing: since a plugin isn’t linked into the agent, it can carry whatever license its author chooses, so OSS and proprietary plugins can be onboarded side by side. The developer still implements only ResourcePlugin; the SDK is now a normal Go binary with a main() that we already provide, which bootstraps the Ergo node and hoists the plugin implementation into an actor process.</p><p>The piece that makes this elegant is a foundational Erlang/OTP idea: <strong>network transparency</strong>. The location of a process (same Ergo node, a different node on the same machine, or a node halfway around the world) doesn’t change how you interact with it.</p><p>Concretely this means you send messages the same way regardless of where the receiving process lives. A Send to a local process and a Send to a process on another continent are written identically; only the target process identifier (PID) differs. We rely on this in three places:</p><ul><li><strong>Workflow tests</strong> (our automated tests that validate actor orchestration) must <em>not</em> spin up a subprocess per test, so there we spawn the PluginOperator locally on the agent’s node and messages stay in-process.</li><li>In the <strong>OSS formae agent</strong>, all plugins run locally on the agent host, so in production the messages cross the process boundary but stay on one machine.</li><li>For running <strong>formae </strong>at scale, the <strong>paid version</strong> supports <em>satellite agents</em>: agents on different hosts that manage plugin processes and relay back to the primary agent.</li></ul><p>Network transparency makes the agent oblivious to topology. From its perspective it’s just exchanging messages with a PluginOperator process, wherever that process happens to run.</p><p>Here is the shape of the change. Before, the whole pipeline lived in the agent process and shared one address space:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IPh0vqHX2tpgXvYCCnnvow.png" /><figcaption>All actors share one address space; messages never leave the process.</figcaption></figure><p>After, each plugin runs in its own separate process, with its own Ergo node and binary. The ResourceUpdater drives them all the same way, and the messages now cross a process boundary instead of staying in-process:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J9xaXouu3qinn3FVR6I59A.png" /><figcaption>Each plugin runs in its own process. The same message send reaches every one, wherever it lives.</figcaption></figure><h3>The first schema change</h3><p>This worked beautifully in the first public SDK release. Then we needed to change the schema, meaning the message contract between the agent and the plugins. Many early SDK adopters will have painful memories of this:</p><pre>ERR ... unable to decode received message: malformed EDF: end of data</pre><p>The binary serialization format Ergo uses on the wire is the <strong>Ergo Data Format (EDF)</strong>. Out of the box, EDF serializes a registered Go type field by field, and both ends of a connection have to share the exact same definition of that type.</p><p>The type that bit us was the schema. On startup, every plugin announces its resource schemas to the agent: the fields each resource has, plus a set of annotations the engine needs, things like whether a field is create-only, whether it’s required, or how it links a resource to its parent. These schemas are authored in Pkl and translated to Go, and they change constantly, because we add new annotations as new engine features land. A trimmed-down version of one annotation type:</p><pre>type FieldHint struct {<br>    CreateOnly       bool<br>    WriteOnly        bool<br>    Required         bool<br>    RequiredOnCreate bool<br>}</pre><p>When a new feature needs a new annotation, we add a field:</p><pre>type FieldHint struct {<br>    CreateOnly       bool<br>    WriteOnly        bool<br>    Required         bool<br>    RequiredOnCreate bool<br>    EdgeKind         EdgeKind // new: how this field links resources in the dependency graph<br>}</pre><p>Under native EDF, that one addition was enough to break things. These annotation types are nested inside the announcement message, so changing FieldHint changed the registered type of the whole announcement. An agent built against the new definition and a plugin still compiled against the old one no longer agreed, and EDF refused to decode the message. That is the malformed EDF error above, and because it happened at announce time, an out-of-date plugin couldn&#39;t even start.</p><p>For a plugin SDK, that’s a non-starter: a backwards-compatible change on our side should never force every plugin author to recompile and re-publish on our release schedule.</p><h3>Third iteration: MessagePack to the rescue</h3><p>Fortunately, iteration three was a far smaller pivot than the jump to separate processes. Ergo lets you supply custom marshalers: MarshalEDF(w io.Writer) error and UnmarshalEDF(data []byte) error. EDF doesn&#39;t disappear as the wire protocol; instead, your custom payload gets wrapped in EDF&#39;s framing. On the wire it looks roughly like this:</p><pre>[ EDF type tag ][ EDF length prefix ][ your MarshalEDF output ]</pre><p>The EDF decoder reads the type tag, and if the type implements Unmarshaler, it allocates a value of that registered type and calls UnmarshalEDF(bytes) on it.</p><p>The important shift is that EDF no longer looks <em>inside</em> the message. It matches on the registered type tag and treats the body as an opaque blob, so the struct’s fields are no longer part of what the two nodes have to agree on. The body’s encoding is now ours to choose.</p><p>We chose <strong>MessagePack</strong>. Protobuf was the obvious alternative, but it brings a codegen step and a schema toolchain we didn’t want, and its main payoff, a language-neutral contract, is wasted on us: these messages change quickly and will only ever be read by Go.</p><p>For most messages the marshaler is a one-liner on each side. The plugin announcement that carries the schemas is just another message type:</p><pre>func (m PluginAnnouncement) MarshalEDF(w io.Writer) error    { return encodeMsgpack(w, &amp;m) }<br>func (m *PluginAnnouncement) UnmarshalEDF(data []byte) error { return decodeMsgpack(data, m) }</pre><p>Because MessagePack encodes each field by name (the body is a map keyed by field name, not a fixed sequence of positions), adding an annotation no longer breaks the announcement. A node that doesn’t know a new annotation skips it; a node that receives an older schema simply doesn’t see the annotations that weren’t there yet. A plugin compiled against last month’s schema announces successfully to today’s agent, and nobody has to recompile.</p><p>Missing annotations come back as zero values, which isn’t always the behavior we want for an older schema. So as we read schemas (they originate as Pkl, and cross into Go as JSON keyed by the same field names), we normalize the older shapes. The deprecated AttachesTo flag, for example, is folded into the newer EdgeKind:</p><pre>func (fh *FieldHint) UnmarshalJSON(data []byte) error {<br>    type alias FieldHint<br>    var raw alias<br>    if err := json.Unmarshal(data, &amp;raw); err != nil {<br>        return err<br>    }<br>    *fh = FieldHint(raw)<br>    if fh.EdgeKind == &quot;&quot; { // schema predates EdgeKind<br>        fh.EdgeKind = EdgeKindDefault<br>        if fh.AttachesTo { // derive it from the deprecated annotation<br>            fh.EdgeKind = EdgeKindAttachesTo<br>        }<br>    }<br>    return nil<br>}</pre><p>A new annotation, which used to break every plugin’s startup, is now an additive change. Schema evolution becomes a local change on our side instead of a coordinated, fleet-wide recompile.</p><p>Getting there did surface one bug in Ergo’s own length-prefix handling, which produced the same malformed EDF error for large custom-marshaled payloads; we <a href="https://github.com/ergo-services/ergo/pull/257">fixed it upstream</a>.</p><h3>Where we are today</h3><p>Three iterations in, the contract for a plugin author is still the same one we started with (implement ResourcePlugin), but everything underneath has been rebuilt for a world where plugins are independent binaries, evolving on a schema we can change without breaking anyone. Today they can be written by anyone (and increasingly by AI agents), deployed across hosts, and published and shared on <a href="https://hub.platform.engineering/"><strong>formae</strong>/hub</a>, our plugin registry.</p><p>Useful links:</p><ul><li><a href="https://docs.formae.io/"><strong>formae</strong> quick start</a></li><li><a href="https://github.com/platform-engineering-labs/formae"><strong>formae</strong> on GitHub</a></li><li><a href="https://hub.platform.engineering/"><strong>formae</strong>/hub</a> — browse and publish plugins</li><li><a href="https://docs.formae.io/en/latest/plugin-sdk/tutorial/">Plugin SDK tutorial</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=353ccfd324ac" width="1" height="1" alt=""><hr><p><a href="https://blog.platform.engineering/go-plugins-actors-and-schema-evolution-inside-formaes-plugin-sdk-353ccfd324ac">Go plugins, actors, and schema evolution: inside formae’s plugin SDK</a> was originally published in <a href="https://blog.platform.engineering/">Platform Engineering Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Solving Infrastructure Drift With formae]]></title>
            <link>https://blog.platform.engineering/solving-infrastructure-drift-with-formae-f1d77ac6459f?source=rss-145657d24429------2</link>
            <guid isPermaLink="false">https://medium.com/p/f1d77ac6459f</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[iac]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <dc:creator><![CDATA[Jeroen Soeters]]></dc:creator>
            <pubDate>Thu, 11 Dec 2025 10:55:06 GMT</pubDate>
            <atom:updated>2025-12-11T11:15:54.765Z</atom:updated>
            <content:encoded><![CDATA[<h4>When your infrastructure code is up-to-date, there is no drift.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kTc4CIMeiYLsevOmsu8W7A.png" /></figure><p>Drift is a common challenge in the infrastructure-as-code (IaC) world. Infrastructure drift is the phenomenon where the actual state of infrastructure diverges from the desired state defined in code. This happens, for example, when changes are made directly to the infrastructure outside of the IaC tools. A core characteristic of the existing IaC platforms is the assumption of centrality and full ownership — that the IaC tool itself is the sole source of truth and the <em>only</em> approved mechanism for infrastructure modifications. This design philosophy inevitably leads to the questionable best practice of prohibiting or severely restricting changes outside of the IaC workflow.</p><p>Meet <em>formae</em>, the IaC tool built to embrace the operational challenges from using various tools and approaches. <em>formae </em>continuously monitors your live infrastructure and codifies it into a single, versioned source of truth. When out-of-band changes are detected, <em>formae </em>automatically<em> </em>turns them into infrastructure code, and allows you to incorporate these changes into your own IaC codebase.</p><p>In this article, we will look at how this works in practice.</p><h3>Synchronization</h3><p>The <em>formae </em>agent is an active component that typically runs in the target infrastructure. When you submit a <em>formae </em>command via the CLI, the agent executes this command, not the CLI process. This allows <em>formae </em>to be intelligent about command execution. When conflicting commands are submitted by multiple people, <em>formae </em>can decide whether to queue the commands, reject the commands or execute them in parallel.</p><p>But this is not all the agent is doing. The agent also periodically synchronizes its internal state with your live infrastructure to detect any out-of-band changes. You can customize at what interval this synchronization happens in the <em>formae</em> configuration. During the synchronization <em>formae </em>will compare all resources it knows about with their counterparts in the live infrastructure. If any differences are found, <em>formae </em>will store new versions of those resources internally. It also keeps record of which changes were made by the user via <em>formae </em>commands, and which were made outside of the tool.</p><p>You can view the current state of your infrastructure by using the formae inventory command. Let’s take a look at how this works.</p><h3>Creating a Resource with formae</h3><p>First, let’s create an AWS S3 bucket using <em>formae</em>. The forma Pkl file looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vcvSQi_Yil82bQCe4T2d7Q.png" /><figcaption>buc-ee.pkl</figcaption></figure><p>Based on this forma Pkl, <em>formae </em>will create a stack called buc-ees with a single AWS S3 bucket resource. We can apply this forma using the formae apply command.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*70Z3XCsM1I42AmmT6VwGBQ.png" /></figure><p>We should see the resource being created. The --watch flag allows us to follow the progress of the operation. <em>formae </em>commands execute asynchronously on the agent, so we can at any time safely exit the CLI and at a later point check in on the status using formae status command.</p><p>When the operation completes, we can verify in the AWS Management Console that the bucket is created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*21HqWwnAORm2wV4dRbkzoQ.png" /></figure><p>Success! The S3 bucket has been created as expected. Now let’s use the formae inventory command to see if <em>formae </em>is tracking the resource.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w8iUTNoCi_zKzAs8NT5n6Q.png" /></figure><p>By default <em>formae </em>displays a human-friendly view of the resource. We can see the machine-friendly output using the--output-consumer machine flag. We will do this in a bit.</p><p>Now let’s return to the AWS Console and make an out-of-band change to the AWS S3 bucket. For this example, we will add a tag to the bucket.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fDSt-cTv1o1muF5IXVrxLQ.png" /></figure><p>We wait for <em>formae</em>’s synchronization job to pick up the change, and again run the formae inventory command. Now we request the machine-friendly output in the default JSON format, so we can pipe it to a tool like jq for detailed inspection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xyPFUMQl4q0v4IdWRGifCA.png" /></figure><p>We can see that <em>formae </em>has detected the out-of-band change made to the AWS S3 bucket. The new tag is now part of the resource definition in <em>formae</em>.</p><h3>Reconcile vs. Patch</h3><p>You might have noticed we used the --mode reconcile flag when applying the forma earlier and wondered what it does. The presence of this flag tells <em>formae </em>that you want your actual infrastructure to <em>exactly </em>match the desired state defined in your forma Pkl file. Any resources defined in the forma that do not exist in the live infrastructure will be created, any resources that exist in the live infrastructure but are not defined in the forma will be deleted, and any resources that exist in both but differ, will be updated in the live infrastructure.</p><p>The other mode <em>formae </em>supports is --mode patch, which allows you to make surgical changes with minimal blast radius. We’ll look at this mode in another article.</p><h3>Hard vs. Soft Reconcile</h3><p>Let’s see what happens when we now try to reconcile from infrastructure code by running the formae apply command again — after we detected out-of-band changes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yb3VBvY_Cbx3iJK9IXd80w.jpeg" /></figure><p>As you can see, <em>formae</em> rejects the operation outright.</p><p>This happens because, by default, <em>formae</em> attempts to reconcile in the so-called <strong>soft</strong> mode. In this mode, unlike traditional tools that blindly enforce reconciliation from code, <em>formae</em> assumes that out-of-band changes might be intentional and important. It guarantees that out-of-bound work — whether it was a hotfix or an experiment — is never silently overwritten.</p><p>When <em>formae</em> detects this kind of reconciliation conflict, it offers two clear paths forward:</p><p><strong>1. Hard reconciliation: </strong>You can choose to enforce reconciliation from code. By running formae apply with the --force flag, you perform <strong>hard</strong> reconciliation. This is how the traditional GitOps approach using existing IaC tools works: the manually added tag is discarded, and the infrastructure is reset to exactly match your forma file.</p><p><strong>2. Absorb out-of-bound changes: </strong>Alternatively, you can choose to embrace the change. Since <em>formae</em> knows exactly what changed, it provides a formae extract command in the output. This command generates the Pkl code that matches the live infrastructure. Let’s run the suggested extract command.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NVcRhh_PDFFpWGpVq_UXTw.png" /></figure><p>In the resulting code, we see the tag we manually added. You can now incorporate this code into your existing IaC codebase, bringing it back in sync with reality — without losing the out-of-band changes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kZCIdTiVTUzrSBFOfXFmsg.png" /></figure><p>We can also choose to do <strong>hard</strong> reconciliation using the --force flag and discard the out-of-band changes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HTJ5R0n6AeWrenjUIPNF2A.png" /></figure><p>A perspicacious reader might wonder what happens if we make an out-of-band change, and before <em>formae</em>’s synchronization job detects it, we make another change via <em>formae</em>. No worries, <em>formae </em>has your back. Before any destructive update <em>formae </em>will always check if the live infrastructure matches the last known state. If it doesn’t, <em>formae </em>will reject the command, and it is up to you to resolve the conflict as described above.</p><h3>Conclusion</h3><p>There are many scenarios where out-of-band changes are legitimate. For example, a security team responding to an active incident using their own tools, an infrastructure architect ensuring resources are correctly tagged, cost-optimization tools doing their job etc. Instead of fighting drift, <em>formae </em>embraces reality and puts you in control.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZUxcKxMejg-Rb_lh.png" /></figure><p><strong>Check out formae </strong>— it’s open source:</p><p>Quick start: <a href="https://docs.formae.io/">https://docs.formae.io</a></p><p>Code: <a href="https://github.com/platform-engineering-labs/formae">https://github.com/platform-engineering-labs/formae</a></p><p>formae 101: <a href="https://docs.formae.io/en/latest/formae-101/fundamentals/">https://docs.formae.io/en/latest/formae-101/fundamentals/</a></p><p>Web: <a href="https://platform.engineering/formae">https://platform.engineering/formae</a></p><p><strong>Follow us</strong> on <a href="https://www.linkedin.com/company/platform-engineering-labs">LinkedIn</a> or <a href="https://platform.engineering/contact">register</a> for news and updates.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f1d77ac6459f" width="1" height="1" alt=""><hr><p><a href="https://blog.platform.engineering/solving-infrastructure-drift-with-formae-f1d77ac6459f">Solving Infrastructure Drift With formae</a> was originally published in <a href="https://blog.platform.engineering/">Platform Engineering Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unlocking Concurrency in Go]]></title>
            <link>https://blog.platform.engineering/unlocking-concurrency-in-go-67a530807616?source=rss-145657d24429------2</link>
            <guid isPermaLink="false">https://medium.com/p/67a530807616</guid>
            <category><![CDATA[concurrency]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[actors]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <dc:creator><![CDATA[Jeroen Soeters]]></dc:creator>
            <pubDate>Thu, 15 May 2025 08:53:15 GMT</pubDate>
            <atom:updated>2025-05-15T14:44:21.681Z</atom:updated>
            <content:encoded><![CDATA[<h3>Unlocking Concurrency in Go: A Practical Guide to the Actor Model with Ergo</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UAghwfKTJoVQX2Wz" /></figure><p>Hollywood stars in software engineering, you think? No, it’s not the actors you see in TV shows we’ll cover in this article. These are the actors the late Carl Hewitt coined in the early 70s when he was researching how to program highly parallel computers.</p><h3>The Actor Model: A Different Paradigm for Concurrent Computation</h3><p>The actor model is a paradigm for concurrent computing. It’s an alternative to traditional threading/locking, which is the prevailing way of writing concurrent code in C++ or Java, or Communicating Sequential Processes (CSP), which forms the basis of Go’s native concurrency primitives: goroutines and channels.</p><p>In the actor model, everything is an actor. As E.O. Wilson famously said, “One ant is no ant.” The same holds for actors: “One actor is no actor.” Actors come in systems, and because they come in systems, they need addresses, so they can send messages to each other. Of course, an actor can also send a message to itself, which is how recursion is implemented in the actor model.</p><p>As the fundamental unit of computation, an actor encompasses three things:</p><ul><li>Processing: the actor needs to be able to get something done</li><li>Storage: actors need to be able to remember things</li><li>Communication: actors need to be able to communicate with one another</li></ul><p>The behavior and capabilities of actors are defined by a fundamental set of axioms, primarily articulated by Hewitt, Bishop, and Steiger. When an actor receives a message, it can perform three fundamental actions:</p><ol><li>It can create more actors, allowing the system to dynamically expand, delegate tasks, and form hierarchical structures. For example, a ShoppingCartActor might create subordinate PaymentActor and ShippingActor instances to handle specific parts of an order process. This capability is essential for building systems that adapt their resource allocation and structure in response to varying workloads or events. This axiom is what enables advanced fault-handling strategies by creating so-called supervision hierarchies on which we’ll talk more in a future blog post.</li><li>It can send messages to other actors of which it has addresses: this is the exclusive mechanism for inter-actor communication and coordination. This axiom underscores the message-centric nature of the actor model.</li><li>It can designate the behavior to be used for the next message it receives. In essence this means it can change its internal state and consequently how it responds to future messages based on the sequence of messages it has already processed. A DepositAccountActor with a balance of $5, after receiving a deposit message of $7, will handle the next message with a balance of $12.</li></ol><h3>Actors: OOP as it was meant to be?</h3><p>Ironically, this is not very dissimilar to what Alan Kay, the inventor of Object-Oriented programming (OOP), had in mind when he coined the term. His core principles for OOP included:</p><ul><li>Messaging: objects must interact by sending and receiving messages</li><li>Encapsulation of state: each object is solely responsible for its own data, which must be shielded from any external interference</li><li>Extreme Late-Binding: the decision which piece of code to execute in response to a received message should be deferred until the message is actually received at runtime (parametric polymorphism).</li></ul><p>He regretted the term “objects” and argued that true object-oriented programming focuses on inter-communicating entities (objects) that interact through messages, rather than the objects themselves. He famously said: “I invented the term Object-Oriented, and I cant tell you I did not have C++ in mind”.</p><h3>Erlang: The Battle-Tested Actor Powerhouse</h3><p>No discussion of the actor model is complete without acknowledging Erlang, a programming language and runtime system where actor-like principles are not just a library feature, but the very foundation of its concurrency model.</p><p>Erlang was developed at Ericsson in the 1980s to build highly available, massively concurrent, and fault-tolerant telecommunications systems, such as telephone switches. The primary design goals were concurrency, distribution, soft real-time performance, and continuous operation. These requirements naturally led to an architecture closely resembling the actor model.</p><p>Erlang’s decades of success in building some of the world’s most reliable and scalable concurrent systems (e.g., WhatsApp, Ericsson’s AXD301 switch) has profoundly influenced many subsequent actor model implementations in other languages and platforms, including Akka for the JVM, Orleans for.NET, and, by extension, Go’s Ergo framework.</p><h3>Ergo: Actors for the Next Generation of Infrastructure Tooling</h3><p>At Platform Engineering Labs, we are building the next generation of infrastructure tooling. Although we have our fair share of Erlang aficionados among our colleagues, we have adopted Go as our primary programming language. Go has become the leading language for modern infrastructure tooling and cloud-native development. A vast majority of CNCF projects, which form the bedrock of the cloud-native ecosystem, are all written in Go. This quickly led us to the <a href="https://github.com/ergo-services/ergo">Ergo Framework</a>. Ergo aims to bring the power and elegance of Erlang/OTP’s actor-based design patterns to the Go programming language. It provides a robust platform for building scalable, fault-tolerant, and distributed applications.</p><p>Ergo is heavily inspired by Erlang. It is built upon lightweight processes, which are themselves implemented using Go’s native goroutines. A notable characteristic is its “zero external dependencies” philosophy, relying solely on the Go standard library. Ergo offers a rich set of features to facilitate actor-based development. We’ll walk through the basic ones with a simple task dispatcher example. In this example, we will create a TaskDispatcherActor, which dynamically spins up a pool of Worker actors to calculate prime numbers within a given range. We divide the range across the pool of workers, and each worker will report the primes they found to a ResultsCollector actor.</p><p>Sounds simple enough? Let’s create our first actor. In Ergo, we create an actor by embedding the`Actor` type in a struct.</p><pre>type TaskDispatcher struct {<br>  act.Actor<br>  numberOfWorkers int<br>}</pre><p>You interact with the actor via the ActorBehavior interface, which Actor implements. All methods in this interface are optional. The first one we implement for our TaskDispatcher actor is the Init(args …any) error method. This method runs at the start of process initialization, and allows us to pass in arguments and set up some initial state. It is important to note that during this initialization, functionality like creating actors or sending messages to other actors is not yet available to us. We can only do this later in the lifecycle upon receiving messages. Remember those axioms? All we do in this initialization is parse the number of workers we want the dispatcher to create from the arguments and store this in the actor state.</p><pre>func (td *TaskDispatcher) Init(args …any) error {<br>  if (len(args) &gt; 0) {<br>    td.numberOfWorkers = args[0].(int)<br>  } else {<br>    // default number of workers to 4<br>    td.numberOfWorkers = 4<br>  }<br>  return nil<br>}</pre><p>Now let’s define some messages for our actors to pass around. We define a message CalculatePrimes with a range and another message PrimesFound to pass along the results to our ResultsCollector actor.</p><pre>type CalculatePrimes struct {<br>  RangeStart int<br>  RangeEnd int<br>}<br><br>type PrimesFound struct {<br>  Primes []int<br>}</pre><p>The next method we have to implement is the HandleMessage(from gen.PID, message any) error method. Our TaskDispatcher actor only receives a single type of message, the CalculatePrimes message. When it receives this message, the TaskDispatcher first spawns the requested number of worker actors. It uses the Spawn(factory ProcessFactory, options ProcessOptions, args …any) (gen.PID, error) method of the Process interface, which Actor embeds. The Spawn method returns a PID. PID stands for process identifier, and it represents the address of the actor that we need to be able to send messages to it. We divide the requested range among the workers and send each worker a CalculatePrimes message for the assigned range.</p><pre>func (td *TaskDispatcher) HandleMessage(from gen.PID, message any) error {<br>  switch msg := message.(type) {<br>  case CalculatePrimes:<br>    for i := range td.numberOfWorkers {<br>      rangeSize := (msg.RangeEnd - msg.RangeStart + 1) / td.numberOfWorkers<br>      workerRangeStart := msg.RangeStart + i*rangeSize<br>      workerRangeEnd := workerRangeStart + rangeSize - 1<br><br>      pid, err := td.Spawn(<br>        func() gen.ProcessBehavior {<br>          return &amp;Worker{}<br>        }, gen.ProcessOptions{})<br>      if err != nil {<br>        log.Fatalf(&quot;failed to spawn worker actor: %v&quot;, err)<br>      }<br><br>      td.Send(pid, CalculatePrimes{workerRangeStart, workerRangeEnd})<br>      if err != nil {<br>        log.Fatalf(&quot;failed to send messag to worker actor: %v&quot;, err)<br>      }<br>    }<br>  default:<br>    td.Log().Error(&quot;Unknown message: %v&quot;, msg)<br>  }<br>  return nil<br>}</pre><p>Now we have to implement HandleMessage(…) for the Worker actor. The worker actor finds the primes in the range and forwards the results to the ResultsCollector actor. Note that the Worker actors do not know the PID of the ResultsCollector. In addition to Spawn, Ergo provides another method to create new actors: SpawnRegister(register gen.Atom, factory ProcessFactory, options ProcessOptions, args …any) (gen.PID, error). This allows us to associate a name with an actor; other actors can send messages using this name as the address.</p><p>Actors can terminate themselves by returning a non-nil error from the handler. Here, we return the gen.TerminateReasonNormal error type for a normal shutdown scenario after the worker is done processing.</p><pre>func (w *Worker) HandleMessage(from gen.PID, message any) error {<br>  switch msg := message.(type) {<br>  case CalculatePrimes:<br>    primes := findPrimesInRange(msg.RangeStart, msg.RangeEnd)<br>    err := w.Send(gen.Atom(&quot;ResultsCollector&quot;), PrimesFound{primes})<br>    if err != nil {<br>      log.Fatalf(&quot;failed to send messag to worker actor: %v&quot;, err)<br>    }<br>    return gen.TerminateReasonNormal<br>  default:<br>    w.Log().Error(&quot;Unknown message: %v&quot;, msg)<br>  }<br>  return nil<br>}</pre><p>Next, we implement HandleMessage(…) of the ResultsCollector actor, which waits for all results to come in and prints them to the screen.</p><pre>func (rc *ResultsCollector) HandleMessage(from gen.PID, message any) error {<br>  switch msg := message.(type) {<br>  case PrimesFound:<br>    rc.results = append(rc.results, msg.Primes…)<br>    rc.numberOfResults = rc.numberOfResults + 1<br>    if rc.numberOfResults == rc.expectedNumberOfResults {<br>      fmt.Printf(&quot;All results collected. Found the following primes: %v&quot;, msg.Primes)<br>    }<br>  default:<br>    rc.Log().Error(&quot;Unknown message: %v&quot;, msg)<br>  }<br>  return nil<br>}</pre><p>To create our initial actor, we have to create an Ergo node. The node is the core service in Ergo that hosts the actor runtime. Each node has a name that consists of two parts: the name of the node and the network interface the node binds to.</p><p>We create a node using the Start(name gen.Atom, options gen.NodeOptions, frameworkVersion gen.Version) (gen.Node, error) method. In the main goroutine, we create our TaskDispatcher actor as well as the ResultCollector actor and send the CalculatePrimes message to the TaskDispatcher to start our example.</p><pre>func main() {<br>  name := gen.Atom(&quot;example@localhost&quot;)<br>  node, err := ergo.StartNode(name, gen.NodeOptions{})<br>  if err != nil {<br>    log.Fatalf(&quot;Failed to start the Ergo node: %v&quot;, err)<br>  }<br>  tdPid, err := node.Spawn(factoryTaskDispatcher, gen.ProcessOptions{}, 5)<br>  if err != nil {<br>    log.Fatalf(&quot;Failed to spawn the task dispatcher: %v&quot;, err)<br>  }<br>  _, err = node.SpawnRegister(gen.Atom(&quot;ResultsCollector&quot;), func() gen.ProcessBehavior { return &amp;ResultsCollector{} }, gen.ProcessOptions{}, 5)<br>  if err != nil {<br>    log.Fatalf(&quot;Failed to spawn the task dispatcher: %v&quot;, err)<br>  }<br>  node.Send(tdPid, CalculatePrimes{0, 100})<br>  node.Wait()<br>}</pre><p>The full example code is available on <a href="https://github.com/JeroenSoeters/erg-task-dispatcher/tree/master">GitHub</a>.</p><p>In a future blog post, we will dive deep into Ergo’s advanced fault-handling strategies using supervision hierarchies.</p><p>At Platform Engineering Labs, we’re on a mission to democratize platform engineering. Making it universally accessible and seamlessly adoptable across engineering teams. Behind the scenes, we’re using complex but battle-tested technology to make things feel simple, reliable, and intuitive for our users.</p><p>We’re building some powerful tools, and while most of it is still under wraps, we can say this: <strong>if you’re into infrastructure that just works (and works hard), you’re going to love what’s coming.</strong></p><p>Stay tuned.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=67a530807616" width="1" height="1" alt=""><hr><p><a href="https://blog.platform.engineering/unlocking-concurrency-in-go-67a530807616">Unlocking Concurrency in Go</a> was originally published in <a href="https://blog.platform.engineering/">Platform Engineering Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>