Infra at Scale: Agents, Tools and Infra-as-Code
In a previous post, How We Run Terraform At Scale, we shared how Benchling moved from a painful, laptop-driven, apply-then-merge workflow to an automated infrastructure release system built on Terraform Cloud (TFC). That migration dramatically reduced conflicts and drift, saved an estimated 8,000 developer hours annually, and gave us the foundation to scale from 350 to 625 workspaces managing many thousands of cloud resources across 38 AWS accounts. We also walked through the ecosystem of custom tooling we built around TFC to make it work at our scale: a CLI for bulk review and approval, Slack notifications for pending runs, and automated workspace dependency mapping.
Benchling continues to expand both in platform capabilities and geographical reach (close to a thousand Terraform workspaces!). This post will discuss how we leverage AI agents to operate faster and more safely than ever before. We will focus on how we now roll out routine updates to our Kubernetes clusters.
What’s good for humans is good for agents
Before we dive into our AI agent workflows it’s important to state what has not changed: Infrastructure as Code (IaC). It turns out the values we embraced to ensure human change management was safe still hold true when AI agents are introduced into the mix:
- Modular, declarative Terraform: changes at scale are clearly represented in code
- Changes require authorized human review/approval in a GitHub Pull Request (PR) before merge
- Terraform Cloud (or any CI/CD) provides a secure environment where approved changes are rolled out automatically
Since our customers and their work are our highest priority, it was critical that AI agents be incorporated into this process in the safest manner possible, with a human in the loop. So this is where we landed: AI agents can produce PRs but an authorized engineer must review and approve them. This is a great fit because the majority of effort lies in understanding the required change, making the right code change and testing it. And it turns out agents are really good at that.
The CLIs for running Terraform plans and for bulk-approving runs we mentioned in our previous blog post are also still providing value. We’ll come back to that later in the post.
Many components, endless updates
Just keeping various components of our infrastructure up to date was a big source of toil. Our Kubernetes cluster configuration alone involves over thirty different components that require periodic updating:
- Helm Charts (Karpenter, Argo CD, etc)
- Amazon Machine Images for Kubernetes nodes
- Container Images (e.g., Datadog agent)
Components vary in terms of frequency of updates, blast radius in the event of failure, possible dependencies and method of validating the update.
Performing updates in stages
Because updating a component can occasionally result in failure (e.g., a backward incompatible release), we roll out the update in a staged manner. Benchling has a number of additional stages but we’ll simplify here:
- development (private sandbox for testing, requires no approval)
- staging (requires approved PR, internal and customer testing)
- production (only after testing in staging passes)
Each stage spans many clusters, across regions and AWS accounts. Rather than configure each one, we keep component versions in checked-in constants files, keyed by stage, and a pre-commit script compiles those into per-cluster Terraform variable files.
The upshot is that updating a component across an entire stage of our global fleet is a one-line change:
prod:
- datadog_agent_version: "x.y.z"
+ datadog_agent_version: "x.y.z+1"Which means any of these updates can be rolled out in a staged manner by simply putting up a series of PRs: PR1 dev, PR2 staging, PR3 prod.
So, now we’ve got N update types times M stages. That’s a lot of PRs!
Get Brendan Neutra’s stories in your inbox
Join Medium for free to get updates from this writer.
In addition, the initial rollout to dev usually involves inspecting the release notes of the component and addressing backward incompatibility or doing additional testing. With this work straining our capacity we had been skipping over releases, which only increased the risk of each update. “Patch” releases (the “z” in “x.y.z”) and “minor” releases (the “y”) are supposed to be backward compatible but the more you skip the more you risk hitting a bug. Plus, you miss out on important fixes, functionality and security patches.
Rollout with a scheduled agent
With all the IaC pieces in place, the agent is well positioned to take on this task. And just to reiterate: The agent doesn’t need privileged access to our production systems, it just needs permission to run Terraform plans, put up PRs and (optionally) test in a dev sandbox. We’ve been using Cursor Cloud Agents/Automations, which is a hosted solution. Its access to AWS, Datadog, and GitHub is limited by policies the customer defines.
A config per component
Components have a change-managed rollout config file. It’s a mix of parameters and prose that tells the agent: where to look for new releases, what to read before touching anything, how long to let each stage bake, and how to tell whether what it just upgraded is healthy. The scheduled automation prompt basically instructs the agent to follow a sequence of steps where the key input details are provided by the rollout config file:
new_version_detection:
- Where to find published releases (url) for this component
- Constraints on which releases to pick, e.g. we only pick up stable
releases and typically wait until a release has been out for 7 days
before picking it up
research:
- Instructions on what to look out for in release notes: breaking changes,
deprecated features, schema changes
- This could extend to asking it to actually walk commits since the
previous release
- Dependencies, i.e. other rollout components whose published compatibility
with this component should be checked
verification:
- Instructions on where to deploy the change in dev and specific
commands to run or Datadog metrics to check to validate the component
is healthy
bake_time:
- How many days to "bake" in a stage before advancing
plan_matchers:
- An expression describing the expected Terraform resource changes
(more on this later)The tick
An agent wakes on a schedule and walks the configs with the aim of identifying and upgrading one component. For each component it works down a chain of gates, where any single “no” ends the tick:
Most ticks create nothing at all. The dev-cluster apply/verify step is only included for the PR that initiates a rollout; if verification fails there, no PR is opened and the failure goes to the owning team in Slack.
The agent is stateless. Its position is derived every tick from the constants files we mentioned earlier (i.e. version of this component each stage is on) and the git log (i.e. when the last stage merged).
This has been running since April. Since then, the agents have opened and merged hundreds of rollout PRs covering over thirty different components. This is about five times our previous cadence! We’re not only saving human time but also doing much better at keeping up to date.
Why an agent and not a script
Where the agent really shines is in the non-deterministic phases of this process and not in the more mechanical steps. A recent example: the rollout agent picked up what looked like a routine dependency bump for one of the controllers we rely on to keep network configuration in sync across our clusters.
The upstream release notes for that version had a documented warning: a change to a default value the controller uses to identify which resources it manages, in some cases, could result in it ceasing to sync. This would cause a network interruption! Figuring out whether that applied to us required studying our own configuration. The agent found that, indeed, we were relying on that old default. The key here is providing the right context, leveraging AI inference over arbitrary text: release notes, the component source code and our codebase.
The agent put up a PR doing the initial upgrade but included the fix to preserve the default value. The PR included a full write-up of what it found. Finally, it applied the change to the dev cluster and confirmed the network settings had been preserved.
It’s important to note that if the agent had failed to recognize the breaking change, or test it properly, our staged rollout process still would have prevented damage to prod (dev clusters would have failed and we’d be forced to investigate the issue). The agent saved us that headache and got it right the first time!
Terraform run approval
We require human review of all production Terraform runs and in the previous blog post we described our CLI to provide an easy method to bulk-approve runs. With this increased velocity of changes, we’ve made another optimization to streamline that part of the process. The agent can include an expression of the expected Terraform plan in the PR description. We call this a “Plan Matcher” which is expressed as:
ACTION:*RESOURCE_TYPE.RESOURCE_ID*In this way, the human PR reviewer is approving both the code being changed as well as pre-approving the Terraform runs. As long as the planned changes for that commit match the expression, our Reviewer service will apply them for you. If a plan contains anything the matcher doesn’t cover, nothing is applied and the run waits for human review. A block in the PR description representing a terraform resource address like: module.cluster.module.controllers[0].module.component_x.helm_release.component_x
would look like:
## Plan Matchers
update:*helm_release.component_x*For the scheduled rollouts the Cloud Agents can execute Terraform plans and derive the plan matchers using CLI tools. The plan matchers can also be encoded in the checked-in rollout config. Post-merge, the Reviewer service validates the runs and applies them in Terraform Cloud.
Recap
Cloud infrastructure requires significant maintenance work and we’ve found that Infrastructure as Code, good tooling, and scheduled agents together are an effective way to drive it — while still keeping a human in the loop for every production change.
The agent’s value here is in reading release notes carefully, connecting them to our own particular configuration, and testing the result. This scheduled agent framework is effectively keeping our infrastructure components up-to-date and allowing us to spend more time making Benchling infrastructure more capable and reliable for our customers.
Acknowledgements
Many engineers within our Production Infrastructure organization have contributed to the systems described above. A special shoutout to Krys Weeton who introduced and evolved our Terraform Reviewer service and CLI, Ted Crossman, who developed the tooling we use to organize/compile/test inputs to Terraform, and Kayne Barclay who developed the agent rollout framework.
