Humans Own The Loop
TL;DR
We took a task from a real enterprise system: migrate two legacy projects into a single microservice, add a new feature, and fix two production bugs. We ran the same task twice, with the same LLM, the same agent platform (OpenCode), and the same prompts. Only the setup changed. Option A had just the 3 skills that hold the migration rules. Option B had our own harness, which also includes those 3 skills.
To score the results, we used a service that the developers had already built for the same job and that is running today. We call it the reference service (reference implementation). The agents never saw it.
The table below sums up the results:
Both agents finished every task, but they built different things. Option B put the business rules where the reference service puts them; Option A copied them into the new service. Most of the 2.7x gap in tokens and time comes from the coding phase: Option A spent 274 minutes on it, Option B spent 49. The parts of the harness that made the difference were also the simplest: a catalog of our projects and how they relate, and a routing table that says which skill to use for which job.
1. Introduction
AI coding platforms make us faster and help us ship better products, and we keep trying new ones to grow our AI literacy. But we don’t want to be tied to any single tool. As we often discuss in our architecture committee and AI workshops, we would rather build our own setup that works across all of them. How we work with these tools has changed a lot over the last two years. We started using them early. First we shared prompts. Then we wrote skills and built plugins. Today we design loops for analysis, development, and operations, and we wrap the tools in layers that align them with our organization, keep them secure, and verify their output.
We do this for two reasons. First, models can already write code that works. What they can’t do on their own is follow our technical choices and rules, and that is where people add the most value today (Goedecke, 2026). Second, an agent is a loop: at each step, the model decides whether to keep going. The harness is everything that runs around the model. It gives the model input, runs the tools it asks for, and keeps the loop going (Aycı, 2026). No matter how good the model is, we need our own loop and harness to fit it into how we work.
Published research also points to enterprise context as the bottleneck. In METR’s randomized study with experienced developers, tasks in mature repos took 19% longer when AI tools were allowed, while the developers believed they had been 20% faster (METR, 2025). On SWE-Bench Pro, the best models solve about 44% of the public set but only 18% of the set built from closed-source enterprise repos (Deng et al., 2025).
2. Our harness
A harness is everything in an AI agent except the model. Böckeler splits it into two layers (Böckeler, 2026):
- The inner harness is what the agent’s makers build into the platform: the agent loop, system prompt, tools, sandbox, memory, conversation history, compaction, model selection.
- The outer harness is what users build on top of the platform for their own system.
The same article splits harness controls into two kinds. Guides steer the agent before it acts (feedforward). Sensors watch the output after it acts and help the agent fix it (feedback). Both can run as deterministic code (computational) or as model judgment (inferential).
Our outer harness works the same way on every AI platform and operating system we use. Its main parts:
For this experiment we used a base version of the harness. We removed the code hosting, wiki, issue tracker, code quality, and log tools because they contain the reference service. Option A is not empty. Without the three platform skills, the model would know nothing about our backend service standards and the comparison would be meaningless. So this experiment compares the harness against minimum platform knowledge.
3. Methodology
3.1 Tasks
The source has two legacy projects: a selection screen with dozens of business rules and a preferences screen. The goal was to turn both into a single backend service. Each task was run in a single session, in three steps:
The prompts were exactly the same in both options. They did not mention the harness, any skill name, or any MCP tool. The name of the target platform and the task context appeared the same way in both. The harness had to bring its own mechanisms into play. Both options started from the same source code. The reference service, its source code, and the documents describing it were removed from the workspace.
3.2 Measurement
The three parts of M1:
4. Results
4.1 Cost and time
In the coding phase, Option A used 3.5M non-cached tokens over 274 minutes. Option B used 0.54M tokens over 49 minutes in the same phase. Option B also used less in the analysis phase (0.43M vs. 0.69M). In the feature phase, Option B took longer but used fewer tokens. In the bugfix phase, Option B used 1.9x more tokens (515K vs. 277K), because it found the second bug and fixed it (see 4.3).
The number of compactions is an important side effect. Compaction is when the platform summarizes and shrinks the conversation because the context is full. Option A had to compact its context 13 times during the coding phase, and each time it rebuilt part of its earlier decisions from the summary. More code means more context, and more context means more compactions and more re-reading. The harness saves cost by never starting this cycle. The same task can use up to 30x different token counts across runs (Bai et al., 2026), so our 2.7x gap should be read with that variance in mind.
4.2 Code quality
The code in both options follows the platform rules. That is expected, since the three platform skills were in both. Option B reached higher coverage with less code. Option A’s code grew because of the validation rules it copied into the backend service (see 4.3) and extra endpoints that carry the legacy screen’s state.
4.3 Correctness
Both agents found almost all of the legacy rules. The difference was in where they put them. In the legacy system, the validation rules lived in a validator class in the UI layer. The reference service did not move these rules into the backend service. It delegated them to a domain service that already existed. A domain service is the microservice that owns the business rules. The backend service only forwards the request to it and turns the roughly 30 error codes it returns into user messages.
- Option A ported the legacy validator into the backend service, method by method. The result behaves correctly and passes its tests. But the same rules now live in two places: the backend service and the domain service. When one changes, the other will fall behind.
- Option B kept only four UI-specific pre-checks in the backend service and delegated the rest to the domain service. The request it sends matches the contract the reference service uses almost field by field. The one gap: it does not send the first and last name fields of one person record. One of the reference service’s latest commits added exactly that.
How did Option B find the domain service? The reasoning steps in the session log show the chain clearly:
- In the first few minutes the agent looks at the topology: “topology says [shared backend service] migrates-from [legacy component]”. From this it learns there is a shared backend service, used by several screens, that already migrated part of the same legacy.
- It reads the shared backend service’s clients and finds the domain service’s validation endpoint.
- It applies the rule from the platform guide skill: “legacy service calls should be made from domain services, not from the backend service.”
The rule in step 3 was available to Option A too, because the same skill was in both options. The difference comes from two places:
So the harness gave the agent information that Option A also had, but at the right time and in the right context.
Get Alper M.’s stories in your inbox
Join Medium for free to get updates from this writer.
Feature and bugfix: In M2, both options matched all eight points of the new rule exactly as in the legacy. The rule was written in a single legacy method and both agents found it. Both also found and fixed the first bug in M3. They split on the second bug. Option B found a real bug in its own code: the process code was sent empty to the domain service, and in that case the service ran the default process. Option B derived the process code from the order type, the same fix as in the reference service. Option A said “no bug”, because in its own design the UI sends the process code. The team that built the reference service moved that responsibility into the backend service, because the UI knows the order type but not the process code. The harness-based architecture created a new kind of bug, but the same agent found and fixed it.
4.4 Impact of the harness
Impact analysis:
Pros and cons:
5. Discussion
Passing tests is not enough. Both backend services compiled and passed all their tests. A typical benchmark would count both as a success. The difference only showed up when we compared them with the reference service that is already running: Option A wrote a service that behaves correctly but does not fit our architecture. The copied rules work today, but the first change in the domain service will make the two copies drift apart. In enterprise migration work, agent output has to be judged not only by “does it work” but also by “is it in the right place”. Most benchmarks measure success by passing tests, so they cannot see this kind of difference.
When the agent gets the knowledge matters as much as whether it has it. The architecture rule was available in both options. Option A read it after writing most of the code; Option B read it at the start of analysis. A single line of routing in AGENTS.md made that happen. When the agent saw the same information at the right moment, it made a different design. This fits Goedecke’s view that the human contribution is alignment, not capability: the model’s ability to write code was the same in both options. What changed was when and how our preferences reached the agent. One study found that repository-level context files raise cost by more than 20% without improving success (Gloaguen et al., 2026). We saw the opposite: cost went down and architectural correctness went up. The reason may be what the file contains. Our instruction file did not carry general coding rules. It carried a topology reference and skill routing, which is enterprise knowledge the model cannot work out on its own.
The most valuable knowledge is the knowledge that is not in the code. What led Option B to the right architecture was an infrastructure topology. This information is scattered across the code, but it is not written down in one place. The agent could have worked it out in Option A too, but it did not. The METR study found that experienced developers working in mature repos were 19% slower with AI tools (METR, 2025). We think that slowdown comes from a similar gap in unwritten knowledge. Preparing this kind of list is a fairly small job, but its effect spreads across the whole coding phase. This also fits the SkillsBench finding that focused skills work better than large skill bundles (Li et al., 2026): in our case too, a few focused components made most of the difference.
The cost gap comes from a chain. The wrong architecture means more code. More code means more context and more frequent compaction. After each compaction the agent rebuilds earlier decisions from the summary and reads more files. 13 of Option A’s 17 compactions happened in the coding phase. The harness does not save cost step by step; it saves cost by never starting this chain. The Scaffold Effect study, which compares different harnesses, also shows that token usage can vary up to 40x while success rates barely move (Vats and Golev, 2026). In our case the difference showed up not only in tokens but also in the architecture the agents built.
How should these results be read? This is a case study, with runs designed around each task. In agent evaluations, results are known to shift by a few points across repeated runs (Bjarnason et al., 2026), and the weights in our scoring rubric are our own choice. Still, the gap we saw does not look like random noise: the reasoning log shows, step by step, the information that made the difference and the moment it reached the agent. A natural next step is to repeat the experiment with more runs and on the other platforms the harness supports.
References
- Aycı, G. AI Agents Mental Model: Agentic is a spectrum. Substack, 2026. https://gonulayci.substack.com/p/ai-agents-mental-model
- Bai et al. How Do AI Agents Spend Your Money? arXiv:2604.22750, 2026.
- Becker, Rush, Barnes, Rein. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. METR, 2025.
- Bjarnason, Silva, Monperrus. On Randomness in Agentic Evals. arXiv:2602.07150, 2026.
- Böckeler, B. Harness engineering for coding agent users. martinfowler.com, 2026. https://martinfowler.com/articles/harness-engineering.html
- Deng et al. SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? arXiv:2509.16941, 2025.
- Gloaguen et al. Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? arXiv:2602.11988, 2026.
- Goedecke, S. Human-AI partnerships are for alignment, not capability. 2026. https://www.seangoedecke.com/human-ai-partnerships-are-for-alignment-not-capability/
- Li et al. SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks. arXiv:2602.12670, 2026.
- Vats, Golev. The Scaffold Effect in Coding Agents. arXiv:2607.22585, 2026.
