I started on an AS/400. Here is what AI-driven development actually changed
My first programming job was in 2012, writing RPG on an IBM midrange system. Green screens, five-character variable names, a compile cycle you waited on. I spent three years on that, then three more administering the box.
Fourteen years later I shipped a headless CMS with 126 endpoints, 17 NuGet packages and multi-architecture images, and I wrote very little of it by hand.
I want to be useful about what happened in between, because most of what I read about AI-driven development is either a sales pitch or a eulogy, and neither matches my week.
The hype, stated fairly
The claim is that a developer with an AI agent is several developers. Type what you want, get working software, ship faster.
The honest version of that claim is that it is partly true, and the part that is true is not the part being sold.
I do produce more. Last week I shipped a 4.0 release: a security audit that found and fixed four real defects, a capability permutation test, a 3.x upgrade path, multi-arch images, fourteen module packages. That is not a sprint’s work for one person at my old pace. It was days.
But “faster” is the wrong frame, and it is the frame that gets people hurt.
What actually changed: the bottleneck moved
Writing code was never the slow part. It felt like the slow part because it was the visible part.
The slow part was always knowing whether the code was right. AI did not remove that. It removed the thing standing in front of it, so now you can see it, and it is enormous.
Here is the difference in one sentence. When a person writes a bad test, it usually looks bad. When an agent writes a bad test, it looks excellent. Good name, clear arrange-act-assert, sensible assertion. It runs green. And it proves nothing.
That is the actual failure mode of AI-driven development, and I have not seen it described honestly anywhere. It is not hallucinated APIs, you catch those at compile time. It is plausibility. The work looks like the work.
Receipts
Every rule I now follow exists because something passed that should not have. A few from the last month, all from my own project.
A test that could not fail. A test suite went green over a release gate that had been a no-op for weeks. The gate checked a condition that was skipped by an earlier condition, so it never executed. It first ran on the day it mattered, during the release, and failed there.
A gate that could not pass. The opposite, same week. A check that compares a module’s source changes against its version compared against the wrong commit, so it failed on a clean checkout of the main branch and blocked every pull request. Both of these are the same defect wearing different clothes: a check whose reference point is something other than the thing it claims to be about.
A build that reported success while failing. I ran a Docker build piped into tail. It reported exit code 0. The exit code belonged to tail. The build had failed on a base image pull, and I only noticed because the image was not there afterwards.
Metadata I did not know I was shipping. Generated icons in my repository carried C2PA content credentials naming the tool that made them, 5,758 bytes per file. Every text-reading check in the repository was green over them, because none of them read binary. Three of five review agents found it. Two did not.
A fix that fixed the wrong race. I closed an issue about a database schema race by adding an advisory lock around the explicit schema apply. Mutation-tested, proven, merged. The race came back three days later, because the lock guarded one door and the ORM creates tables lazily through a second one. My test proved the lock worked. It could not prove the lock was in the right place, because it only ever called the locked path.
None of these were exotic. All of them looked fine.
The design I ended up with
I run Claude Code with a written method, because the thing that does not scale is me remembering the rules. It has been rewritten several times, always after something got through.
Get Baryo Dev’s stories in your inbox
Join Medium for free to get updates from this writer.
Triage by hand, not by agent. I read the open issues myself and sort each into a tier. Documentation and config are tier 0. A feature inside one module is tier 1. Anything touching core, auth, tenancy, data shape or an anonymous endpoint is tier 2. Spending an agent to decide what is cheap is how you spend more than you save.
The diff decides the tier, not the ticket. This one cost me. A ticket labelled “documentation” whose diff touches the permission resolver is not documentation. I have a script that reads the diff and prints every rule it fires: auth surfaces, raw SQL, schema and migrations, secrets and logging, concurrency primitives, deletion, workflow files, and a test diff that removes assertions. Any output means a reviewer runs. Checked against a real week, it fired on the six pull requests where defects were found and stayed quiet on the seven where reviewers found only wording.
A cheaper model drafts, a second one reviews, an expensive one only when the reviewer cannot close it. Not every task needs the best model. Most need the cheapest one that can be checked.
The mutation check is the gate. This is the most important thing I do. Before a pull request opens, the author names the production line each new test depends on. A script reverts that line and reruns those tests. They must fail. A test that passes either way is a gate failure, not a review nit.
I adopted that after a red team of my own process found that nine of twelve defects would have passed my previous gate. The sentence I wrote down at the time is the one thing I would keep if I lost everything else:
The gate cannot see what a test does not test.
Scripts do the mechanical checks, not agents. Build, named test classes, changelog validation, version checks, a locked-mode restore, a scan for banned characters, a duplicate-key parse of any edited workflow file. As agent instructions these cost tokens every time and get skipped under pressure. As a script they cost nothing and cannot be skipped.
Four changes in flight, not ten. I tried ten. The merge queue serialises them anyway, and every later merge needed the main branch merged in again and half needed lock files regenerated. The parallelism was fake and the reconciliation was real.
The numbers, since people ask
One night early on cost about 7 million tokens for 19 pull requests, roughly 370,000 each. Two agents per change, ten in flight, and every review comment became a new ticket to file. The method above targets under 150,000 for an ordinary change, with no ticket growth.
That is a real cost and I track it per batch. But it is not the interesting number.
The interesting number is that in one release cycle, adversarial review of my own code found four security defects I had shipped: a privilege escalation where an admin could grant themselves the top role, a cross-tenant read where any tenant administrator could read every tenant’s audit log, a placeholder signing key that booted in production because the check was length-only, and an unescaped field in an RSS feed. All four were mine. All four were found by pointing a machine at my work and telling it to be hostile.
That is the honest value proposition, and it is not “faster”. It is more scrutiny than I could afford before.
What I actually believe now
AI-driven development is real, and the productivity claim is measured on the wrong axis. I do not type less and finish sooner. I attempt things that were previously not worth attempting: a permutation test across every role and every capability, an adversarial security pass before a release, a full audit of what a repository ships in its binaries. Work I would have skipped and told myself was fine.
The skill that matters now is knowing what would prove it. Not writing the code, and not writing the prompt. Being able to look at a green test suite and ask what would have to be true for this to pass while the system is broken. That is a fourteen-year skill and it did not transfer from anywhere else. The people I would hire for this are not the fastest prompters. They are the ones who have been burned enough to distrust a green build.
Everything unverified is a claim, including mine. I have a standing rule that I trace a code path to its end before asserting anything publicly about it, because I made two confidently wrong public claims in one day and both were checkable. An agent will produce a confident summary of code it did not read carefully. So will I, at 11pm.
The tell is still there if you look. I strip em dashes from everything I publish, not out of style preference but because they are a signature. I strip provenance metadata from generated images. Not to hide anything, I am writing this article about exactly how I work, but because output that arrives pre-labelled with the tool that made it is output nobody edited.
If you are starting
Build the gate before you build the velocity. The order matters, because velocity without a gate does not feel dangerous. It feels great, right up until the release.
Start with one rule: for every test an agent writes, break the code it covers and watch the test fail. If it does not fail, you have a test that documents behaviour rather than defending it, and you will find out which at the worst possible moment.
Everything else in my method is an elaboration of that one habit.
The project all of this comes from is barakoCMS, an API-first headless CMS for .NET, open source under MPL-2.0. The issues, the failures and the gates are all public.
