<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by John Munn on Medium]]></title>
        <description><![CDATA[Stories by John Munn on Medium]]></description>
        <link>https://medium.com/@johnmunn?source=rss-e26a476910a2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*yhI-uucNASxukWBOkNyUjA.jpeg</url>
            <title>Stories by John Munn on Medium</title>
            <link>https://medium.com/@johnmunn?source=rss-e26a476910a2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 20:35:10 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@johnmunn/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Code Was Always a Liability. AI Is Letting It Grow Faster Than Ever.]]></title>
            <link>https://medium.com/@johnmunn/code-was-always-a-liability-ai-is-letting-it-grow-faster-than-ever-d1777e1d0a17?source=rss-e26a476910a2------2</link>
            <guid isPermaLink="false">https://medium.com/p/d1777e1d0a17</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[John Munn]]></dc:creator>
            <pubDate>Tue, 16 Jun 2026 12:51:00 GMT</pubDate>
            <atom:updated>2026-06-16T12:51:00.604Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>What happens when software grows faster than organizational understanding?</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y7o91lgYXJVMyrKBzXUZlQ.png" /></figure><p>For years, the software industry has been focused on making software easier to create.</p><p>We built higher-level programming languages so developers could stop thinking about the underlying machine, and later the adoption of frameworks and cloud platforms helped remove alot of the repetitive work in application development and infrastructure management. Each wave of tooling lowered the amount of effort needed to transform an idea into working software, allowing organizations to expand what they could deliver without requiring a proportional increase in engineering effort.</p><p>AI is simply the next step in that evolution, although it is a far bigger leap than most of the advances that came before it.</p><p>The difference is that this time the improvement is measured in orders of magnitude rather than incremental gains. Features that once required weeks of development can now take only hours. Teams can stand up new services and prototype entirely new products in a fraction of the time those efforts once required.</p><p>Most conversations about AI focus on how quickly teams can turn ideas into working software. The discussion usually centers around speed and output, how to get the software to product, rather than what once it is in production.</p><p>I think the real conversation should be about ownership and the long-term consequences of dramatically increasing the amount of software teams can write.</p><h3>Every Line Is a Future Obligation</h3><p>While software can create enormous value, every line that enters production also creates an obligation that a team must carry forward. It becomes something that must be maintained, tested, secured, monitored, supported, and understood for years to come. Eventually it will need to be modified, replaced, or retired. The value generated by a feature may justify those costs, but the costs themselves never go away.</p><p>Historically, the relationship between software creation costs and software ownership costs remained somewhat balanced because software was expensive to write.</p><p>If a team wanted a new application, somebody had to spend a lot of time building it. That effort acted as a natural brake on complexity because every addition had to justify the engineering investment required to bring it into existence. Before software entered production, there was usually enough friction in the process for teams to ask whether the thing they were building was worth owning for the next five or ten years.</p><p>The cost of creating software is falling rapidly, while the cost of maintaining, understanding, and evolving that software remains largely unchanged.</p><h3>Complexity Has Always Compounded</h3><p>Programs don’t become hard to understand due to a single bad decision, they move toward confusion with each of the thousands of reasonable decisions that are made over time.</p><p>A feature is added for an important customer, a new service is introduced to support a business initiative, and years later all those decisions still live inside the program along with the assumptions they where built on, the dependencies they created, and the interactions they introduced when those decisions were made.</p><p>Every addition expands the amount of software the team must keep in mind. It increases the number of relationships between components and makes it slightly harder to predict the consequences of changes with-in the program.</p><h3>The Software Still Works</h3><p>When people think about code rot, they usually imagine technical failures such as outdated dependencies, dead code, missing tests, or documentation that no longer reflects reality. While, those problems still matter, I suspect the next generation of code rot will look very different from the traditional examples that engineers typically associate with an aging program.</p><p>Imagine an engineer trying to retire a service that has existed for six years. The original team is gone. The architect who designed it left years ago. The documentation suggests that the service is no longer required. Monitoring shows traffic flowing through the system, but nobody is entirely sure whether that traffic originates from customers or from another internal dependency. After two weeks of investigation, the service remains in production because removing it feels riskier than leaving it alone.</p><p>Nothing in this scenario is technically broken, and the software continues to function exactly as intended. But, problem is that the team no longer understands their systems well enough to make decisions about them.</p><p>Traditionally, we thought of code rot as something that happened to software itself. Dependencies became outdated. Tests fell behind. Documentation drifted out of sync with reality. I suspect the next generation of code rot will be less about the condition of the software and more about the condition of the organization’s understanding.</p><p>In many cases, the software itself is not deteriorating at all. Instead, the organization’s ability to explain why the software exists, who depends on it, or what would happen if it disappeared gradually erodes over time.</p><p>Historically, organizations accumulated these mysteries slowly, with a handful of forgotten services emerging over the course of several years. AI raises the possibility that organizations could create the same amount of future uncertainty in a matter of months simply because they can generate software so much faster than before.</p><p>That is the kind of code rot that concerns me most because it affects decision-making rather than functionality.</p><h3>The Real Risk</h3><p>The risk is not that AI generates bad code, although that possibility will always exist.</p><p>The more significant risk is that AI allows organizations to create software faster than they can manage the complexity that accompanies it.</p><p>For the first time, companies can dramatically increase the amount of software they own without making a corresponding increase in the number of people responsible for understanding how that software works and why it exists.</p><p>At first glance, that sounds like a productivity breakthrough capable of transforming software development.</p><p>It may also become an organizational memory problem that grows larger with every new feature, service, workflow, or automation that enters production.</p><p>Features remain in production because nobody knows who depends on them. Services continue running because ownership is unclear. Workflows survive because nobody is willing to risk removing them. Over time, more and more decisions become constrained by uncertainty rather than guided by understanding.</p><p>The consequences are not necessarily catastrophic outages or dramatic failures.</p><p>Instead, organizations may experience a gradual decline in their ability to move quickly because every change requires more investigation and more caution than the one before it.</p><p>Teams become increasingly hesitant because nobody fully understands the blast radius of a change. Features remain in production because removing them feels risky. Services continue running because nobody can confidently say they are no longer needed. Every modification demands additional analysis because understanding has become the scarce resource within the organization.</p><p>As a result, organizations may find themselves generating software faster than ever while simultaneously becoming slower at evolving the systems they already own.</p><h3>Won’t AI Understand It?</h3><p>Whenever I bring this up, someone eventually argues that AI will understand the complexity for us and eliminate much of this concern.</p><p>Perhaps it will help, and there is good reason to believe it will become increasingly useful in this area.</p><p>AI is already remarkably good at reading code, tracing execution paths, identifying dependencies, and explaining behavior in ways that would have seemed impossible only a few years ago.</p><p>The problem is that software consists of far more than source code.</p><p>Software is also the accumulation of customer commitments, business decisions, operational compromises, regulatory requirements, historical context, and years of organizational knowledge that often exists outside the repository.</p><p>Questions such as why a service exists, which customer depends on a particular behavior, why a workaround was introduced, or what would happen if a component disappeared tomorrow are often far more important than understanding the implementation details themselves.</p><p>Most of that knowledge does not live in source control, architecture diagrams, or technical documentation.</p><p>Instead, it lives in conversations, institutional memory, and the people who were present when those decisions were made.</p><p>The challenge is that people leave. Teams change. Priorities shift. Organizational memory fades over time.</p><h3>Stewardship Becomes the Job</h3><p>This is why I believe the role of senior engineers and architects is becoming more important rather than less important.</p><p>For years, engineering leadership focused on helping organizations build faster because the primary constraint was the effort required to create software.</p><p>The challenge ahead may be helping organizations remain understandable even as the volume of software they produce continues to increase.</p><p>That requires treating complexity as a cost rather than an inevitability. It requires being willing to delete software that no longer serves a purpose and questioning whether every new service, workflow, abstraction, or agent truly deserves to exist.</p><p>The organizations that manage complexity most effectively may not be the ones that generate the greatest amount of software. Instead, they may be the organizations that become disciplined about removing software that no longer creates meaningful value and preserving understanding around the systems that remain.</p><p>Most importantly, it requires recognizing that every line of code generated today becomes something future engineers will inherit.</p><h3>The New Bottleneck</h3><p>The software industry has spent decades trying to make software easier to create, and by most measures that effort has been extraordinarily successful.</p><p>What AI may reveal, however, is that creating software was never the hardest part of the problem, even if it consumed most of our attention for many years.</p><p>The harder challenge has always been preserving enough understanding to safely evolve what we have already built while continuing to adapt it to changing business needs.</p><p>As a result, the central question is shifting from “Can we build it?” to “How do we prevent complexity from growing faster than our ability to understand it?”</p><p>The reason this matters is that code does not become truly dangerous only when it breaks. It becomes dangerous when nobody can confidently explain why it still exists, who depends on it, or what consequences would follow if it were removed.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d1777e1d0a17" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Documents as Code]]></title>
            <link>https://medium.com/@johnmunn/documents-as-code-c216d0cb4b6a?source=rss-e26a476910a2------2</link>
            <guid isPermaLink="false">https://medium.com/p/c216d0cb4b6a</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[developer-experience]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[John Munn]]></dc:creator>
            <pubDate>Thu, 21 May 2026 12:46:00 GMT</pubDate>
            <atom:updated>2026-05-21T12:46:00.488Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>When Product Specs Stop Guiding Code and Start Generating It</strong></h4><figure><img alt="A lone developer sits at a desk between two worlds. On the left, chaotic product and collaboration artifacts float through the darkness: sticky notes, sketches, Slack-like messages, Confluence-style documents, and abandoned diagrams glowing in warm orange and red light. On the right, cool blue folder structures and structured document trees flow into clean code repositories and organized system architecture. Streams of digital energy connect both sides, converging behind the developer to symboli" src="https://cdn-images-1.medium.com/max/1024/1*kToFRzE_U0QkTJ_9EuBTeQ.png" /></figure><p>A couple months ago I was chatting with an friend whose company had started dipping their toes into AI-assisted development. They were trying their own version of spec-driven development, but they had run into a problem. Product and commercial teams were still working out of Confluence, Jira, Slack, Figma, and whatever other odds and ends had piled up over the years while the engineering work lived in Git Repos.</p><p>Before AI, that setup mostly worked because developers knew where to look for the requirements and specs they needed to complete the work. They also, knew which Jira tickets were old, which Confluence pages were out of date, and which Slack thread had the final decision on the important project questions.</p><p>The problems started because the AI agents they were using needed context. Since the tooling didn’t have access to Confluence or Jira, the team created a product spec document directly inside the repo to give the agents that project context. The goal was to let developers pull it locally and keep implementation consistent across individuals.</p><p>But this workaround caused unforeseen issues. An engineer pulls a ticket and uses a Cursor agent to code the work in the ticket, using that repo spec as a project guideline. Meanwhile, the PM changes a requirement in Confluence and sends a message in the project Slack channel about it. The developer might see the message and update the ticket, but the codebase spec now has to be edited just to keep the agent’s context up to date</p><p>This all came to a head when a Developer implmenting an unrelated bit of work, unknowling had an agent re-implment a feature that had been removed a day before. The feature made it live before the PM noticed it.</p><p>A big meeting was held to figure out how this could have happened and that is when the problem was surfaced. The team had two sources of truth which had become desynced.</p><p>For most of software history, Product and Engineering were separated. And the places that held our work reflected that.</p><p>Back in waterfall days, none of this really mattered because the teams barely interacted anyway. Specs got written, handed to engineering, and eventually turned into software.</p><p>Agile changed the people side of it. Product sat closer to engineering. Designers got embedded into teams and everybody talked more.</p><p>But the artifacts themselves never really merged.</p><p>Instead, developers acted as human bridges carrying context between mismatched systems. We used to consult the Confluence documents manually and turn them into tickets. Backlog grooming meetings were used to update those tickets to reflect the changes in product documents. We relied on those rituals and meetings to mitigate the distance between where the documents lived and where the work actually got done. All of that worked fine as long as a person was doing the actual mental work to make sure the specs and the code stayed aligned</p><p>AI tooling completely changes what a document does inside our development workflows. Specs and architecture notes were always meant to guide the work, but they used to rely on humans to interpret them, spot any gaps, and apply common sense. Now, those files are steering automated code generation. A wrong document can quietly break a feature because the agent doesn’t stop to ask questions a human would. The moment your docs start driving production code without a human buffer, they inherit the exact same engineering responsibility as the codebase itself. That’s the real weight behind “documents as code.”</p><p>The way we solve the two sources of truth problem isn’t to force PMs to write markdown documents or pretend Git can solve organizational design issues. It’s about accepting that if an artifact is going to feed an AI tool directly, it has to be treated like code. It needs to be versioned, it needs to be tracked, and it can’t just live in a siloed wiki that someone updates whenever they remember to.</p><p>An LLM has no concept of a document’s lifecycle. When requirements inevitably pivot mid-sprint, those updates are captured in places like Confluence, Slack, or Jira. A human engineer knows that a pivot happened, so they mentally override the text in front of them because they know it just hasn’t been updated yet. But an agent won’t realize that a pivot happened and its spec doc isn’t up to date. It treats the lagging markdown file as the sole source of truth, blindly writing code for features the team decided to cut twenty-four hours ago.</p><p>This isn’t a debate about whether docs should live in Git. This is about when the way we work changes and the documents we use to complete the work need to be in two places at once.</p><h3>Authority</h3><p>When you shift to spec-driven development with AI, you are forced to confront a question most companies have safely ignored for years.</p><blockquote><em>Where does the final authority over your system behavior actually live?</em></blockquote><p>In a traditional workflow, authority is fluid and human. The same feature might exist in a dozen mismatched places at once, like the PRD, the Jira ticket, a Figma mock, and a Slack thread. Humans navigate this mess using social consensus. If a spec doc in the repo contradicts a recent Slack thread, a developer applies common sense, asks a teammate, and just ignores the outdated document.</p><p>But when you feed that document into an AI agent, that fluid human authority disappears. The model doesn’t understand social consensus, and it doesn’t know that a Slack conversation overrides a piece of the spec doc in the repo. It just treats whatever text it can see as the source of truth.</p><p>By turning documents into the direct fuel for automated code generation, we lose the human buffer that can reason over what is the current state of what we are trying to build. Without that buffer, you can no longer afford ambiguity. You have to decide which document is actually allowed to define what we are building.</p><h3>Multimodal Context</h3><p>And all this gets stranger once you move beyond text documents.</p><p>A lot of product decisions don’t live in documents. They live in Figma files, screenshots, walkthrough videos, architecture diagrams, and all the other visual junk companies accumulate over time. Those systems were built assuming humans would consume the output, but now AI systems are consuming them.</p><p>A Figma file might contain approved designs sitting beside abandoned ideas, hidden layers, half-finished experiments, and flows nobody ever cleaned up. In a traditional workflow, a designer just says, “ignore that frame, that’s old,” and the problem goes away. The model can’t do that unless the system around it makes those states explicit.</p><p>That’s why context management is going to matter more than people realize. Right now, the entire industry is focusing on comparing LLM benchmarks and choosing models. But the model isn’t the true bottleneck. The bottleneck is that we are feeding incredibly smart models an uncurated archive of old specs, forgotten tickets, and dead diagrams, and then wondering why the agent built the wrong thing.</p><p>The hardest part of AI-driven development isn’t the AI. It’s the managing the context we give the AI.</p><h3>Early Signs</h3><p>You can already see parts of this change showing up across the industry. OpenAPI specs generate clients and tests automatically. Terraform turned static infrastructure definitions into live, executable systems years ago. Design tools are starting to expose highly structured, component-driven context directly to agents.</p><p>Artifacts that used to merely describe software are starting to influence systems directly. The boundary between documentation and code is completely dissolving.</p><h3>Organizational Compression</h3><p>This structural shift is inevitably going to change the shape of engineering organizations.</p><p>As code generation becomes a commodity, understanding the precise business context around the software becomes the real differentiator. Engineers are being pulled closer to product decisions because the implementation is getting easier to generate. Product teams are being pulled closer to engineering constraints because their docs, diagrams, and workflows are now affecting production code more directly.</p><p>For years, there was a clean separation between the people deciding <em>what</em> the product should do and the people actually implementing it. That separation becomes impossible to maintain once your implementation tooling reads directly from your planning systems.</p><h3>What Happens Next</h3><p>Nobody fully knows where this all ends. The repository spec problem teams are running into right now feels like a minor logistical problem, but it won’t stay small for long.</p><p>The moment you build a workflow where implementation tooling reads directly from organizational artifacts, your technical debt becomes operational debt. A requirement changes but the repository spec doesn’t. A deprecated flow stays inside your retrieval pipeline. An old diagram gets treated like live architecture. An AI agent blindly implements a ticket the product team junked two weeks ago.</p><p>Humans have spent decades compensating for these internal communication gaps. The tooling cannot. Most companies were never designed for a world where organizational context shapes software execution without a human buffer filtering out the noise.</p><p><strong>Enjoyed this article? Here are more you might like:</strong></p><ul><li><a href="https://medium.com/@johnmunn/keep-simple-things-simple-let-complex-things-earn-their-complexity-c51dff7a826c">Keep Simple Things Simple. Let Complex Things Earn Their Complexity.</a></li><li><a href="https://medium.com/@johnmunn/mental-models-a-senior-engineering-leader-uses-5cebd38717ec">Mental Models a Senior Engineering Leader Uses</a></li><li><a href="https://medium.com/@johnmunn/temporal-leadership-the-skill-of-when-not-just-what-2599cecff09e">Temporal Leadership: The Skill of When, Not Just What</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c216d0cb4b6a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Keep Simple Things Simple. Let Complex Things Earn Their Complexity.]]></title>
            <link>https://medium.com/@johnmunn/keep-simple-things-simple-let-complex-things-earn-their-complexity-c51dff7a826c?source=rss-e26a476910a2------2</link>
            <guid isPermaLink="false">https://medium.com/p/c51dff7a826c</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[design-thinking]]></category>
            <category><![CDATA[leadership]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[John Munn]]></dc:creator>
            <pubDate>Thu, 07 May 2026 12:46:01 GMT</pubDate>
            <atom:updated>2026-05-07T12:46:01.200Z</atom:updated>
            <content:encoded><![CDATA[<h4>Why most systems get complicated long before they need to</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YDH__9cMM0a7T_8zy0x4_w.png" /></figure><p>I keep coming back to the same idea.</p><blockquote><em>Keep simple things simple and let complex things earn their complexity.</em></blockquote><p>I didn’t really set out with that in mind, but the more I think about it, the more it feels like it makes sense. I’m not against complexity at all. Some problems are very hard, some systems need real sophistication, and sometimes you just don’t get a choice. What I’ve seen more often, though, is complexity showing up before the problem actually has a need for a it. complex solution. After a while you start to recognize the pattern, things get harder not because the business needed it, but because someone wanted to work on something more interesting.</p><h3>The Gatsby Lesson</h3><p>Several years ago we decided to try headless WordPress, this was back when Gatsby.js was one of the main options and well before Next.js became the default solution for this. We pulled it into our stack and started building sites with it, even though, in hindsight, it was early enough that the Gatsby team probably would have told us to wait before using it in production.</p><p>The thing is, the sites we were building were not complicated. They were mostly brochure-style sites on WordPress with Timber and Twig, and they all sat behind a CDN. So, we already had something fast and predictable, aswell as a tech stack the team actually understood end to end. Nothing was broken with the sites, but the newer approach felt more modern and more aligned with where the industry seemed to be heading, so we went ahead anyway.</p><p>The system didn’t just collapse,but it did degraded over time as build times ticked up, edge cases started appearing in deployments, and debugging became harder because the surrounding ecosystem wasn’t mature, and we were dealing with all of that with only a couple of people on a team who really understood how Gatsby worked.</p><p>The part that reallys sucked was, the friction and instability did not just make development harder, it changed the level of faith the rest of the business has in our engineering department. We had promised better performance, and easy to work with sites, we delivered none of this.</p><p>Looking back, the uncomfortable part is that we did not introduce that complexity to solve a real problem. We took something that was already working and made it harder to work with, mostly because Gatsby felt more interesting then our old approach.</p><h3>Complexity Often Arrives Socially Before It Arrives Technically</h3><p>I think this is where a lot of organizations get themselves into trouble.</p><p>Complexity does not always enter because systems demand it, and more often than not it shows up when people get restless with the ordinary work of maintaining stable systems and start looking for something that feels more interesting to work on.</p><p>When the bread-and-butter work starts to feel routine, there is a shift that happens in the team. People start wanting to try something new, pull in a tool they saw somewhere, and before long you’re making decisions based more on what feels interesting than what’s actually needed.</p><p>You see a version of this everywhere once you start looking for it. Teams reach for Kubernetes before they need it. Products that are still pretty linear end up with event-driven archetectures. AI features show up in tools because, everything needs a chat bot now a days.</p><p>The tricky part is that complexity can look impressive. It reads sophisticated and the modren approach, that makes it harder to question, even when it isn’t really improving anything.</p><p>We need to remember, complexity is never free. Each abstraction layer adds another place where ownership can fail. Workflows add coordination overhead, and new systems come with some operational weight that someone has to carry.</p><p>And eventually simple things start becoming surprisingly hard, and that is usually the point where you realize something drifted further than it should have.</p><h3>You Can Feel It When Simplicity Is Gone</h3><p>One of the clearest signs of unnecessary complexity is when basic work suddenly requires navigating systems instead of solving problems.</p><p>You look at a task and it feels harder than it should be, which usually shows up in ways you can’t ignore, like when something simple turns into a chain of steps and dependencies that were never part of the original problem. Sometimes onboarding drags on for weeks because the local environment depends on a dozen interconnected services that all have to be understood before anything works.</p><p>Sometimes a deployment needs approvals and extra steps just to fix something small. Or a developer hits an issue and realizes only a couple of people can understands the system and what is happening. Or departments stop talking directly and everything gets routed through a ticketing system that was supposed to make things easier.</p><p>You can often measure organizational complexity by watching how basic communication flows.</p><p>If someone from Department A cannot walk over and talk directly to Department B, and instead has to go through managers, tickets, portals, workflows, or layers of process, complexity has entered the system.</p><p>Now, sometimes that complexity is necessary, especially as scale increases, compliance requirements grow, and organizations become more distributed, which naturally introduces more structure whether you intend it or not.</p><p>But the important question is always:</p><blockquote><em>Is this complexity solving a real problem, or are we maintaining complexity because it already exists?</em></blockquote><p>That ends up being a very different kind of conversation when you actually think about it.</p><h3>Complexity Should Match the Shape of the Problem</h3><p>When I talk about simplicity, I do not mean stripping everything down to the bare minimum or pretending hard problems do not exist. Some systems absolutely outgrow simpler approaches.</p><p>I worked on projects involving live game data flowing across dozens of websites where different sites needed different subsets of information delivered in near real time. At that point, introducing things like message queues and more advanced infrastructure actually made sense because the problem itself had changed shape. The old approaches were starting to crack under the pressure.</p><p>That difference shows up pretty clearly when you’ve been on both sides of it.</p><p>It was one of those cases where the complexity wasn’t coming from some big forward-looking idea, it was just the system reacting to what the problem had actually become.</p><p>To me, simplicity really just means the solution fits the problem cleanly. You are not adding extra systems, abstractions, workflows, or operational burden unless the pressure of the problem genuinely requires them.</p><p>You start noticing the same kinds of patters repeating, things like scaling bottlenecks, production getting a bit more fragile, issues that take longer to track down, and infrastructure costs creeping up. You also feel it in deployments getting riskier and coordination taking more effort than it used to.</p><p>At some point the existing approach just stops fitting the problem. You feel it in the friction, in the workarounds, in that nagging sense that things that used to be straightforward now take more effort than you expect. That’s usually when adding complexity starts to make sense. Not because it’s interesting, but because the problem has changed and the old way is starting to become simple to handle the problem.</p><h3>AI Is About to Accelerate This Problem</h3><p>I think AI is going to make this worse, not because there is anything inherently wrong with the technology, but because it removes a lot of the friction that used to slow complexity down. In the past, adding a new layer or system took time, people who actually understood it, plus the effort to keep the whole thing running day to day, which meant you felt the cost early. Now you can generate whole systems and stitch together integrations or even spin up infrastructure almost instantly, so it is much easier to introduce something complicated without really noticing what you have taken on.</p><p>You can already see it in how products are evolving. Features get added because they are available, not because anyone stopped to ask if they were needed. Like bolting a chatbot onto something so users can ask questions the interface should answer on its own. At the same time, the tools developers use make it easier to add systems they do not fully understand, which means the complexity shows up both in the code itself and in how many moving pieces get introduced at once. It all looks productive in the moment, the code compiles, everything looks fine on the surface. Underneath, ownership starts to muddle, the architecture drifts a bit, and over time people understand less of the system until you end up with something that works but is hard to fully understand.</p><h3>The Arms Race Problem</h3><p>There is also a social side to this that organizations rarely talk about, and you start to notice it not in architecture diagrams but in behavior. Over time it shows up in how teams talk and what they reach for by default, and it slowly shapes what starts to feel like the “right” way to build something.</p><p>You might see one team bring in Kafka because they want to move toward event-driven systems, and that makes sense in their context, but then another team feels the pull to introduce their own version of something modern, and before long a third team is trying something else entirely. None of these decisions are necessarily wrong on their own, but they rarely happen in isolation.</p><p>What you end up with is a mix of patterns and systems, along with a pile of assumptions that were all chosen at the team level. Eventually those decisions start to bump into each other. Teams try to stitch everything together, and that is usually when the cracks start to show, because the edges between these systems were never designed to meet.</p><p>At that point the organization stops feeling like a coherent platform and starts feeling more like a set of local decisions that all made sense when they were made but do not quite line up anymore, and instead of helping you move faster it just gets in the way.</p><h3>The Real Cost of Unearned Complexity</h3><p>If organizations do not learn this lesson, the impact tends to build gradually rather than all at once. Things start taking longer than they should, more people get pulled into work that used to be straightforward, and you notice a kind of background fragility where small changes have a habit of breaking something somewhere else. Over time that wears people down, because instead of solving problems they are spending more and more energy dealing with the system itself.</p><p>What I am really interested by, is what it does to the harder work, because when everything is complicated it becomes difficult to take on the problems that actually need real thought and care, and before long most of your time is going into things that should have been simple, which slowly chips away at the organization’s ability to deal with anything that is truly difficult.</p><p>At that point, change itself starts to feel risky. Every adjustment carries the possibility of something cascading in a way no one fully understands. You end up in a place where the system is technically capable of more, but practically stuck.</p><p>That’s usually where my head ends up. Not in trying to avoid complexity, but in making sure there’s still room for it when it actually matters. Because if everything around you is already heavy, already tangled up, then when something genuinely difficult shows up you don’t really have anywhere to put it. And a lot of what passes for complexity in the meantime doesn’t really deserve the weight we give it.</p><p><strong>Enjoyed this article? Here are more you might like:</strong></p><ul><li><a href="https://medium.com/@johnmunn/aiops-wont-save-you-clean-ops-will-d46637414099">AIOps Won’t Save You. Clean Ops Will.</a></li><li><a href="https://medium.com/@johnmunn/youre-solving-the-wrong-problems-well-57e8d465f343">You’re Solving the Wrong Problems Well</a></li><li><a href="https://medium.com/@johnmunn/why-simple-local-code-will-always-outperform-smart-architecture-7ad26949595a">Why Simple, Local Code Will Always Outperform ‘Smart’ Architecture</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c51dff7a826c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[You’re Solving the Wrong Problems Well]]></title>
            <link>https://medium.com/@johnmunn/youre-solving-the-wrong-problems-well-57e8d465f343?source=rss-e26a476910a2------2</link>
            <guid isPermaLink="false">https://medium.com/p/57e8d465f343</guid>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[engineering-mangement]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[leadership]]></category>
            <dc:creator><![CDATA[John Munn]]></dc:creator>
            <pubDate>Thu, 30 Apr 2026 12:41:01 GMT</pubDate>
            <atom:updated>2026-04-30T12:41:01.410Z</atom:updated>
            <content:encoded><![CDATA[<h4>Why strong engineering teams are missing what matters</h4><figure><img alt="A neatly organized project roadmap with completed tasks and checkmarks sits above a tangled network of strings and notes representing confusion, misalignment, and disconnected work beneath the surface." src="https://cdn-images-1.medium.com/max/1024/1*ZcuzOEpCgYPX4fVS_3h4Aw.png" /></figure><p>Some teams look like they’re doing everything right, but when you investigate a bit further something doesn’t quite add up. A feature hits sprint planning. The team walks through how they’ll build it, handles the weird edge cases, cleans up the tickets, and leaves with a plan. Everything about it looks right, but something is still off.</p><p>No one really talks about why the feature exists, at least not in a way that actually stays with the team once the work starts. It’s doesnt show up in the ticket, or in the PR, By review time, the reason for it is gone.</p><p>This isn’t a one-off problem. It’s a drift that happens slowly enough to feel normal. At some point I realized we didn’t really understand the company we were in, or the work we were doing. Decisions made sense in isolation, but not when you stepped back and looked at what actually mattered.</p><h3>Alignment doesn’t feel like process</h3><p>When things are actually aligned, it doesn’t feel like coordination or ceremony. Engineering and product aren’t negotiating priorities; they sit down, look at the same problem, and work through it together until it makes sense. People ask better questions because they understand the context, and decisions don’t require as much effort because the goal is shared.</p><p>Alignment is when, you don’t need to spend time convincing anyone something matters, because they already know what matters.</p><h3>Growth introduces distance</h3><p>Teams don’t start misaligned. When a company is young, it is small and scrappy, everyone knows everyone else, and this makes it easy to keep alignment locked in, but as the company grows, that proximity fades; more people join, layers take shape, and the distance between decisions and outcomes stretches further with each step.</p><p>Leaders begin to shape their own areas around what they believe is important. Some of that lines up with the company’s direction, some of it doesn’t, and over time the gaps widen. Process creeps in to manage the complexity, conversations turn into handoffs, and people become careful about stepping on each other instead of using friction to find better solutions.</p><p>There isn’t one moment where it fails. Things just get harder to follow, and teams keep shipping work that looks right but doesn’t really change anything.</p><h3>We solve the wrong problems well</h3><p>Most engineering leaders don’t struggle with the technical side of the job. If anything, that’s the part we’re most comfortable with. We’re trained to see system problems, uptime, performance, scalability,and we’re good at solving them.</p><p>What gets missed is the cost of those decisions outside the system: the time they take, the money they consume, the focus they pull away from other work, and whether the team can realistically support what they build six months later.</p><p>I inherited a project that had been running for three years. The goal was to save product hours, and on paper it made sense. When I looked at it more closely, even if we shipped that day, it would have taken roughly six years to earn back what we’d spent building it. And it still wasn’t done.</p><p>Technically sound, but economically wrong.</p><p>We shut it down.</p><h3>Bad bets don’t look like mistakes</h3><p>We made an early bet on Gatsby when it was still new. It looked like the right move at the time, better performance, a more modern stack, a step forward from what we had.</p><p>What we actually got was instability, long build times, and a developer experience that degraded as the system grew. Product teams lost confidence in it, and within a year we were backing out of the decision.</p><p>The bigger cost wasn’t the rework. It was the loss of trust. That one decision made it harder to introduce similar ideas later, even when the technology had improved. It took years before we could even suggest trying something like Next.js again.</p><p>It wasn’t just that the technology didn’t work. We took on risk the company didn’t need at that point in time.</p><p>On the other side, we had constant pressure to move off WordPress to something more modern. That instinct wasn’t wrong, but the justification was often thin. Each time it came up, the same question had to be answered in practical terms: what do we actually gain, and when do we gain it?</p><p>Does it help us make money faster, open a new path, or remove a real limitation we’re hitting today? If not, it doesn’t matter how good the stack is in theory. Technology doesn’t get a pass just for being better on paper.</p><h3>Simple problems get overbuilt</h3><p>There’s a pattern in software where something useful becomes harder to use over time. A React dropdown library I remember started simple and solved one problem well. As more people adopted it, more features were added, more edge cases were handled, and the API grew until you needed documentation just to use it.</p><p>It stopped being simple.</p><p>We repeat that pattern in our own systems. Straightforward problems get layered with abstractions and infrastructure they don’t need, while the harder kind of complexity gets avoided,the complexity of how work moves through the company, how decisions connect across teams, and how product and engineering stay aligned as the organization grows.</p><p>That’s the complexity that actually matters, and it’s the one we tend to under-invest in.</p><h3>Aligned teams work differently</h3><p>The difference shows up in what gets talked about. Aligned teams don’t limit themselves to implementation details; they keep the goal visible in planning, in tickets, and in pull requests. You can see not just what is being built, but why it exists and what it’s supposed to change.</p><p>That context changes how work gets reviewed. You’re not only asking whether the code works,you’re asking whether the solution serves the intent. Without that, reviews become purely technical, and the opportunity to correct direction gets lost.</p><p>Aligned teams also avoid working in isolation. Engineering and product plan together, adjust together, and challenge each other when something doesn’t line up. When teams start making decisions unilaterally, alignment has usually already drifted.</p><h3>Serving the company isn’t passive</h3><p>Saying that engineering exists to serve the company can be misread as a call to agree with everything. It isn’t, serving the company means engaging with it, pushing back when something doesn’t make sense and bringing a perspective that others don’t have.</p><p>It also requires dropping the idea that engineering is separate from “the business.” That language creates distance and turns collaboration into negotiation. The best teams don’t operate with that split; they’re working toward the same outcome from different angles.</p><h3>The part that actually matters</h3><p>Technology is rarely the thing that makes a company win. Plenty of companies have stronger stacks than the ones that succeed, and it doesn’t change the outcome.</p><p>What matters is whether the team understands what they’re building and what it’s meant to do, and whether their decisions line up with that understanding.</p><p>The job isn’t to build the most advanced system. It’s to build the one that lets the company move.</p><p><strong>Enjoyed this article? Here are more you might like:</strong></p><ul><li><a href="https://medium.com/@johnmunn/stop-saying-the-business-e3506fde1b87">Stop Saying “The Business”</a></li><li><a href="https://medium.com/@johnmunn/proactive-engineering-5752e66f7940">Proactive Engineering</a></li><li><a href="https://medium.com/@johnmunn/planning-fights-are-really-about-time-169eb091a564">Planning Fights Are Really About Time</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=57e8d465f343" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[We Lost a Developer Because of Our Stack]]></title>
            <link>https://medium.com/@johnmunn/we-lost-a-developer-because-of-our-stack-4dd89898c256?source=rss-e26a476910a2------2</link>
            <guid isPermaLink="false">https://medium.com/p/4dd89898c256</guid>
            <category><![CDATA[hiring]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[engineering-mangement]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[developer-experience]]></category>
            <dc:creator><![CDATA[John Munn]]></dc:creator>
            <pubDate>Wed, 22 Apr 2026 12:31:01 GMT</pubDate>
            <atom:updated>2026-04-22T12:31:01.539Z</atom:updated>
            <content:encoded><![CDATA[<h4>Why system could cost you your best people</h4><figure><img alt="A minimalist illustration of a developer standing in a dim workspace, looking out a large window at a city skyline during sunset, with a computer showing code on a desk nearby and soft, muted colors creating a reflective mood." src="https://cdn-images-1.medium.com/max/1024/1*TlEuIkrFwf5Fzg5J1GsIxQ.png" /></figure><p>It was a Wednesday in the spring, rain tapping against the boardroom windows. I was sitting across from one of my top developers, going through an exit interview.</p><p>I asked why he was leaving.</p><p>He paused for a second, then said, “The tech stack. It’s not what I want to be working on. It’s not helping me grow my skillset.”</p><p>Two months earlier, he and one of our architects had come to me with a proposal. Rebuild the backend, move off PHP, onto Python.</p><p>At the time, it didn’t really make sense. The system worked, we had features to ship, bugs to fix, and a roadmap that was already tight. A full rebuild with no clear tie to revenue wasn’t something we could justify.</p><p>But in that earlier conversation, he mentioned other developers felt the same way. The team wasn’t excited about the system anymore and it felt dated. I had been seeing something similar while hiring too, we weren’t attracting the people we wanted, and I had a feeling the stack was part of it.</p><p>Sitting there in that exit interview, I realized I hadn’t weighed the impact of the decision properly. I’d been focused on cost, delivery impact, system risk, all the obvious things, and that part made sense at the time, but I hadn’t thought about what it was doing to the people working on the system.</p><p>Truthfully, I wasn’t really thinking about the impact on our people at the time, so it just felt like an easy call to make then. Looking back on it now, it probably wasn’t as simple as I thought.</p><p>If you leave the system as it is, things keep moving. It’s predictable, and you don’t have to deal with the disruption that comes with trying to rebuild something that already works, which matters when the business depends on consistent output.</p><p>Over time though, hiring gets harder, strong candidates lose interest when they see the stack, and the team starts to disengage because they want to work on a modern system, not one that they feel is old and dated.</p><p>If you move to something newer, you get a different set of problems, hiring usually gets easier and people are more engaged, there’s more curiosity in the work again, but delivery slows down while the team is getting up to speed on the new stack and bugs start showing up in places that used to be stable. So you end up dealing with issues you didn’t have before while still being expected to deliver the work that was already planned.</p><p>There are times when staying put makes sense, if the system is doing its job, the team is shipping, and reliability matters more than change, introducing risk just to feel current usually backfires, and I’ve seen teams start pulling apart stable systems because they didn’t want to be working in that stack anymore, and then spend the next year rebuilding a lot of the same things anyway.</p><p>Other times, you’re still on the old stack and nothing is breaking, but hiring gets harder, candidates pass when they see it, people leave for work they’d rather be doing, and the day-to-day work gets less exciting, features take longer and fewer people are into what they’re building.</p><p>At that point someone usually asks if we should just rewrite it, which sounds reasonable in the moment but isn’t really the right question.</p><p>Most teams go to extremes and either try to keep everything or replace everything, but in practice it usually ends up somewhere in between, you don’t need to rebuild the foundation to make progress, you can introduce new services in a different stack, build new products on something more current, and give the team exposure to better tools without putting the entire platform at risk.</p><p>That doesn’t remove the tradeoff, it just keeps it manageable, and timing matters more than people want to admit, because even if a change is the right call, the business still has to be able to carry it, and if delivery pressure is high and the roadmap can’t slow down, a large migration creates problems on its own, while the same decision made later, with more room, can play out very differently.</p><p>At a certain point, the decision stops being technical. It becomes a balance between system health, business pressure, and team health. Push too far in one direction and something else starts to slip.</p><p>Explaining that balance to non-technical leadership is a different problem. Talking about modern stacks or what developers want to learn doesn’t usually come across, it just sounds like preference.</p><p>What people do pay attention to is impact.</p><p>If hiring is getting harder, if strong candidates are passing, if you’re at risk of losing experienced developers, those are business problems. They affect delivery in a real way.</p><p>It also helps to be clear about scope, what stays the same, where the new tech lives, and how the risk is being managed, because it’s easier to deal with when you know what’s changing and what isn’t.</p><p>Looking back on that conversation, approving a full rebuild wouldn’t have been the right call. Shutting it down because the revenue case wasn’t obvious wasn’t right either.</p><p>What I should have asked was where we actually needed to change to keep both the system and the team in a good place.</p><p>Not a full rebuild, but not ignoring it either, just enough change to keep delivery moving and still ship what we had planned without breaking what already worked. I wasn’t thinking about it like that at the time.</p><p><strong>Enjoyed this article? Here are more you might like:</strong></p><ul><li><a href="https://medium.com/@johnmunn/everyone-deserves-to-know-where-they-stand-ce13cd757637">Everyone Deserves to Know Where They Stand</a></li><li><a href="https://medium.com/@johnmunn/you-arent-leading-you-942ffdfb3796">You Aren’t Leading You</a></li><li><a href="https://medium.com/@johnmunn/holding-engineering-teams-accountable-for-delivery-0312701b56cf">Holding Engineering Teams Accountable for Delivery</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4dd89898c256" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Everyone Deserves to Know Where They Stand]]></title>
            <link>https://medium.com/@johnmunn/everyone-deserves-to-know-where-they-stand-ce13cd757637?source=rss-e26a476910a2------2</link>
            <guid isPermaLink="false">https://medium.com/p/ce13cd757637</guid>
            <category><![CDATA[career-development]]></category>
            <category><![CDATA[management]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[feedback]]></category>
            <category><![CDATA[leadership]]></category>
            <dc:creator><![CDATA[John Munn]]></dc:creator>
            <pubDate>Thu, 16 Apr 2026 12:18:28 GMT</pubDate>
            <atom:updated>2026-04-16T12:18:28.074Z</atom:updated>
            <content:encoded><![CDATA[<h4>Setting clear expectations is a leadership responsibility.</h4><figure><img alt="Two colleagues stand in front of a planning board covered with notes, checkmarks, and charts, discussing expectations and progress in a bright office, symbolizing the importance of clear leadership communication and alignment." src="https://cdn-images-1.medium.com/max/1024/1*bs7-oM_QqsLbcRCaQ029tA.png" /></figure><p>Everyone deserves to know where they stand, and not just in a document, job description, or buried inside a quarterly performance review.</p><p>They need to hear it directly from you as their leader, in a real conversation. That’s part of the obligation, and I think a lot of leaders are missing it.</p><h3>The cost of getting it wrong</h3><p>I wasn’t always clear about what I expected from my reports.</p><p>Because of that, people were left trying to piece together what mattered from context clues. They would overanalyze passing comments, what I praised in meetings, or whatever I happened to focus on in our 1:1s, and then build their own picture of success from that.</p><p>This all came to a head in a conversation with one of my reports. I sat him down and told him he was performing below expectations, and he looked at me like I had two heads. He was genuinely stunned.</p><p>He immediately started listing off all the things he had been doing to move what he thought was my vision forward, and to be fair, he had been working hard. The problem was all of that hard work had been towards the wrong goal.</p><p>He missed the mark, but the failure was mine.</p><p>I had never made the expectations clear enough for him to know what he was supposed to be doing.</p><p>When expectations are vague, people become interpreters. They start reading signals and optimizing around them. They focus on what gets praised, what gets noticed, and what seems to matter most to their leader that week, and over time that creates politics, anxiety, and wasted effort.</p><p>The hardest part is that most leaders think they are being clear. The expectations feel obvious because they are formed in their heads, but that clarity never really makes it to the team.</p><h3>Why leaders avoid setting expectations in the first place</h3><p>I think a lot of leaders avoid this because setting expectations forces them to confront their own lack of clarity.</p><p>Sometimes they don’t actually know what the bar should be.</p><p>No one above them ever made it clear either, so what gets passed down is a weird mix of instinct, what got rewarded last quarter, and whatever the business happens to be shouting about right now. It turns into this inherited chaos where everyone feels like they know what the target is, but no one can really articulate it.</p><p>I’ve also seen leaders pull back the second something starts looking like a metric, because people learn how to optimize for the metric instead of the value underneath it. Once you’ve watched that happen a few times, it’s easy to start convincing yourself that vagueness is safer.</p><p>The thing people really don’t like admitting, though, is that clear expectations create clear misses.</p><p>The moment the bar is visible, everyone can see when it isn’t being hit, and now the leader has to step into the conversation instead of letting ambiguity carry it for a little longer.</p><p>A lot of managers would rather wait, gather more evidence, hope the problem fixes itself, or tell themselves it’s still too early to say anything.</p><p>But that waiting is usually what turns a small course correction into a much harder conversation later.</p><h3>The trap on the other side</h3><p>When leaders do set expectations, they tend to overcorrect into hard metrics.</p><p>And then Goodhart’s Law kicks in.</p><p>When a measure becomes the target, people optimize for the number instead of the outcome behind it. The unmeasured work starts getting pushed aside, and people begin chasing the kinds of wins that used to lead to recognition, even if the business needs something different now.</p><p>I saw this play out with a senior developer on my team.</p><p>His managers loved him. They told me his code was chef’s kiss, that he was dependable, and honestly that he was easy to manage. That had already earned him a promotion before I stepped into the role.</p><p>The disconnect was that my bar for a senior developer was different.</p><p>I needed my seniors to do more than write strong code. They needed to help junior and intermediate developers level up, be a presence across the engineering organization, work comfortably with product and other departments, and help move our technology forward.</p><p>From where he was standing, it made complete sense why he stayed focused where he did. The leaders around him had consistently reinforced that this was what great looked like.</p><p>The real miss was higher up the chain. We weren’t aligned on what “senior” was supposed to mean.</p><h3>Ownership, not scorecards</h3><p>The best expectation you can set usually isn’t a metric, it’s a domain.</p><p>I like to give someone something real to own. That ownership is going to look different depending on the persons level. A senior engineer might own SLA performance across multiple properties, while a junior developer might own PR flow and make sure reviews aren’t sitting too long and work keeps moving. The scope changes with the role, but the important part is that both people know there is a real part of the system they are accountable for.</p><p>I saw this clearly when I brought an intern onto the team.</p><p>One of my managers pulled me aside because he wasn’t sure if the way I usually operate, expecting everyone to own something, was going to work with her. In his mind she was too junior and too new to really own anything meaningful yet.</p><p>So I gave her ownership of the new employee onboarding process.</p><p>She ended up doing an amazing job with it. She started by talking to team members about what felt out of date in the onboarding flow, then went back through the Confluence documentation and rebuilt it. She removed old setup snippets that no longer matched the codebase, replaced them with working examples, and even put together a small script that let new hires run most of the setup steps from the terminal instead of piecing it together manually.</p><p>That was a good reminder that deciding too early what someone can and can’t own is its own kind of unclear expectation setting. You end up putting a ceiling over someone before they’ve had a chance to show you what they can do.</p><p>Once someone owns something real, the conversation gets easier because the expectation is already sitting there in front of both of you. You can focus on making it better instead of spending half the 1:1 trying to align on what matters.</p><p>I find this is a much more useful kind of accountability because it gives people something they can reason about, improve over time, and grow their judgment inside of.</p><h3>Two kinds of expectations</h3><p>There are two different things I’m actually managing when it comes to expectations.</p><p>The first is what I need from the team as a unit. Delivery cadence, how they work, where the company is trying to go. I set those as constraints, but I don’t tell the team how to meet them. That’s their job. A good team lead takes those constraints and turns them into something the team actually owns. How they’ll communicate with product, what they’ll build to move the needle, how they’ll hold each other to it. That translation layer is where team accountability actually comes from. When it’s missing, everything flows back through the leader, and that’s a problem.</p><p>The second is what I need from each person individually, and that differs from person to person.</p><p>While holding everyone to the same expectations sounds fair, I don’t think it is.</p><p>When you do that, you’re fitting people into a mold instead of figuring out who they actually are and what they’re capable of. I think about it like a baseball team. Not everyone’s a pitcher. Some people are outfielders. If you try to make every outfielder pitch, your team suffers, and so does the outfielder, because they’re spending all their energy trying to be something they’re not instead of getting better at what they’re actually good at.</p><p>The job is figuring out where someone excels and where they need to grow, and building around that. Make your people the best outfielder they can be, don’t try to force them to be a mediocre pitcher.</p><h3>You should revisit your expectations</h3><p>I had a developer on my team they were solid, reliable, consistent. Then two sprints in a row his cadence was off. It wasn’t like him, so I sat him down and asked what was going on. His son had been sick and he had been staying up late nights with him.</p><p>I asked him what cadence he could honestly commit to while he was going through it. I let him set the expectation. I asked when he thought he’d be back to full capacity, and he gave me a timeline. Over the following weeks I checked in, not to pressure him, just to see where he was at. And when his son got better, his output came back.</p><p>A lot of leaders use expectations as a stick. Miss them and you’re the problem. I’ve never seen that work well. I see them more as a conversation starter. Life happens, and people go through seasons. The job isn’t to catch someone falling short, it’s to stay close enough to know why, and whether it matters.</p><p>That’s what expecations are about, they’re not a performance plan. They’re the start of conversations that can help catch small things before they become large ones. Failure rarely happens all at once, there is always a series of smaller problems before the big one hits. It’s your job is to notice these early and ask “if this keeps going, where does it lead?”</p><p>Not every miss warrants a conversation. The way I think about it is risk and impact. A genuine one-off that was low stakes might just be noise. A one-off that could have been catastrophic is an early warning sign whether or not it repeats.</p><p>Say a team member has been missing their cycle time metric. You ask why, and they say the QA engineer didn’t get a ticket through in time and they didn’t want to bother them. Some managers hear that and will move on. But if you own something, you own it. That means at minimum making contact with the blocker. This could be as simple as messaging the QA engineer, flagging the risk. If you reached out and they were unavailable to work on your problem and you flagged it with your manager, you did your job. If you didn’t reach out at all, you absorbed the problem and hoped it resolved itself.</p><p>One of the best questions I’ve learned to ask is</p><blockquote>“What do you think I expect from you?”</blockquote><p>Sometimes the answer aligns with what I actually expect. Sometimes it’s nowhere close. Either way it’s useful, because now the missalignment has been surfaced and we can work on fixing it.</p><p>If someone does keep missing the mark, the first question isn’t whether to act. It’s whether you’ve actually done the work, given the feedback, had the conversations, figuring out what’s actually in the way. Most performance problems aren’t capability problems, It’s a clarity problem. If you’ve done that honestly and the gap is still there, that’s a different conversation. But most leaders have that conversation before they’ve earned the right to.</p><h3>Final thought</h3><p>People can handle high standards. What they struggle with is invisible standards.</p><p>People deserve to know where they stand. They deserve to know what good work looks like and to be able to align themselves to it. That’s on us as leaders, we need to understand what’s going on in their lives, what they’re capable of, and where they can grow. To set the standard, be clear about it, and stay close enough to know when it needs to change.</p><p>Everyone deserves that, not just the high performers. Everyone.</p><p><strong>Enjoyed this article? Here are more you might like:</strong></p><ul><li><a href="https://medium.com/@johnmunn/you-arent-leading-you-942ffdfb3796">You Aren’t Leading You</a></li><li><a href="https://medium.com/@johnmunn/holding-engineering-teams-accountable-for-delivery-0312701b56cf">Holding Engineering Teams Accountable for Delivery</a></li><li><a href="https://medium.com/@johnmunn/legacy-isnt-the-product-you-ship-it-s-the-people-you-grow-be850d7581ca">Legacy Isn’t the Product You Ship. It’s the People You Grow</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ce13cd757637" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Challenge Your Technical Architects So They Actually Grow]]></title>
            <link>https://medium.com/@johnmunn/how-to-challenge-your-technical-architects-so-they-actually-grow-7b14dc075b78?source=rss-e26a476910a2------2</link>
            <guid isPermaLink="false">https://medium.com/p/7b14dc075b78</guid>
            <category><![CDATA[career-growth]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[engineering-leadership]]></category>
            <category><![CDATA[leadership]]></category>
            <dc:creator><![CDATA[John Munn]]></dc:creator>
            <pubDate>Tue, 07 Apr 2026 12:41:01 GMT</pubDate>
            <atom:updated>2026-04-07T12:41:01.513Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Why great architecture leaders stop teaching patterns and start expanding the tradeoffs</em></h4><figure><img alt="A senior engineering leader stands with hands on hips facing a large whiteboard covered in architecture diagrams, service flows, and system dependencies, while several team members watch from a conference table behind them. The scene emphasizes architectural complexity, leadership guidance, and the shift from clean system design to broader organizational thinking." src="https://cdn-images-1.medium.com/max/1024/1*jWcnG7eKj8yRz-ng4bDnWw.png" /></figure><p>I worked with an architect once who was at their best in classic layered systems.</p><p>The whole thing had a reassuring shape to it.</p><p>You had your controllers where they belonged, services sitting cleanly beneath them, and the data layer tucked exactly where anyone on the team would look first. Requests moved cleanly from one layer to the next, and when something went sideways, you could usually follow the thread back to the source without much drama.</p><p>Then the company started to outgrow that shape.</p><p>The issue was no longer whether the design was technically sound. It was that too much of our delivery speed depended on everyone moving through the same synchronous chain. Teams were waiting on each other. Releases were tied together. One delayed response in the wrong place could slow work far beyond the system itself.</p><p>The move toward event-driven architecture was the right one.</p><p>Technically, the architect understood the pattern quickly.</p><p>Topics, consumers, retries, dead-letter queues, eventual consistency. None of that was the real challenge.</p><p>The hard part was helping them get comfortable in a world where cause and effect no longer lived in the same moment.</p><p>A customer action would fire in one place, the state would settle somewhere else, and a downstream workflow might not wake up until much later. A team they did not manage owning the consequence.</p><p>That was the growth moment. It was not about learning events. It was about learning what asynchronous systems do to ownership, failure, team boundaries, and business patience.</p><p>That is when it really clicked for me, for engineering leaders and org owners responsible for architecture capability, the fastest way to accidentally stall a strong architect is to keep challenging them only on technical depth.</p><p>Most senior architects are already good at system design. They can map service boundaries, reason through scale, and spot infrastructure risks before the rest of the room even realizes there is a problem.</p><p>That usually is not where their next level lives.</p><p>The real growth edge is what happens <strong>around</strong> the architecture.</p><p>The growth edge is in the tradeoffs they make, the business language they can speak, the ownership model they create, and how honestly they can reason about the cost of being wrong.</p><p>That is where strong architects become organizational force multipliers.</p><p>And this is where I think engineering leaders have a responsibility that often gets missed.</p><p>Most of the time, your architects are going to know more than you in the technical domain they live in, and that usually is not the problem you are there to solve.</p><p>The value you bring as the engineering leader is not superior technical depth. It is broader context.</p><p>You can see the business pressure they may not fully feel yet. You can connect org design to technical consequences. You can expose the hidden cost of ownership friction, delayed decisions, funding realities, and cross-team dependency risk.</p><p>Your role is not to out-architect them.</p><p>It is to help widen the system they are optimizing for.</p><p>Architects do not usually outgrow their blind spots on their own.</p><p>The same strengths that made them exceptional in one phase of their career can quietly become the thing that caps them in the next.</p><p>If all we do is reward them for being technically right, they will keep optimizing for technical rightness.</p><p>It is our job to help them expand the frame.</p><p>To expose them to business tension. To force conversations about ownership. To let them feel the consequences of org design, failure modes, and reversibility. To move them from designing systems to shaping how the company learns.</p><p>That growth rarely happens by accident.</p><p>It happens because someone in leadership deliberately creates the pressure, safety, and context for it.</p><p>I have found that the best way to help architects grow is to stop asking them how they would build something and start asking them what the system does to the people, the business, and the company’s ability to adapt.</p><h3>Push them past the clean design</h3><p>A lot of architects naturally drift toward the cleanest design.</p><p>It all looks good on paper, clean boundaries, smart abstractions, interfaces that seem ready for whatever might come next.</p><p>That is useful, but it is also incomplete.</p><p>The real question is rarely whether something is technically elegant. It is whether it is the <strong>right tradeoff for the moment the business is in</strong>.</p><p>That is where I like to apply pressure.</p><p>Instead of asking whether the design is good, I ask:</p><ul><li>Why is this the right tradeoff right now?</li><li>What becomes harder six months from now because of this choice?</li><li>What happens if this bet is wrong?</li><li>How reversible is this decision?</li><li>What is the smallest version of this that proves the idea?</li></ul><p>Those questions force architects out of purity and into judgment.</p><p>That is where maturity shows up.</p><h3>Make them translate architecture into business risk and leverage</h3><p>A lot of architects can explain systems beautifully to engineers.</p><p>Far fewer can explain them clearly to executives.</p><p>That gap matters.</p><p>A technical decision that cannot be translated into business language usually struggles to get the support, funding, or organizational patience it needs.</p><p>So one of the most useful growth exercises is simple:</p><blockquote><em>Explain this architecture decision to the CFO in 90 seconds.</em></blockquote><p>No infrastructure jargon. No Kubernetes detours. No long explanations about event buses.</p><p>Just:</p><ul><li>what risk this reduces</li><li>what speed this creates</li><li>what cost this avoids</li><li>what customer pain this removes</li><li>what optionality this preserves</li></ul><p>Once architects learn to do that, they stop being system designers and start becoming strategic operators.</p><p>The deeper version of this is anchoring architecture to the company vision.</p><p>One of the easiest ways for a strong architect to drift into over-engineering is to let the system become the mission.</p><p>The architecture starts optimizing for theoretical future scale, perfect extensibility, or edge cases the business may never actually face.</p><p>That is where engineering leadership has to reconnect the work to the company vision.</p><ul><li>What are we actually trying to become?</li><li>What does this product need to do in the next 12 to 18 months?</li><li>What customer or market move are we betting on?</li><li>What kind of speed does the business need from the teams building it?</li></ul><p>Those questions matter because the right architecture for a company trying to find product-market fit is very different from the right architecture for a company protecting scale, compliance, and operational resilience.</p><p>Sometimes the best growth conversation with an architect is not about a pattern at all.</p><p>It is about helping them see the vision of the work.</p><p>The system only needs to be sophisticated enough to serve the mission in front of us.</p><p>Anything beyond that can quietly become technical vanity.</p><p>Part of helping architects grow is teaching them to ask a harder question than, <em>Can we build this?</em></p><p>The better question is, <em>Does this level of architecture serve the company we are trying to become, or the one we are imagining in our heads?</em></p><p>A simple question I like to leave architects with in a 1:1 is:</p><blockquote><em>If the company vision is still true a year from now, what part of this design will feel obviously overbuilt?</em></blockquote><p>The most effective way to help an architect get there is rarely by giving them the answer.</p><p>Usually the better move is to ask the question that widens the frame, if the company vision is to become the fastest product in the category, how do we make sure this design does not quietly slow delivery six months from now? And if the vision is trust and reliability, where might this introduce failure modes that chip away at that?</p><p>The aim is not to steer them toward your answer.</p><p>It is to help them see the full system they are making decisions inside.</p><h3>Use disagreement as a growth tool</h3><p>Those questions do not always create instant alignment.</p><p>Sometimes the architect will still believe the technically stronger path is the right one.</p><p>That is not a problem to avoid.</p><p>Some of the best growth moments happen when both sides stay in the disagreement long enough to understand what each person is optimizing for.</p><ul><li>Are they protecting long-term system integrity?</li><li>Are you protecting company timing, capital efficiency, or market pressure?</li><li>Are they seeing operational risks that have not surfaced yet?</li><li>Are you seeing a business horizon the system does not need to solve for yet?</li></ul><p>The productive move is not winning the argument.</p><p>It is surfacing the assumptions underneath it.</p><p>Once those assumptions are visible, the disagreement usually becomes far more productive.</p><p>It stops being leader versus architect.</p><p>It becomes two people trying to better understand the system the company is actually inside.</p><p>The next part of that growth is helping them see what the architecture does to the people carrying it.</p><p>The best architects eventually realize they are not designing software systems.</p><p>They are designing <strong>human systems that happen to run on software</strong>.</p><p>Every architecture choice creates consequences for:</p><ul><li>team boundaries</li><li>ownership friction</li><li>incident response</li><li>onboarding speed</li><li>release confidence</li><li>dependency politics</li></ul><p>That means the conversation has to move beyond services and data flow.</p><p>I like to ask:</p><ul><li>Which team should own this?</li><li>Where does this create dependency bottlenecks?</li><li>Who becomes the human pager because of this decision?</li><li>Does this mirror our org in a healthy way?</li><li>Where will our team shape quietly make this harder than it needs to be?</li></ul><p>Sometimes the cleanest technical answer creates the messiest ownership model.</p><p>That is not good architecture.</p><p>A strong architect learns to optimize for <strong>clarity of ownership and confidence of execution</strong>, not just system performance.</p><p>A related part of that same conversation is helping them recognize when good architectural instincts drift into unnecessary centralization.</p><p>Senior architects often start to centralize intelligence. It is a natural instinct: shared gateways, central pipelines, smart platform layers, reusable abstractions, global policy engines.</p><p>These are all useful until they quietly become the place where flexibility goes to die.</p><p>One of the most valuable questions you can ask an architect is:</p><blockquote><em>Should this logic live in the middle at all?</em></blockquote><p>Sometimes the right growth move is forcing them to defend why something belongs in shared infrastructure instead of at the edge where the business context actually exists.</p><p>This is where you start sharpening architectural judgment instead of architectural cleverness.</p><p>The place I have seen this come together most clearly is when you give them a messy problem with no obvious technical answer.</p><p>The fastest way to grow an architect is to stop giving them architecture-shaped work and start giving them ambiguity.</p><p>Something like:</p><blockquote><em>Delivery is slowing down, incident coordination is messy, and roadmap confidence is dropping. Figure out whether this is architecture, tooling, process, or team design.</em></blockquote><p>Now they have to reason across the full operating reality of the org.</p><p>Not just APIs, queues, or deployment workflows.</p><p>They have to understand:</p><ul><li>communication paths</li><li>escalation friction</li><li>ownership gaps</li><li>platform DX</li><li>release psychology</li><li>political constraints</li></ul><p>That is where future principal engineers, directors, and CTOs begin to separate themselves.</p><h3>The destination this all points toward</h3><p>The biggest shift I have seen in great architects is this:</p><p>Early in their growth, they want to be right. Later, they want the <strong>organization to stay adaptable even when they are wrong</strong>.</p><p>That is a very different mindset. They start caring more about reversibility than certainty, optimize for learning speed, reduce blast radius, and create space for teams to move safely.</p><p>That is the level I try to coach toward.</p><p>Because the best architects are not the ones who create the most impressive systems.</p><p>They are the ones who help the company learn faster, recover faster, and change direction without tearing itself apart.</p><p>That is where architecture becomes leadership.</p><p><strong>If you like this article? Check out my articles:</strong></p><ul><li><a href="https://medium.com/@johnmunn/holding-engineering-teams-accountable-for-delivery-0312701b56cf">Holding Engineering Teams Accountable for Delivery</a></li><li><a href="https://medium.com/@johnmunn/the-case-for-dissent-8c69b563bcb3">The Case for Dissent</a></li><li><a href="https://medium.com/@johnmunn/if-youre-having-hard-conversations-all-the-time-you-re-the-problem-865a53fa62ff">If You’re Having Hard Conversations All the Time, You’re the Problem</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b14dc075b78" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Measure AI Productivity When the Cost Moves Instead of Disappearing]]></title>
            <link>https://medium.com/@johnmunn/how-to-measure-ai-productivity-when-the-cost-moves-instead-of-disappearing-9b70767b7f0e?source=rss-e26a476910a2------2</link>
            <guid isPermaLink="false">https://medium.com/p/9b70767b7f0e</guid>
            <category><![CDATA[engineering-mangement]]></category>
            <category><![CDATA[developer-productivity]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[leadership]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[John Munn]]></dc:creator>
            <pubDate>Tue, 31 Mar 2026 12:16:01 GMT</pubDate>
            <atom:updated>2026-03-31T12:16:01.329Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Why throughput alone lies, where AI quietly moves cost, and how engineering leaders can measure real leverage without creating theater.</em></h4><figure><img alt="Illustration showing AI’s impact on software engineering productivity, connecting workflow speed, code review, developer learning, codebase complexity, and executive ROI decisions through an interconnected futuristic system." src="https://cdn-images-1.medium.com/max/1024/1*0et8Pxjs5N7PHZWN1GfwCg.png" /></figure><p>Two weeks after an AI rollout, cycle time dropped 38%. We celebrated.</p><p>Then review queues doubled. The junior engineers were moving faster, but in design reviews their explanations started losing depth. Three months later, an incident review exposed the real problem.</p><p>We were moving faster, but it was getting harder to get changes safely across the line.</p><p>And I found myself wondering a much harder question:</p><p><strong>What does good performance actually look like in the age of AI?</strong></p><p>The moment leadership asks, “Are we getting ROI from AI?” most engineering orgs make the same mistake.</p><p>They start measuring usage.</p><p>Seat utilization. Prompt counts. Heavy users. Light users. Token burn. Acceptance rates.</p><p>Those numbers feel good because they’re visible. A dashboard shows up, everyone feels like they finally have control, and suddenly the room starts talking like the problem is solved.</p><p>But usage is not value.</p><p>Worse, the second AI becomes a KPI, teams stop optimizing for outcomes and start optimizing for <strong>proof of usage</strong>.</p><p>That’s where the measurement starts to distort behavior.</p><p>The issue isn’t measurement itself. The issue is that most teams are measuring the wrong layer of the system.</p><p>Good performance in the age of AI comes down to one thing: <strong>understanding how the shape of work shifted inside the engineering system once AI entered the workflow.</strong></p><p>That is the layer worth measuring.</p><h3>Start with your baseline, not someone else’s benchmark</h3><p>Before you can talk about improvement, you need to understand what healthy work already looks like in <em>your</em> system.</p><p>There is no universal good cycle time. There is no magic throughput multiplier. There is no industry-standard prompt-to-PR ratio that suddenly means AI is paying for itself.</p><p>A legacy monolith team, a platform group, and a greenfield product squad will all have completely different work signatures.</p><p>So start by baselining your <strong>natural work topology</strong>:</p><ul><li>Typical cycle time by work type</li><li>Average time spent in each workflow status</li><li>PR review loops</li><li>Escaped defects</li><li>Rework rate</li><li>Rollback frequency</li><li>Average PR complexity</li><li>Time for a new developer to reach useful contribution</li></ul><p>The goal is not to beat an industry number.</p><p>The goal is to notice when AI changes the system in a way that is either helping or quietly pushing cost somewhere else.</p><h3>The 5-lens framework for measuring AI productivity</h3><p>One thing that became obvious the first time I looked at this in a real team setting: the numbers only became useful once we could <strong>see where the cost moved</strong>.</p><p>We had a team whose cycle time improved almost overnight after AI tooling landed. On paper it looked like a win. More tickets moved. More PRs opened. Sprint throughput looked healthier.</p><p>But a week later PR review started ballooning.</p><p>Nothing was actually wrong with drafting speed. The problem was that the cost had shifted from <em>writing</em> to <em>verification</em>. Reviewers were now spending significantly longer validating logic paths, edge cases, and test assumptions because more code was arriving faster than human confidence could keep up.</p><p>Local speed improved. System throughput didn’t.</p><p>That’s the trap this framework is trying to expose.</p><p>The real question is not simply whether work moved faster.</p><p>It’s <strong>what changed, where the cost relocated, and whether the system got healthier or just busier.</strong></p><p>Once you have a baseline, the real question becomes:</p><p><strong>What changed in the shape of work?</strong></p><p>I like to look at that through five lenses.</p><h4>1) Flow lens: where did time compress, and where did it expand?</h4><p>Most leaders start and stop at cycle time.</p><p>That’s too shallow.</p><p>The better question is:</p><p><strong>Where did time compress, and where did it expand?</strong></p><p>For example, we might see backlog-to-PR shrink by 40% while review-to-merge quietly doubles.</p><p>That tells a much richer story than raw cycle time ever could.</p><p>AI may be helping teams draft faster while simply moving validation cost downstream.</p><p>Local speed went up. Global friction got worse.</p><p>That is not the same thing as productivity.</p><p>A healthy signal is:</p><p><strong>time compression without downstream queue inflation.</strong></p><p>If one workflow status suddenly starts growing, that’s usually where the cost moved.</p><h4>2) Quality lens: where did defects move?</h4><p>Faster ticket movement means almost nothing if the defect curve just moved two columns to the right.</p><p>Track:</p><ul><li>PR revision rounds</li><li>Escaped bugs</li><li>Hotfix frequency</li><li>Rollback events</li><li>MTTR</li><li>Incident volume tied to recent AI-heavy changes</li></ul><p>This is how you normalize speed gains against quality drag.</p><p>If tickets close faster but production instability rises, you are not measuring productivity.</p><p>You are measuring <strong>deferred cleanup</strong>.</p><p>That distinction matters.</p><h3>3) Learning lens: is judgment compounding?</h3><p>This is the one most teams miss, and it’s where the real 12-month cost usually hides.</p><p>AI can absolutely remove toil.</p><p>It can also remove <strong>desirable difficulty</strong>.</p><p>I’ve seen this show up in a very specific way: junior developers start shipping significantly more code, sprint metrics improve, and everyone feels like the leverage story is working.</p><p>Then three months later the same developers struggle to explain why a generated abstraction was chosen, repeat the same architecture mistakes in adjacent services, and become noticeably weaker in debugging conversations.</p><p>The ticket board improved. The engineering judgment curve flattened.</p><p>Most will mistake this for leverage, but it is really learning debt.</p><p>The system is only truly improving if <strong>developer judgment compounds alongside throughput</strong>.</p><p>Otherwise, you’re trading long-term capability for short-term board movement, and the bill usually arrives during the next major incident or redesign.</p><h4>4) Codebase lens: what long-term tax are we creating?</h4><p>This is where local speed can quietly become future drag.</p><p>I’ve watched this happen after an AI rollout where feature throughput looked fantastic for a few sprints.</p><p>The codebase started to grow fast, and even with solid reviews, things still slipped through.</p><p>Helper functions were being recreated left and right. Different services ended up with slightly different versions of the same utility. Patterns that looked fine locally were slowly making the repo harder to reason about globally.</p><p>The moment it really surfaced was during an auth bug. What should have been a quick trace took three engineers half a day because every service now had its own slightly different retry, helper, and logging chain generated over months of locally correct completions.</p><p>That is the real failure mode here.</p><p>Track:</p><ul><li>code duplication</li><li>pattern inconsistency</li><li>dependency sprawl</li><li>brittle tests</li><li>static analysis drift</li><li>documentation mismatch</li><li>architectural reversibility</li></ul><p>The real question here is:</p><p><strong>Is AI increasing local velocity while reducing global coherence?</strong></p><p>That is where teams get fooled into mistaking motion for leverage.</p><p>A sprint board can look incredible while the codebase becomes harder and harder to reason about.</p><p>That hidden complexity becomes interest payments on today’s speed, and usually shows up later as slower onboarding, riskier refactors, and incident debugging that feels more like archaeology than engineering.</p><h4>5) Economic lens: cost per durable outcome</h4><p>The economic signal usually lagged the delivery signal.</p><p>By the time the budget conversation happened, the team had already normalized to higher output. More tickets closed. More experiments shipped. More visible throughput.</p><p>But the harder question always came a few weeks later: <strong>was the extra work driving enough value to justify the growing cost of all the AI tools around it?</strong></p><p>The moment this usually became real was budget review.</p><p>AI spend had doubled quarter over quarter, but no one in the room could confidently tie it to lower incident load, faster onboarding, reduced review time, or better experiment win rates.</p><p>That uncertainty is the real failure mode in the economic lens.</p><p>Seat cost by itself is almost useless.</p><p>The metric that matters is:</p><p><strong>cost per validated useful change.</strong></p><p>That includes:</p><ul><li>seat spend</li><li>token spend</li><li>human review time</li><li>rework cost</li><li>incident cleanup</li><li>knowledge debt</li><li>onboarding drag from messy generated patterns</li></ul><p>This shifts the conversation away from “who are the heavy AI users?” and toward:</p><p><strong>Are we reducing the cost of durable outcomes?</strong></p><p>That is the ROI conversation leadership should actually be having.</p><h3>Segment by work type inside every lens</h3><p>This part is not optional. It needs to be part of the framework itself.</p><p>Every lens above should be measured <strong>by work type before you draw conclusions</strong>.</p><p>Otherwise the averages will lie to you.</p><p>AI impacts different work shapes very differently.</p><p>Measure each lens separately across:</p><ul><li>greenfield feature work</li><li>legacy maintenance</li><li>debugging</li><li>test generation</li><li>infrastructure</li><li>migrations</li><li>incident response</li><li>onboarding tasks</li></ul><p>AI might 3x test writing. It might barely move debugging. It might help migrations while hurting architecture.</p><p>Averages hide that.</p><p>Segmentation is where the real signal starts to show up, and once you can see that signal, the intervention path becomes much more obvious.</p><h3>The decision layer: what changed, and what do we do about it?</h3><p>Measurement only matters if it changes how you lead.</p><p>This is the intervention layer.</p><h4>If flow improves but quality worsens</h4><p>Drafting got faster. Validation got weaker.</p><p>Interventions:</p><ul><li>stronger PR review heuristics</li><li>test-first workflows for AI-heavy work</li><li>generated-code explanation requirements</li><li>move AI earlier into design and spec phases</li></ul><h4>If flow improves but learning drops</h4><p>AI is replacing productive struggle.</p><p>Interventions:</p><ul><li>AI-off first-pass problem solving zones</li><li>reasoning required in PR descriptions</li><li>architecture rationale checkpoints</li><li>reviews focused on tradeoffs, not syntax</li></ul><h4>If throughput rises but code entropy spikes</h4><p>You’re buying short-term velocity with long-term tax.</p><p>Interventions:</p><ul><li>shared scaffolding templates</li><li>architecture fitness functions</li><li>pattern libraries</li><li>refactor budgets</li><li>repo-specific AI rules</li></ul><h4>If cost rises faster than durable outcomes</h4><p>You’re buying rework with better marketing.</p><p>Interventions:</p><ul><li>move AI to the highest leverage phases</li><li>reduce low-value completion usage</li><li>role-specific workflows</li><li>separate drafting from production-critical use</li></ul><p>This is where measurement turns into real leadership judgment.</p><h3>The operating loop: from measurement to leadership intervention</h3><p>The real payoff is not the framework itself.</p><p>It’s the leadership behavior it enables.</p><p>What you are really building is a repeatable interpretation loop:</p><ol><li><strong>Baseline the natural shape of work</strong> so you know what healthy looks like in your system</li><li><strong>Measure the five lenses by work type</strong> so averages do not hide the movement</li><li><strong>Locate where cost moved</strong> review, rework, learning, architecture, incidents, onboarding</li><li><strong>Decide whether the shift is leverage or debt</strong></li><li><strong>Intervene at the layer creating drag</strong></li></ol><p>That last step is what matters.</p><p>If review queues are swelling, the intervention is validation design. If learning is flattening, the intervention is productive struggle. If entropy is rising, the intervention is architecture guardrails.</p><p>The measurement only earns its keep when it changes how you lead the system.</p><p>Because the ROI of AI is not whether developers touched the tool.</p><p>It’s whether the engineering system now produces:</p><ul><li>better outcomes</li><li>lower friction</li><li>stronger judgment</li><li>healthier code</li><li>more durable throughput</li></ul><p>without pushing the bill into a different quarter.</p><p>That is what real leverage looks like.<br>That is what is actually worth measuring.</p><p><strong>If you like this article? Check out my articles:</strong></p><ul><li><a href="https://medium.com/@johnmunn/your-ai-architecture-isnt-broken-it-s-just-put-together-wrong-c30db8dad5dc">Your AI Architecture Isn’t Broken. It’s Just Put Together Wrong.</a></li><li><a href="https://medium.com/@johnmunn/how-to-architect-secure-ai-agents-before-they-architect-your-incident-d8a5b82be308">How to Architect Secure AI Agents Before They Architect Your Incident</a></li><li><a href="https://medium.com/@johnmunn/where-ai-systems-drift-3756ca65c3cc">Where AI Systems Drift</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9b70767b7f0e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your AI Architecture Isn’t Broken. It’s Just Put Together Wrong.]]></title>
            <link>https://medium.com/@johnmunn/your-ai-architecture-isnt-broken-it-s-just-put-together-wrong-c30db8dad5dc?source=rss-e26a476910a2------2</link>
            <guid isPermaLink="false">https://medium.com/p/c30db8dad5dc</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-architecture]]></category>
            <dc:creator><![CDATA[John Munn]]></dc:creator>
            <pubDate>Tue, 24 Mar 2026 11:56:00 GMT</pubDate>
            <atom:updated>2026-03-24T11:56:00.533Z</atom:updated>
            <content:encoded><![CDATA[<h4>Where engineering leaders get their systems wrong</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sKw13dynInpeA1ZB_D8HxQ.png" /></figure><p>I’ve spent a fair amount of time over the past year looking at how teams are actually building with AI.</p><p>Not demos. Not prototypes. Real systems people are expected to rely on.</p><p>A pattern shows up pretty quickly.</p><p>These systems don’t break immediately. They break over time. They give just enough correct answers to feel useful, and just enough wrong ones to make people hesitate. You can watch the shift. At first people trust it. Then they double check. Eventually they work around it.</p><p>When that happens, the model gets blamed.</p><p>In most of the cases I’ve seen, it wasn’t the model, It was the design.</p><p>A few architectural patterns show up over and over. None of them are wrong. The problem is how they’re applied.</p><p>Part of why this keeps happening is subtle. These decisions don’t come from carelessness, but momentum. Teams reach for the pattern they saw most recently. The one that they read about or seen in a youtube video. I’ve seen “enterprise-grade” systems assembled from ideas someone picked up in a video a few weeks ago. The system gets built before the problem is shaped.</p><h3>Retrieval-augmented systems</h3><p>This is where most teams start.</p><p>Take a set of documents, chunk them, embed them, retrieve the most relevant pieces at query time, and pass that into the model.</p><p>On paper, it makes sense. The model stays grounded. No retraining. It works for simple questions.</p><p>The boundary shows up when the questions stop being simple.</p><p>If a question can be answered by a single paragraph in your data, basic retrieval will work. If it requires combining sources, comparing versions, or applying context over time, it won’t.</p><p>Most retrieval systems aren’t built for that. They’re built for similarity.</p><p>So you get a familiar pattern:</p><ul><li>a few chunks come back that look relevant</li><li>the model assembles something coherent</li><li>the answer sounds reasonable</li><li>but it’s incomplete, or slightly wrong</li></ul><p>I’ve seen teams respond to this by pulling in more context. It feels like progress. It usually makes things worse.</p><p>At some point the model isn’t reasoning over context, It’s just summarizing noise.</p><p>The issue isn’t retrieval itself, but treating retrieval as if it solves reasoning.</p><p>If the problem requires multi-step understanding, the system has to reflect that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/364/1*tF9_P7Kqs4_qwXGskPL0_w.png" /></figure><p>Most teams skip the decomposition step.</p><p>What they have isn’t a knowledge system. It’s a retrieval shortcut.</p><h3>Data-fed systems without a data model</h3><p>This one looks fine early on.</p><p>Documents get ingested. Embeddings get created. The pipeline runs. You can query it.</p><p>Underneath, the data is a mess.</p><p>The same concept exists in multiple places. Some versions are outdated. Some contradict each other. There’s no clear sense of which source is authoritative.</p><p>The model doesn’t know that.</p><p>It sees all of it and tries to produce a consistent answer.</p><p>That’s when you get responses that read well but don’t hold up. The language is clean. The reasoning is plausible. The facts don’t quite line up.</p><p>People call this hallucination. In most of the systems I’ve heard of, it wasn’t. It was inconsistent input.</p><p>A good data model does something simple but critical. It tells the system which version of a fact wins. It encodes authority, freshness, and ownership so retrieval returns a coherent view instead of a blend.</p><p>In practice, that usually means:</p><ul><li>attaching metadata to every source (timestamp, owner, audience)</li><li>defining a precedence rule (which source wins when they conflict)</li><li>filtering retrieval based on permissions and context</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/564/1*8tQturWR4ZbT5PrEaeuhQA.png" /></figure><p>There’s no notion of source of truth, freshness, ownership, or permissions.</p><p>The model doesn’t resolve that. It smooths it over.</p><p>Teams treat this as a pipeline problem. In practice, it’s a modeling problem.</p><p>Until the system knows what is true, it will keep producing answers that are merely consistent.</p><h3>Agent-based systems for structured problems</h3><p>At some point, teams decide to make it an agent.</p><p>The appeal is obvious. Instead of a fixed flow, the system can decide what to do. It can call tools, plan steps, and adapt.</p><p>Useful in the right context, but overused in most of them.</p><p>I was looking at a system not long ago where an agent had been added on top of a retrieval layer that wasn’t returning the right information. The agent would retry queries, call different tools, and eventually land on something usable. It looked intelligent. What it was actually doing was compensating for a broken retrieval step.</p><p>You end up with a loop that hides problems instead of surfacing them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*jeSjW4rHZFQT2bj8gO6YTQ.png" /></figure><p>When it works, it looks impressive. When it fails, it’s hard to diagnose. The same input can take a different path each time, and the reasoning is buried in intermediate steps.</p><p>For structured problems, the failure is predictable. You’ve replaced explicit decisions with implicit ones.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/130/1*4Cr0A6hxV9e2dy3u9clmOQ.png" /></figure><p>This isn’t flexibility versus rigidity, but visibility. In a workflow you can see where decisions are made and change them. In an agent loop you infer it after the fact.</p><p>Agents make sense when the path is unknown and the system needs to explore. They’re a poor fit when the path is known but hasn’t been made explicit.</p><p>If an agent is consistently “recovering” from earlier steps, that’s usually a sign those steps need to be fixed, not wrapped.</p><h3>Fine-tuned systems solving the wrong problem</h3><p>This shows up later, once everything else is in place.</p><p>The system is close. Outputs are decent, but inconsistent. The format drifts. The tone isn’t quite right.</p><p>So the next step is to fine-tune.</p><p>That works when the problem is behavior, but not when the problem is structure.</p><p>If your system retrieves the wrong data, combines conflicting sources, or executes the wrong flow, training the model harder won’t fix it.</p><p>What it does instead is make the system more rigid.</p><p>Now you’ve encoded assumptions into the model. Updating behavior requires retraining. Debugging gets harder because the logic isn’t entirely in the system anymore.</p><p>You end up with something that looks more polished but is still wrong in the same places.</p><p>Fine-tuning is good at shaping how the model responds, but It doesn’t change what the system feeds into it.</p><h3>A quick way to spot the mismatch</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/43ea7eee9506cdfc8ebcb839bf929aa6/href">https://medium.com/media/43ea7eee9506cdfc8ebcb839bf929aa6/href</a></iframe><h3>The question that gets skipped</h3><p>Before choosing a model, a tool, or a pattern, there’s a simpler question that rarely gets asked.</p><p>What kind of system is this?</p><p>That question isn’t about labels. It determines how the system behaves under pressure.</p><p>If it’s a lookup system, completeness matters less than precision. If it’s a reasoning system, retrieval has to be structured, not just relevant. If it’s a workflow, control and observability matter more than flexibility. If it’s exploratory, you accept variability and design around it.</p><p>Most of the failures above come from crossing those boundaries without noticing.</p><p>A lookup system asked to reason. A reasoning system built on inconsistent data. A workflow hidden inside an agent loop. A structural problem pushed into fine-tuning.</p><p>These aren’t dramatic mistakes. They’re small mismatches that compound as the system grows.</p><p>The model gets blamed because it’s the most visible part. The design set the constraints.</p><p>By the time you notice it, the system already has shape. And now you’re not fixing a model.</p><p>You’re unwinding a set of decisions you made before you understood the problem.</p><p><strong>If you like this article? Check out my articles:</strong></p><ul><li><a href="https://medium.com/@johnmunn/where-ai-systems-drift-3756ca65c3cc">Where AI Systems Drift</a></li><li><a href="https://medium.com/@johnmunn/the-temperature-trap-why-your-ai-keeps-giving-the-same-wrong-answer-8af29b3ced4c">The Temperature Trap: Why Your AI Keeps Giving the Same Wrong Answer</a></li><li><a href="https://medium.com/@johnmunn/temporal-leadership-the-skill-of-when-not-just-what-2599cecff09e">Temporal Leadership: The Skill of When, Not Just What</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c30db8dad5dc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Where AI Systems Drift]]></title>
            <link>https://medium.com/@johnmunn/where-ai-systems-drift-3756ca65c3cc?source=rss-e26a476910a2------2</link>
            <guid isPermaLink="false">https://medium.com/p/3756ca65c3cc</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[John Munn]]></dc:creator>
            <pubDate>Tue, 17 Mar 2026 12:11:00 GMT</pubDate>
            <atom:updated>2026-03-17T12:11:00.858Z</atom:updated>
            <content:encoded><![CDATA[<h4>and What to Fix First</h4><figure><img alt="Layered AI system diagram showing stacked stages labeled Data, Routing, Prompts, Retrieval, and Evaluation, with cracks and diverging arrows illustrating system drift and failure across layers." src="https://cdn-images-1.medium.com/max/1024/1*iSkj5Dw0jBZ_58leyhR8Lg.png" /></figure><p>Most AI systems do not fail in one dramatic moment.</p><p>They slip.</p><p>A user asks one thing. The application interprets it slightly differently. The prompt leaves room for ambiguity. Retrieval returns something related but not quite right. The model produces a confident answer. Output checks are weak. The problem surfaces only after users notice.</p><p>Then someone says:</p><blockquote>“We should fix the prompt.”</blockquote><p>Sometimes that helps. Often it does not.</p><p>The prompt is only one layer of the system. Failures can enter through routing, retrieval, model behaviour, or evaluation. Patching the prompt will not fix problems that start elsewhere.</p><p>Drift is rarely a model problem. It is usually a systems problem.</p><p>The moment you start thinking about failures this way, another question shows up:</p><p><strong>Where should the controls actually live?</strong></p><p><em>(I put together an interactive version of this if you want to explore how those failures compound across the stack. </em>→ <a href="https://johnmunn.tech/interactive/where-ai-systems-drift"><em>Open the interactive systems drift guide</em></a><em>)</em></p><h4>Identify the failure first</h4><p>Picture a support assistant. An engineer asks:</p><blockquote>What changed in the payment gateway after last week’s migration?</blockquote><p>The system returns a confident answer about the previous gateway configuration. It sounds right. It is wrong. The evidence was stale, the query was ambiguous, and nothing caught either problem before the answer reached the user.</p><p>This scenario shows up constantly in production systems. And it almost never has a single cause.</p><p>What looks like hallucination might be retrieval failure. What looks like retrieval failure might be poor task classification. Sometimes the model has not changed at all. The context did.</p><p>The same wrong answer can enter from several directions. Fixing the wrong layer leaves the others untouched.</p><p>Before choosing a fix, place the failure.</p><ul><li><strong>Wrong problem being solved </strong>→ check routing and task framing</li><li><strong>Right problem, wrong evidence</strong> → check retrieval</li><li><strong>Right evidence, unstable behaviour</strong> → check prompts and decoding</li><li><strong>Correct format, wrong truth</strong> → check validation and evaluation</li></ul><p>Most teams start with the model because it is the most visible part of the stack. The largest gains usually come from working earlier and later. Routing upstream. Validation downstream.</p><p>The rest of this article walks through each layer using that payment query as the running example.</p><h4>Data integrity</h4><p>Start here when the underlying data has changed.</p><p>Before routing, before retrieval, before prompts, there is data. And data changes. Schemas evolve. Source tables get renamed. A field that previously stored a transaction status now stores a transaction type.</p><p>The pipeline does not know. The retrieval layer does not know. The model certainly does not know.</p><p>In the payment example this is the failure that hides everything else. The migration changed how events were logged. The changelog still exists, but the schema it describes no longer matches the schema being queried. Retrieval returns documents that are technically valid and operationally obsolete.</p><p>This is different from retrieval quality. Re-ranking and query rewriting cannot fix documents built on a broken foundation.</p><p><strong>Schema and source monitoring<br></strong>When source systems change such as database schemas, API contracts, or event structures, downstream AI pipelines need to know. Treat schema changes the same way you treat dependency upgrades. Version them. Alert on them. Assign ownership.</p><p><strong>Data freshness checks<br></strong>Stale data and incorrect data are different problems, but they produce the same symptom. Build freshness checks into the pipeline. If the newest document in the retrieval index is three weeks old and the question is about last week, that gap should surface before generation begins.</p><p><strong>Provenance tracking<br></strong>Track where retrieved content came from and when it was written. Documentation created before a major infrastructure change should not silently support answers about the system after the change.</p><p><strong>Rule of thumb:</strong> if the data feeding the system is wrong, every layer above it inherits the problem.</p><h4>User intent and application controls</h4><p>Start here when the system attempts the wrong task.</p><p>The payment query runs into trouble immediately. “What changed after the migration” is actually several questions. Root cause analysis. Changelog lookup. Impact assessment.</p><p>Without decomposition the system picks one interpretation and commits. The engineer wanted all three.</p><p><strong>Task decomposition<br></strong>When a request spans multiple decisions or steps such as migration analysis, research workflows, or multi step assistants, break it into stages. Each stage receives a clear objective.</p><p>Ambiguity stops compounding.</p><p><strong>Intent classification<br></strong>When one interface handles multiple task types, classify before generating.</p><p>Examples:</p><ul><li>support assistants</li><li>knowledge copilots</li><li>workflow tools</li></ul><p>Without routing the model treats every question the same way.</p><p>Adding classification has a cost. It introduces an extra model call at the front of the pipeline. For high volume systems that latency matters. The typical solution is a smaller fast model for classification and a stronger one for generation.</p><p><strong>Structured inputs<br></strong>Missing context changes results. The payment query contains no timeframe, no system scope, and no definition of “changed”.</p><p>Collect those fields explicitly rather than letting the model infer them from prose.</p><p><strong>Policy enforcement and fallbacks<br></strong>Some decisions should not live in prompts.</p><p>Examples:</p><ul><li>refunds</li><li>approvals</li><li>regulated processes</li><li>escalation flows</li></ul><p>Models can assist with decisions. Boundaries belong in application logic.</p><p><strong>Rule of thumb:</strong> if the system should not have attempted the task at all, start here.</p><h4>Prompt controls</h4><p>Use prompt changes when the task is correct but behaviour varies.</p><p>Suppose the system correctly identifies the payment query as a changelog lookup. It still needs to know what a good answer looks like. How much technical detail. What format. Whether missing evidence should be flagged or ignored.</p><p>Those are prompt questions.</p><p>Prompts accumulate over time. Exceptions get added. Old examples remain. Rules begin contradicting each other. Teams often blame the model when the prompt has become unstable.</p><p><strong>System prompts<br></strong>Use system prompts for stable instructions such as:</p><ul><li>tone</li><li>evidence requirements</li><li>refusal behavior</li><li>escalation paths</li></ul><p>Avoid filling the prompt with every edge case. A prompt that tries to handle everything usually handles nothing reliably.</p><p><strong>Few shot examples<br></strong>Examples help when the expected output is subtle.</p><p>Common uses:</p><ul><li>specific summary formats</li><li>domain reasoning patterns</li><li>nuanced classification</li></ul><p>Examples show the behaviour instead of describing it.</p><p><strong>Output schemas<br></strong>When responses feed other systems, structure matters as much as content.</p><p>If the changelog answer needs to trigger a downstream ticket, a schema keeps the structure consistent even when the details change.</p><p><strong>Prompt cleanup<br></strong>Older systems carry years of accumulated prompt changes.</p><p>Typical problems include:</p><ul><li>repeated instructions</li><li>outdated examples</li><li>conflicting rules</li></ul><p>Removing unnecessary instructions often improves stability more than adding new ones.</p><p><strong>Rule of thumb:</strong> if the task is correct but behaviour shifts unpredictably, inspect the prompt layer.</p><h4>Retrieval controls</h4><p>Use retrieval changes when answers rely on the wrong evidence.</p><p>In the payment scenario the query enters retrieval as something like “payment gateway migration changes”. Vector search returns conceptually related documents, but the top results describe the previous architecture rather than the migration.</p><p>The ranking looks reasonable. The evidence is stale. Nothing in the pipeline signals a problem. The model generates an answer from the wrong documents because it has no way to know they are wrong.</p><p><strong>Query rewriting<br></strong>Engineers ask questions conversationally. Documentation rarely uses the same phrasing.</p><p>Rewriting the query to match how documents are written can change the retrieved material completely.</p><p>Example rewrite:</p><pre>gateway migration changelog week_of:&lt;date&gt;</pre><p><strong>Hybrid search<br></strong>Vector search finds conceptual matches. Keyword search finds exact identifiers.</p><p>Queries involving product names, error codes, version numbers, or migration identifiers need both.</p><p><strong>Re-ranking<br></strong>Retrieval sometimes surfaces the right documents in the wrong order.</p><p>Re-ranking improves which material actually enters the prompt. This becomes important in large documentation sets where older related material can dominate the ranking.</p><p><strong>Chunking and context management<br></strong>A passage may reference the migration while omitting the surrounding explanation needed to answer the question accurately.</p><p>Chunking that splits explanations mid context produces answers that appear grounded but are missing the detail that makes them correct.</p><p><strong>Rule of thumb:</strong> if answers reference the wrong source material, review retrieval first.</p><h4>Model and decoding controls</h4><p>Use these when the task and context are correct but generation still drifts.</p><p>In the payment scenario the query is correctly classified and retrieval returns the correct documents. The engineer asked a precise factual question. What changed in the system during a specific window.</p><p>That task does not benefit from variation.</p><p><strong>Temperature and top-p<br></strong>Lower temperature works best for constrained tasks:</p><ul><li>factual lookups</li><li>extraction</li><li>policy answers</li><li>code generation</li></ul><p>Higher temperature is useful for exploration or brainstorming.</p><p>These settings control variation and risk.</p><p><strong>Model selection<br></strong>Different workloads require different models.</p><p>Consider:</p><ul><li>latency</li><li>reasoning ability</li><li>cost</li><li>reliability</li></ul><p>A smaller model can handle classification cheaply while a stronger one handles synthesis. Splitting workloads often improves performance and cost.</p><p><strong>Tool use<br></strong>When exact results exist in structured systems, query them directly.</p><p>If the migration produced a structured changelog in a database, generation is the wrong instrument. The system should query the source of truth.</p><p><strong>Rule of thumb:</strong> adjust this layer only after reviewing earlier layers.</p><h4>Output and evaluation controls</h4><p>These controls detect problems and measure performance over time.</p><p>In the payment example the system produces a plausible answer about the previous gateway. Without output controls that answer ships to the user. The engineer trusts it and spends hours investigating the wrong system.</p><p>No bug report is filed against the AI. Confidence simply drops.</p><p><strong>Schema validation<br></strong>When responses feed other systems validate the structure first.</p><p>Structural errors can propagate through tickets, dashboards, and automation pipelines.</p><p><strong>Second pass review<br></strong>A second model pass can detect problems deterministic checks miss.</p><p>Examples:</p><ul><li>weak grounding</li><li>thin evidence</li><li>outdated sources</li></ul><p>The cost is latency. Response time increases. For high trust queries that tradeoff is usually worth it. For high volume systems sampling strategies often work better.</p><p><strong>Regression tests and drift detection<br></strong>Systems evolve constantly.</p><ul><li>prompts change</li><li>retrieval indices update</li><li>models are replaced</li></ul><p>Without regression tests those changes are invisible until users lose confidence. Tests make behavior comparable across time.</p><p><strong>Rule of thumb:</strong> if users discover failures before the team does, evaluation is missing.</p><h4>The real lesson</h4><p>Return to the payment query.</p><p>An engineer asks what changed in the gateway after last week’s migration. The source schema shifted during the migration and the pipeline was never updated. The system misclassifies the task, retrieves documents based on the old schema, generates an answer from outdated evidence, adds variation where none belongs, and passes the result through unchecked.</p><p>The failure spans multiple layers.</p><p>Fixing the prompt would not have caught it.</p><p>That is why the layered view matters. It separates different kinds of failure so they can be placed before they are fixed.</p><blockquote>“Fix the prompt” is not a strategy.<br>“Tune retrieval” is not a strategy.<br>“Add tests” is not a strategy.</blockquote><p>A strategy places the right control in the right layer.</p><p>Early systems depend on clever prompts.</p><p>Reliable systems depend on architecture.</p><p>Explore the layered model visually in the interactive guide:</p><p><a href="https://johnmunn.tech/interactive/where-ai-systems-drift">Visit the interactive guide!</a></p><p><strong>If you like this article? Check out my articles:</strong></p><ul><li><a href="https://medium.com/@johnmunn/the-drift-dilemma-why-great-models-go-bad-over-time-1e2132ffd562">The Drift Dilemma: Why Great Models Go Bad Over Time</a></li><li><a href="https://medium.com/@johnmunn/the-temperature-trap-why-your-ai-keeps-giving-the-same-wrong-answer-8af29b3ced4c">The Temperature Trap: Why Your AI Keeps Giving the Same Wrong Answer</a></li><li><a href="https://medium.com/@johnmunn/the-illusion-of-explainability-921e409fc0b4">Why Your AI Explanations Are Lying to You (And What to Do About It)</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3756ca65c3cc" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>