<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by eamonn faherty on Medium]]></title>
        <description><![CDATA[Stories by eamonn faherty on Medium]]></description>
        <link>https://medium.com/@eamonn.faherty_58176?source=rss-0021d5c19a0e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*O8fgIB5BSJYTPoRs</url>
            <title>Stories by eamonn faherty on Medium</title>
            <link>https://medium.com/@eamonn.faherty_58176?source=rss-0021d5c19a0e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 20:02:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@eamonn.faherty_58176/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[I Generated 170,000 Tests by Accident. Here’s What That Taught Me About Coverage]]></title>
            <link>https://medium.com/@eamonn.faherty_58176/heres-the-updated-article-with-that-explanation-woven-in-naturally-d17a04a35661?source=rss-0021d5c19a0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/d17a04a35661</guid>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[software-testing]]></category>
            <category><![CDATA[formal-verification]]></category>
            <category><![CDATA[formal-methods]]></category>
            <dc:creator><![CDATA[eamonn faherty]]></dc:creator>
            <pubDate>Wed, 25 Mar 2026 20:29:01 GMT</pubDate>
            <atom:updated>2026-03-25T20:32:40.434Z</atom:updated>
            <content:encoded><![CDATA[<p>I build local AWS emulators for development and testing, and I’ve been using formal specs to generate Gherkin test scenarios automatically. For a while, this worked well. Then I ran my test suite one day and noticed it was taking an unreasonably long time. When I looked at the numbers, I had 170,174 test scenarios across my specs. I hadn’t noticed it creep up on me.</p><p>This is what I learned from fixing it.</p><h3>The workflow that got me here</h3><p>I use FizzBee to write formal specifications. If you haven’t come across it, FizzBee is a formal methods language built on Starlark — you model a system as state variables and actions, and the model checker exhaustively explores every possible sequence to verify your invariants hold. It’s genuinely useful for the kind of thing I’m building, where correctness isn’t just about individual operations but about sequences — create a cluster, update it, delete it, try to delete it again.</p><p>Here’s a simplified excerpt from my SQS spec to give you a sense of what that looks like:</p><pre>QUEUE_NAMES = [&quot;q1&quot;, &quot;q2&quot;, &quot;dlq1&quot;]<br>MSG_IDS     = [&quot;msg-1&quot;, &quot;msg-2&quot;, &quot;msg-3&quot;]<br><br>action Init:<br>    queue_status = {}<br>    msg_status   = {}<br>    msg_queue    = {}<br><br># step: a queue is created<br># result: the queue is ACTIVE<br># guard: the queue does not already exist<br># guard_violation: the queue already exists<br>atomic action CreateQueue:<br>    oneof qname in QUEUE_NAMES:<br>        if qname not in queue_status:<br>            queue_status[qname] = &quot;ACTIVE&quot;</pre><p>The # step:, # result:, # guard: comments are annotations I added — FizzBee ignores them completely. They&#39;re my bridge to Gherkin.</p><p>I wrote a Python script (fizztogherkin.py) that reads those annotations and emits Gherkin feature files in three tiers. <strong>Minimal</strong> gives you one happy-path scenario per action. <strong>Standard</strong> adds negative scenarios for each guard condition. <strong>Exhaustive</strong> chains multiple actions together into sequence scenarios.</p><p>Those sequence scenarios matter. They’re the ones that catch ordering bugs — bugs where the emulator’s state after one operation is wrong, causing a subsequent operation to misbehave. You can’t catch that with happy-path tests alone.</p><p>The generated feature files run against the emulator via Cucumber-compatible runners. Write the spec, generate the Gherkin, wire up any missing step definitions, and the test suite is done. I’ve found this repeatable enough that adding a new service is mostly a solved problem.</p><h3>Where it went wrong</h3><p>When I added the MemoryDB spec — which models clusters, ACLs, users, snapshots, and their lifecycle operations — the --tier exhaustive flag produced 25,230 scenarios just for that one service. Across all my specs, the total reached 170,174.</p><p>At 10ms per scenario that’s 28 minutes. In practice it was longer.</p><p>The generator was producing sequence scenarios using itertools.permutations:</p><pre># All 2-step sequences<br>chains = [[a, b] for a, b in itertools.permutations(names, 2)]<br><br># All 3-step sequences<br>for triple in itertools.permutations(state_changing, 3):<br>    chains.append(list(triple))</pre><p>To explain what those numbers mean in practice: a <strong>length-2 pair</strong> is a sequence of exactly two consecutive actions — CreateQueue → SendMessage. The &quot;length&quot; is just how many steps are in the chain. With 30 actions, you get 30 × 29 = 870 of these, covering every action followed by every other action. A <strong>length-3 triple</strong> adds a third step — CreateQueue → SendMessage → DeleteQueue. With 30 actions, that&#39;s 30 × 29 × 28 = 24,360, because for each pair you can append any of the remaining 28 actions as a third step.</p><p>For MemoryDB with 30 actions, that’s 870 length-2 pairs and 24,360 length-3 triples. The pairs looked fine. The triples were the problem.</p><h3>The thing I hadn’t stopped to check</h3><p>I wrote a quick script to count the unique consecutive transition pairs across all 25,230 scenarios:</p><pre>from collections import defaultdict<br><br>current_whens = []<br>all_pairs = set()<br><br>with open(&quot;sequences.feature&quot;) as f:<br>    for line in f:<br>        stripped = line.strip()<br>        if stripped.startswith(&quot;Scenario:&quot;):<br>            if len(current_whens) &gt;= 2:<br>                for i in range(len(current_whens) - 1):<br>                    all_pairs.add((current_whens[i], current_whens[i+1]))<br>            current_whens = []<br>        elif stripped.startswith(&quot;When &quot;):<br>            current_whens.append(stripped[5:])<br>print(f&quot;Total scenarios:         {25230}&quot;)<br>print(f&quot;Unique transition pairs: {len(all_pairs)}&quot;)</pre><p>Output:</p><pre>Total scenarios:         25,230<br>Unique transition pairs: 870</pre><p>The 870 length-2 scenarios already covered every unique consecutive transition pair exactly once. Every one of the 24,360 triples was repeating a pair that was already tested.</p><p>Here’s the thing about triples: CreateQueue → SendMessage → DeleteQueue, CreateQueue → SendMessage → ReceiveMessage, and CreateQueue → SendMessage → PurgeQueue all contain the same length-2 pair CreateQueue → SendMessage inside them. If the bug you&#39;re looking for lives between those two steps — did CreateQueue leave state that breaks SendMessage? — you&#39;ve already found it the first time. Running it 28 more times with different third steps doesn&#39;t re-examine that interaction, it just adds noise. The pair CreateCluster → DeleteCluster appeared in 28 different triples for exactly this reason — once for each possible third action that could follow — but none of them were adding new coverage.</p><p>I hadn’t stopped to ask: what are sequence tests actually for?</p><h3>What sequence tests actually catch</h3><p>Sequence tests catch ordering bugs. Bugs where the emulator’s state after operation A is wrong, causing operation B to behave incorrectly. Things like:</p><ul><li><strong>Create → Delete → Create</strong>: does re-creation work, or is there stale state?</li><li><strong>Create → Update → Delete</strong>: did the update persist correctly, and does delete clean up what the update touched?</li><li><strong>Send → Receive → Send again</strong>: does the queue allow a second message after the first was received?</li></ul><p>A length-3 triple (A → B → C) is only meaningfully different from a length-2 pair (A → B) if the state after B — having come from A — affects C in a way that wouldn’t otherwise be tested. In a well-implemented state machine, what matters is the state between B and C, not the full history before B.</p><p>That’s the idea behind transition-pair coverage: if you’ve exercised every consecutive pair (B → C) at least once, you’ve tested every local interaction between adjacent operations. The 870 pairs already gave me that. The 24,360 triples were giving me nothing extra.</p><h3>The fix</h3><p>I replaced the permutation generator with one that builds a covering set:</p><pre>def build_sequence_scenarios(spec, depth=3):<br>    names = [a.name for a in spec.non_init_actions]<br>    if not names:<br>        return []<br><br>    # All length-2 pairs: complete transition-pair coverage<br>    pairs = list(itertools.permutations(names, 2))<br>    chains = [list(p) for p in pairs]<br><br>    # One triple per pair, cycling through third-action candidates<br>    if depth &gt;= 3 and len(names) &gt;= 3:<br>        for i, (a, b) in enumerate(pairs):<br>            candidates = [n for n in names if n != a and n != b]<br>            c = candidates[i % len(candidates)]<br>            chains.append([a, b, c])<br><br>    # deduplicate and emit...</pre><p>The length-2 pairs stay as-is. For length-3, each pair gets exactly one triple — a third action chosen by cycling through the remaining candidates, so coverage is distributed rather than concentrated. Every pair gets a depth-3 representative without being repeated 28 times.</p><p>Results across the full suite:</p><p>Spec Before After Reduction MemoryDB 25,230 1,740 93% Lambda 16,250 1,300 92% API Gateway 8,400 840 90% All specs 170,174 19,340 89%</p><p>I added tests to verify the generator’s properties — that every pair is a prefix of at least one triple, and that the covering set is substantially smaller than the naive approach:</p><pre>def test_covering_set_much_smaller_than_permutation_approach():<br>    spec = _spec(30)<br>    n = 30<br>    naive_count    = n * (n - 1) + n * (n - 1) * (n - 2)  # 25,230<br>    covering_count = n * (n - 1) * 2                        # 1,740<br><br>    scenarios = build_sequence_scenarios(spec, depth=3)<br><br>    assert len(scenarios) == covering_count<br>    assert (1 - len(scenarios) / naive_count) &gt; 0.90</pre><h3>What I took away from this</h3><p><strong>Formal verification and testing are not the same job.</strong> FizzBee’s model checker exhaustively proves invariants hold for all possible paths. That proof doesn’t need to be re-executed as tests — the checker already did it. Tests verify that your implementation matches the spec for specific cases. I was conflating the two, and the test suite was paying for it.</p><p><strong>Counting unique transition pairs was the right diagnostic to run first.</strong> The question “how many unique consecutive pairs of transitions exist?” immediately showed the redundancy. I should have asked it sooner. Once the numbers were visible, the fix was obvious.</p><p><strong>State explosion applies to generated test suites too, not just model checkers.</strong> I knew about state explosion in formal methods. I hadn’t connected it to the fact that I was generating test paths naively. The exponential growth was happening in my test suite for the same reason it happens in model checking.</p><p>I’m still figuring out where the right depth cutoff is for sequence scenarios in general. I’ve landed on 3 for now, but I’m not fully confident that’s correct for every spec — it depends on how much shared state there is between actions. I’m also curious whether there are smarter ways to choose which third action to append to each pair, rather than just cycling.</p><p>If you’re doing something similar - generating tests from formal specs, or thinking about coverage criteria for state machine testing — I am curious how you’re approaching it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d17a04a35661" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Improving your coding agents by reducing your human intervention rate]]></title>
            <link>https://medium.com/@eamonn.faherty_58176/improving-your-coding-agents-by-reducing-your-human-intervention-rate-23929fa57a91?source=rss-0021d5c19a0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/23929fa57a91</guid>
            <category><![CDATA[software-testing]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[formal-methods]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[eamonn faherty]]></dc:creator>
            <pubDate>Mon, 16 Mar 2026 10:46:04 GMT</pubDate>
            <atom:updated>2026-03-16T10:53:10.271Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*cjpodBfoNtKqBaXngyZrog.png" /></figure><p>I have been trying to optimize my agent setup to improve my Human Intervention Rate.</p><p>HIR is one of four metrics in <a href="https://aura-metrics.github.io/">AURA</a> — a proposed framework for measuring AI agent performance modelled on DORA, but applied to agent deliverables rather than software deployments. Where DORA measures how reliably your pipeline ships code, AURA measures how reliably your agents deliver value. Human Intervention Rate specifically tracks the percentage of deliverables that required a human to take over or were abandoned entirely. Elite is under 5%.</p><p>I’ve spent a lot of time looking at where those interventions are coming from, and I want to share something that has made a meaningful difference for me: using formal methods to validate designs before any agent session starts.</p><p>This might sound academic. It isn’t. Here’s what I mean with a real example.</p><h3>The problem agents make worse</h3><p>Agents make you faster. That is the point. A feature that would have taken a day now takes an hour, and that hour is mostly spent thinking about what you want rather than how to build it.</p><p>But speed has a cost that is easy to miss. When you are moving quickly from one feature to the next, you are less likely to stop and think about the big picture. The cross-cutting concerns — how this feature interacts with the one you built last Tuesday, or the one you are planning for next week — get less attention, not more. You are heads-down in the current session, and the current session only knows what you tell it.</p><p>This is compounded by how agent sessions work. Each session is independent. There is no persistent memory, no ambient awareness of decisions made in previous sessions. You open a session, you give the agent context, it does the work, the session ends. The next time you open a session, you start again.</p><p>When you design features one at a time — which is the natural way to work with agents — you tend to think clearly and carefully about each feature in isolation. The agent helps you explore the design space, you agree on an approach, you implement it. It works. You move on.</p><p>The problem is that features do not live in isolation. They share state. They interact with the same queues, the same rate limiters, the same caches, the same retry logic. And because each session is independent, the agent that helped you design feature B has no memory of the constraints you agreed when designing feature A. Neither, if you are honest, do you — at least not fully.</p><p>The result is that you end up with a collection of locally correct designs that have not been tested as a system. The cross-cutting issues are invisible until you run everything together and something breaks mid-deliverable. At that point, the agent cannot self-correct because the problem is not in the code it wrote — it is in the interaction between two pieces of code from two different sessions. A human has to step in. HIR goes up.</p><p>This is not a new problem in software engineering. It is exactly why system design reviews exist. But the agent workflow — stateless, session-based, feature-at-a-time — makes it worse, not better.</p><h3>What formal methods actually give you</h3><p>I have been using <a href="https://fizzbee.io/">FizzBee</a> recently, and it has changed how I think about the design phase.</p><p>FizzBee is a formal specification and model checking tool. You describe your system as a set of states and actions — what the system can do, what invariants must always hold, what sequences of actions should eventually be possible. FizzBee then exhaustively explores the state space and tells you whether your design violates any of those invariants.</p><p>The key word is exhaustively. FizzBee does not sample your design. It does not run a test suite against a handful of scenarios you thought to write. It explores every reachable state and checks every invariant against all of them. If there is a problem, it finds it, and it gives you the exact sequence of actions that leads to it.</p><p>This is qualitatively different from unit tests or integration tests. Tests check the paths you thought to check. Formal methods check all paths, including the ones you did not think of — which, in my experience, are exactly the ones that cause mid-deliverable failures.</p><p>The thing I have found most useful is that you can model features independently and then compose the models together. Design feature A in one session, capture its model, design feature B in another session, capture its model, then combine both models and run FizzBee on the result. If the combination introduces a state that violates one of your invariants, you know about it before you write a line of implementation code.</p><h3>A concrete example</h3><p>Here is a situation I ran into recently that illustrates this well.</p><p>I was building an agent workflow with two components designed in separate sessions:</p><p><strong>A task queue</strong> that accepted work items from an external source and held them for processing. The design principle was simple: never block the ingest side. Accept everything, process as fast as possible.</p><p><strong>A rate limiter</strong> that controlled how quickly those work items were dispatched to a downstream API. The design principle was also simple: never exceed the API’s rate limit, back off if you get a 429.</p><p>Both designs were sound in isolation. The agent sessions that produced them were clean. Neither design had any obvious flaws.</p><p>But when I modelled them together in FizzBee, it found a state I had not anticipated.</p><p>Under normal conditions, ingest rate and processing rate stayed balanced. But the model identified a sequence where a sustained burst of ingest, combined with a backoff period on the rate limiter, caused the queue depth to grow faster than the rate limiter could drain it. Not dramatically — just a little bit each cycle. FizzBee showed that this growth was unbounded: given enough time and the right ingest pattern, the queue would grow without limit.</p><p>In practice, this would not cause an immediate failure. It would cause a slow degradation. The queue would grow, the agent would spend increasing time managing it rather than processing work, and eventually — under the right conditions — the agent would reach a state it could not recover from without human help. Classic HIR problem, and completely invisible from either design session in isolation.</p><p>Here is a simplified version of what that model looked like in FizzBee:</p><pre># Constants — frozen, declared at top level<br>MAX_QUEUE_DEPTH = 20<br>RATE_LIMIT = 5<br># Mutable state — must be declared in action Init:<br>action Init:<br>    queue_depth = 0<br>    api_calls_this_window = 0<br>    rate_limited = False<br># step: a new item arrives and is added to the queue<br># result: queue depth increases by one<br>atomic action Ingest:<br>    queue_depth = queue_depth + 1<br># step: an item is processed and dispatched to the downstream API<br># result: queue depth decreases and api call count increases<br># guard: the rate limiter is not active and the queue is not empty<br># guard_violation: the rate limiter is active or the queue is empty<br>atomic action Process:<br>    if not rate_limited:<br>        if queue_depth &gt; 0:<br>            queue_depth = queue_depth - 1<br>            api_calls_this_window = api_calls_this_window + 1<br>            if api_calls_this_window &gt;= RATE_LIMIT:<br>                rate_limited = True<br># step: the rate limit window resets<br># result: api call count resets to zero and rate limiting is lifted<br>atomic action WindowReset:<br>    api_calls_this_window = 0<br>    rate_limited = False<br># check: queue depth never exceeds the maximum allowed depth<br>always assertion QueueDepthBounded:<br>    return queue_depth &lt; MAX_QUEUE_DEPTH</pre><p>FizzBee found the path to queue_depth &gt;= MAX_QUEUE_DEPTH in a few seconds. It also gave me the exact sequence: N ingest events, followed by a window reset that triggered a rate limit, followed by more ingest events during the backoff. It was not a likely sequence, but it was a reachable one, and in production with real traffic it would eventually happen.</p><p>The fix was straightforward once I could see the problem: add back-pressure on the ingest side when the queue depth exceeds a threshold. That change took thirty minutes. Finding this problem in production, mid-deliverable, with an agent that had no context on either original design session, would have taken much longer and would have shown up as a human intervention.</p><h3>What this looks like in practice</h3><p>I have settled into a workflow that goes roughly like this:</p><p>Before any implementation session starts, I spend time on the design. For anything beyond a trivial change, I write a FizzBee model of the relevant system components. If I am adding a new feature that will interact with existing components, I model those interactions explicitly.</p><p>I treat the model as the design artefact, not the code. The code is an implementation of the model. If the model changes, the code changes. If I am uncertain about a design decision, I explore it in the model first — changing the model is cheap, changing the code is expensive.</p><p>When I start an agent session, I include the relevant model as context. I have found this useful for two reasons: it gives the agent a precise description of the expected system behaviour, and it gives me a reference point for evaluating the agent’s output. If the agent proposes something that would violate a model invariant, that is a signal worth paying attention to.</p><p>After implementation, I update the model to reflect the actual design. This keeps the model useful for future sessions.</p><p>To get more out of the FizzBee work and reduce the risk of gaps between design and implementation, I built a FizzBee to Gherkin converter. It takes the FizzBee spec and generates human-readable acceptance tests that I use to validate the completeness of the implementation. To make this work cleanly, I added structured annotations directly to the FizzBee actions and assertions — you can see them in the code example above, the # step:, # result:, # guard: and # check: comments. The converter parses those annotations and generates Gherkin without any hardcoded mappings. It has also pushed me to write better specs: if an action is hard to annotate in plain English, that is usually a signal the action is doing too much.</p><p>I will be honest: this still adds time upfront. A FizzBee model for a non-trivial feature interaction might take two to four hours to write properly. That feels expensive when you are used to just starting an agent session and going.</p><p>But the modelling time buys you three things: a verified design, a set of acceptance tests derived directly from that design, and context you can drop into any future agent session. In my experience, that return is reliably more than the investment, and it compounds as the system grows.</p><h3>What I have not figured out yet</h3><p>I am not suggesting this is a complete solution to the HIR problem. It addresses one category of failure — design-level cross-cutting issues — but HIR has other causes too: ambiguous specs, tool failures, hallucinations, incomplete context. Formal methods do not help with those.</p><p>I also have not figured out the right level of modelling granularity. There is a risk of over-modelling: spending time on formal verification of interactions that were never going to cause problems in practice. I tend to focus on anything involving shared state, queuing, rate limiting, concurrency, or retry logic — the classic sources of emergent misbehaviour. I skip purely functional transformations where the interaction surface is minimal.</p><p>And FizzBee has a learning curve. The modelling language is not difficult, but thinking in terms of states and invariants takes some adjustment if you are not used to it. I found the first few models slow going; it gets faster.</p><p>If you are tracking HIR and it is stubbornly high, I would look at your design process before your prompts. The most common thing I have seen is not that the agents are bad at implementing — it is that the designs they are implementing were never validated as a system. That is a solvable problem, and formal methods are a practical tool for solving it.</p><p>Curious whether others have found different approaches to this — I am still refining the workflow and would genuinely like to know what is working for people.</p><p><em>Links: </em><a href="https://aura-metrics.github.io/"><em>AURA metrics framework</em></a><em> · </em><a href="https://fizzbee.io/"><em>FizzBee</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=23929fa57a91" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AgentEx: The Problems We Solved for DevEx Are Worth More for Agents]]></title>
            <link>https://medium.com/@eamonn.faherty_58176/agentex-the-problems-we-solved-for-devex-are-worth-more-for-agents-d7273094dad1?source=rss-0021d5c19a0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/d7273094dad1</guid>
            <category><![CDATA[agentex]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[sdlc]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[devex]]></category>
            <dc:creator><![CDATA[eamonn faherty]]></dc:creator>
            <pubDate>Tue, 10 Feb 2026 00:28:38 GMT</pubDate>
            <atom:updated>2026-02-10T01:21:40.603Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z5dtZK-lbPdMvdhVSYq_lw.png" /></figure><p>I’ve been thinking about developer experience lately, but from a different angle. Not “how do we make engineers faster” — that’s DevEx, and we’ve been investing in it for years. I’m thinking about something adjacent: how do we make AI agents faster? This is <strong>AgentEx</strong>.</p><h3>DevEx Problems Don’t Disappear — They Get Worse</h3><p>If you’ve worked on developer tooling or platform engineering, you know the patterns. Slow feedback loops kill productivity. Long deploy cycles break flow. Environments that drift from production waste debugging time. We’ve spent a decade building better CLIs, local dev environments, and CI pipelines to address these problems for human engineers.</p><p>Here’s what I’ve noticed: every one of these problems hits AI agents harder than it hits humans.</p><p>A human engineer waiting three minutes for a deployment can context switch, check Slack, think about the problem differently. An AI agent waiting three minutes is just burning tokens and wall-clock time doing nothing. The cost of a slow feedback loop scales differently when the thing waiting is an agent that could otherwise iterate dozens of times in those three minutes.</p><p>This is AgentEx. The same class of problems that DevEx addresses — slow feedback, environment complexity, deployment-gated testing — but viewed through the lens of agent productivity. And when you look at it that way, the ROI of solving these problems goes up significantly.</p><h3>Inner Loop vs. Outer Loop</h3><p>The distinction that matters most, in my experience, is inner loop versus outer loop development. Or put differently: pre-deploy versus post-deploy testing.</p><p>For human engineers, we mostly solved the inner loop. You run your code locally, you write unit tests, you get fast feedback before anything touches a shared environment. The outer loop — deploy, test in staging, promote to production — is slower, but humans manage it.</p><p>For agents, we haven’t solved the inner loop for many types of applications. Cloud-native development is a clear example. If your application is a set of Lambda functions wired to DynamoDB tables, SQS queues, and API Gateway routes, your “inner loop” has traditionally been: edit code, run cdk deploy, wait minutes, test against live AWS resources, check CloudWatch logs. That&#39;s not an inner loop. That&#39;s an outer loop pretending to be one.</p><p>A human can tolerate this. It’s painful, but we’ve all done it. An agent cannot operate effectively in this model. The feedback cycle is too slow, too expensive, and too opaque for an agent to iterate meaningfully.</p><h3>Solving the Inner Loop Unlocks Both</h3><p>I’ve been working on a project called <a href="https://local-web-services.github.io/www">Local Web Services</a> that tries to address exactly this. It reads your AWS CDK cloud assembly and recreates your entire application locally — DynamoDB, Lambda, S3, SQS, SNS, EventBridge, Step Functions, Cognito, API Gateway, ECS — all running on your machine.</p><p>The workflow changes from deploy-wait-test to something much tighter:</p><ul><li>Run uvx local-web-services ldk dev — your stack is ready in seconds</li><li>Edit code — hot reload picks up changes instantly</li><li>Test against local services that mirror your AWS architecture</li><li>Inspect data directly with CLI commands</li><li>No credentials, no cost, no waiting</li></ul><p>This started as a DevEx project. I wanted faster feedback for my own CDK development. But I quickly realized the bigger unlock is for agents. An AI code editor that can make a change, hit a local endpoint, inspect a DynamoDB table, see an error, and fix it — all in seconds — is fundamentally more capable than one stuck in a deploy-wait-test cycle.</p><p>The observation that changed my thinking: solving this for agents solves it for humans too, but the performance gain for agents is disproportionately larger. A human might go from a 5-minute feedback loop to a 5-second one — a meaningful improvement. An agent goes from being nearly non-functional in that domain to being able to iterate rapidly. That’s a qualitative shift, not just a quantitative one.</p><h3>Spec-Driven Development Was the First Step</h3><p>Before tools like Local Web Services could exist, we needed a way for agents to understand what infrastructure to expect. This is where spec-driven development comes in — defining your application’s shape declaratively so that tooling (and agents) can reason about it.</p><p>AWS CDK already does this to some degree. Your infrastructure is code. cdk synth produces a declarative output. That output is what Local Web Services reads to spin up your local environment. The spec exists; we just needed tools that consume it in agent-friendly ways.</p><p>I expect we’ll see more solutions like this — tools that take existing declarative specs and create fast, local, agent-accessible environments from them. The spec-driven approach was the prerequisite. The tooling is the payoff.</p><p>I have been looking at better definitions of ready and done using static analysis, architecture tests, acceptance tests, performance tests. You can read about the series here: <a href="https://medium.com/@eamonn.faherty_58176/getting-the-most-fromai-code-editors-1bd7e097acf6">https://medium.com/@eamonn.faherty_58176/getting-the-most-fromai-code-editors-1bd7e097acf6</a></p><h3>AgentEx Is DevEx With Higher Stakes</h3><p>I’m not suggesting AgentEx replaces DevEx. They’re the same problems, and the solutions often overlap. What I am suggesting is that framing these problems through the lens of agent productivity changes the priority calculus.</p><p>A slow feedback loop that’s “annoying but manageable” for a human engineer might be a complete blocker for an agent. When you fix it, you accelerate both — but the agent goes from blocked to unblocked, while the human goes from slow to faster. The value of the fix is asymmetric.</p><p>If you’re working on developer tooling or platform engineering, I think this framing is worth considering. The problems you’ve been solving for humans are the same problems that need solving for agents, and the return on solving them may be higher than you’d expect.</p><h3>What I Haven’t Figured Out Yet</h3><p>I’m still working through what the full landscape of AgentEx problems looks like. Feedback loops and local environments are the ones I’ve been focused on, but there are others — observability, error messages designed for LLM consumption, deterministic test environments, credential-free development.</p><p>I’m curious: if you’re building tools for AI agents, or using agents in your development workflow, what are the biggest friction points you’re seeing? Are there DevEx problems you’ve already solved for humans that you’re re-encountering now that agents are in the picture?</p><p>I’d be interested to hear what others are running into.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d7273094dad1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your Team Already Collapsed — You Just Haven’t Noticed Yet]]></title>
            <link>https://medium.com/@eamonn.faherty_58176/your-team-already-collapsed-you-just-havent-noticed-yet-291f46886f60?source=rss-0021d5c19a0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/291f46886f60</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[engineering-mangement]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[cloud-migration]]></category>
            <dc:creator><![CDATA[eamonn faherty]]></dc:creator>
            <pubDate>Sat, 10 Jan 2026 19:55:18 GMT</pubDate>
            <atom:updated>2026-01-10T19:55:18.134Z</atom:updated>
            <content:encoded><![CDATA[<h3>Your Team Already Collapsed — You Just Haven’t Noticed Yet</h3><p>I wrote recently about <a href="https://medium.com/@eamonn.faherty_58176/is-your-team-collapsing-in-front-of-you-e743e4cd6fcb">how engineering organizations collapse as they scale.</a> Growth causes patterns to fail slowly — the informal coordination that worked at 50 people quietly breaks down at 1200.</p><p>But there’s an inverse problem that’s arguably worse: what happens when organizations shrink.</p><p>Growth causes slow collapse. Shrinkage causes fast collapse. And many organizations that grew to 1200 are now shrinking back toward 500.</p><h3>Knowledge Disappears Instantly</h3><p>When an engineer with deep institutional knowledge leaves, that knowledge is gone overnight. Not degraded. Gone.</p><p>In distributed systems, this is catastrophic. The engineer who knew how service A actually talks to service B — not what the diagram says, but what really happens. The architect who remembered which upstream dependencies have retry logic and which will cascade failures. The person who understood why that service has a 30-second timeout instead of 5 and what breaks if you “fix” it. The engineer who knew the TPS limits of every downstream system and could tell you that no, we can’t increase throughput by 5% because the payment service will fall over.</p><p>None of this was documented because it never needed to be. They just knew. Now nobody knows.</p><h3>Nobody Knows How the Bits Fit Together</h3><p>At scale, distributed systems become incomprehensible to any single person. But collectively, the organization understood the system through overlapping knowledge — different engineers holding different pieces of the map.</p><p>When enough of those engineers leave, the map disappears. You’re left with services that call other services for reasons nobody remembers. Data flows through pipelines that nobody fully understands. Dependencies exist that aren’t documented anywhere because the documentation was the person who built it.</p><p>I’ve seen teams discover critical dependencies only when they break. “We didn’t know service C depended on that database view.” Of course you didn’t — the engineer who knew that left eight months ago.</p><h3>Relationship Networks Collapse</h3><p>Institutional knowledge isn’t just technical. It’s also knowing how the organization actually works.</p><p>The person who knew which team really owned that legacy service — not what the service catalogue said, but who actually maintained it. The engineer who could get a decision made across org boundaries because they’d built trust over years. The principal engineer who remembered why two services were never supposed to be deployed simultaneously and could stop you before you caused an incident.</p><p>When they leave, you don’t just lose their skills. You lose the map of how things actually got done.</p><h3>You Hit All Four Failure Modes At Once</h3><p>In my previous post, I described Jared Diamond’s framework for how groups make disastrous decisions: failure to anticipate, failure to perceive, failure to try something different, and failure to succeed.</p><p>Growth moves you through these stages slowly — often over years.</p><p>Shrinkage hits you with all four simultaneously.</p><p><strong>Failure to anticipate:</strong> You didn’t predict which knowledge would walk out the door. The people who seemed replaceable turned out to be carrying critical context about how services interact. The people you kept don’t know what they don’t know.</p><p><strong>Failure to perceive:</strong> The gaps aren’t visible until something breaks. Services keep running. Deployments keep succeeding. Then someone makes a reasonable change and triggers a cascade that the departed engineer would have caught in code review.</p><p><strong>Failure to try something different:</strong> You attempt to fix things using approaches that assume expertise that no longer exists. You assign the legacy service to someone new and expect them to maintain it. They can’t. They don’t have the context to change it safely.</p><p><strong>Failure to succeed:</strong> The people who could have succeeded are gone. The migration required knowledge that left with them. You’re not just missing capacity — you’re missing capability.</p><h3>Systems Become Untouchable — Or Require Complete Rewrites</h3><p>I’ve watched this happen: a service becomes a black box because nobody remaining understands it well enough to change it safely.</p><p>So it doesn’t get maintained. It doesn’t get upgraded. Security patches don’t get applied because nobody knows what might break. The organisation routes around it, building increasingly fragile workarounds.</p><p>Eventually someone decides it needs to be rewritten. Not because a rewrite is the right technical choice, but because understanding the existing system would require knowledge that no longer exists in the organisation. The rewrite is an admission of defeat — proof that institutional knowledge has degraded past the point of recovery.</p><h3>The Legacy System Blocking Your Data Center Exit</h3><p>Here’s where this gets expensive.</p><p>Many organizations are trying to exit data centers — moving to cloud, consolidating infrastructure, reducing operational burden. The timeline is set, the business case is approved, the migration is planned.</p><p>And then you discover that one legacy application is blocking everything.</p><p>It’s the service that nobody fully understands anymore. The engineers who built it left years ago. The documentation is outdated or missing. It has undocumented dependencies on infrastructure that only exists in that data center. It talks to other services in ways that aren’t captured in any diagram.</p><p>You can’t migrate it because you don’t understand it. You can’t rewrite it because you don’t know what it actually does. You can’t turn it off because you don’t know what depends on it.</p><p>That one application — often something that seemed minor, something nobody prioritized — becomes a multi-million dollar blocker. The data center exit stalls. The lease gets extended. The cost savings evaporate.</p><p>This happens because the knowledge required to migrate the system safely left the building before anyone realised it was critical.</p><h3>The Remaining Engineers Are Set Up to Fail</h3><p>The people who stay are now covering more ground with less context. They’re making changes in services they’ve never seen before. They’re expected to maintain systems written by people they’ve never met, implementing decisions they don’t understand, for reasons nobody remembers.</p><p>This isn’t a skill problem. It’s an information problem. You can’t expect someone to safely modify a distributed system when the context required to understand that system has left the building.</p><h3>What I’ve Seen Work</h3><p>I haven’t figured out how to fully solve this, but I’ve seen some things help.</p><p><strong>Document the “why” aggressively while you still can.</strong> Not just what the service does, but why it does it that way. What alternatives were considered. What constraints existed. Which other services depend on this behaviour. What will break if you change it. This feels like overhead when the knowledgeable people are still around. It’s priceless after they leave.</p><p><strong>Map your service dependencies before you need to.</strong> Not the official diagram — the real one. Which services actually call which? What are the undocumented dependencies? Where does data actually flow? Do this while the people who know are still around to tell you.</p><p><strong>Exit interviews focused on systems, not feelings.</strong> When someone with deep knowledge leaves, spend time extracting what they know. Not HR questions — technical knowledge transfer. Which services are fragile? What’s undocumented? Where are the landmines? What will break the next data center migration?</p><p><strong>Identify your single points of knowledge failure before they leave.</strong> Map which systems depend on knowledge that exists in only one person’s head. That’s your risk register. Either document it, cross-train it, or accept the risk explicitly.</p><p><strong>Find your data center exit blockers now.</strong> Don’t wait until the migration is underway. Identify the legacy systems that nobody fully understands. Start the knowledge extraction while the people who might know something are still reachable.</p><h3>The Hard Truth</h3><p>Some organizations that went through layoffs in the past two years are still running on momentum. The services work because they were built well. The deployments succeed because the pipelines are solid. Everything looks fine.</p><p>But the people who understood how the pieces fit together are gone. The context that would prevent the next major incident has left the building. The knowledge that would enable the next infrastructure migration has evaporated.</p><p>The collapse has already happened — it just hasn’t manifested yet.</p><p>If your organization has shrunk significantly, ask yourself: what knowledge walked out the door that you don’t even know you’ve lost? And which system is going to teach you the answer the hard way?</p><p>I’d be interested to hear how others have navigated this. What’s worked for preserving institutional knowledge about distributed systems during shrinkage?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=291f46886f60" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is Your Team Collapsing In Front Of You?]]></title>
            <link>https://medium.com/@eamonn.faherty_58176/is-your-team-collapsing-in-front-of-you-e743e4cd6fcb?source=rss-0021d5c19a0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/e743e4cd6fcb</guid>
            <category><![CDATA[engineering-mangement]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[testing]]></category>
            <dc:creator><![CDATA[eamonn faherty]]></dc:creator>
            <pubDate>Fri, 09 Jan 2026 15:28:43 GMT</pubDate>
            <atom:updated>2026-01-09T15:28:43.056Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*jfuthNWgcIUGAUHI7RPyKw.png" /></figure><p>I&#39;ve been thinking about why some engineering organizations slowly fall apart while others adapt and thrive. Not the dramatic implosions—those are obvious. I mean the gradual decline that everyone inside the organization struggles to see.</p><p>Jared Diamond&#39;s book Collapse examines why entire societies fail. He identifies a four-stage roadmap of how groups make disastrous decisions. After working in engineering organisations as they scale, I&#39;ve noticed these same patterns show up. What works at 50 people quietly becomes fatal at 1200.</p><p>Here&#39;s Diamond&#39;s framework, translated for engineering leaders navigating growth.</p><h3>1. Failure to Anticipate</h3><p>The group doesn&#39;t see the problem coming because they have no prior experience with it, or they reason by false analogy.</p><p><strong>What this looks like in distributed systems: </strong>At 50 people, when a team changes an API response, they mention it in Slack or walk across the floor to give the consuming team a heads up. It works. Changes get communicated, nobody gets surprised, and the system stays healthy. Leadership assumes this will keep working with more services if we have more Slack channels.</p><p>At 1200 people, the team changing that API has no idea who&#39;s consuming it. The Slack message doesn&#39;t reach the three teams in different time zones who integrated six months ago. The walk across the floor isn&#39;t possible when the consuming team is in another building or another country. The best intentions of engineers trying to coordinate don&#39;t scale. What felt like good communication was actually small-scale informality that silently stops working.</p><p><strong>How to spot it:</strong> Look at how breaking changes get communicated. If the answer is &quot;we tell people,&quot; you&#39;re relying on a pattern that won&#39;t survive growth. Check whether your services validate their inputs or just trust that upstream will send what&#39;s expected. Check whether you have any way of knowing who will break when you change your API response.</p><p><strong>How to mitigate it:</strong> Program defensively. At scale, you can&#39;t assume the teams you depend on will warn you before things change—even if they want to, they might not know you exist. Validate inputs rigorously. Fail fast and explicitly when contracts are violated. Implement contract testing so that breaking changes get caught in CI before they reach production. These practices feel like overhead at 50 people. At 1200, they&#39;re the only thing standing between you and cascading failures caused by changes nobody told you about.</p><h3>2. Failure to Perceive</h3><p>The problem has arrived, but the group doesn&#39;t recognise it. Diamond calls this &quot;creeping normalcy&quot;, a slow decline masked by day-to-day fluctuations.</p><p><strong>What this looks like in testing:</strong> At 50 people, your integration tests ran in serial and completed in minutes. Few issues escaped to production. The tests were reliable, engineers trusted them, and a red build meant something was actually wrong.</p><p>At 1200 people, those same tests now run in parallel across hundreds of services and take four hours. Test environments are flaky. Engineers re-run failed builds &quot;just to check&quot; because nobody trusts the results anymore. Issues escape to production regularly, but each individual escape seems like bad luck rather than systemic failure. New hires assume this is just how testing works at scale.</p><p>The decline happened gradually. Test duration crept up quarter by quarter. Flakiness increased slowly enough that each week felt roughly like the last. Nobody can point to the moment the test suite stopped being trustworthy, it just slowly became background noise that engineers work around rather than rely on.</p><p><strong>How to spot it:</strong> Compare your test metrics to two years ago, not last quarter. How long did tests take then? What was your flake rate? How often did issues escape? Ask engineers who&#39;ve been around longest whether they trust the tests. Track how often people re-run builds without changing code.</p><p><strong>How to mitigate it:</strong> Break up your testing into smaller, isolated groups. Use contract testing to verify the boundaries between groups. Smaller groups means fewer services under test at once, which makes test environments more stable and results more reliable. A ten-service integration test that runs in three minutes and rarely flakes is worth more than a four-hundred-service integration test that takes four hours and that nobody trusts. You trade coverage breadth for coverage you can actually rely on.</p><h3>3. Failure to Try Something Different</h3><p>The group perceives the problem and attempts to solve it—but keeps doubling down on the same approach instead of reconsidering the fundamentals.</p><p><strong>What this looks like in testing:</strong> Teams know testing is broken. They&#39;re trying to fix it. But instead of investing in functional tests that catch issues early, they pour effort into making end-to-end and parity tests run faster. The logic sounds reasonable: &quot;If we can get faster feedback from our E2E tests, we&#39;ll catch issues sooner.&quot;</p><p>But during those E2E test runs, they keep finding functional issues—bugs that should have been caught much earlier in the pipeline, when they&#39;re easier to isolate and cheaper to fix. The team says &quot;if only these tests ran faster, we&#39;d be fine.&quot; But that&#39;s not true. If speed were the only problem, you&#39;d expect the team to be slow but with a low defect rate. Instead, they&#39;re slow with a high defect rate. The speed isn&#39;t the root cause—it&#39;s the symptom of testing at the wrong layer.</p><p>The investment keeps flowing into the wrong place because E2E tests are visible and measurable. &quot;We reduced E2E runtime by 30%&quot; looks good in a quarterly review. &quot;We wrote better unit and integration tests&quot; is harder to quantify and less impressive to leadership who don&#39;t understand the testing pyramid.</p><p><strong>How to spot it:</strong> Look at where defects are actually caught. If your E2E tests are regularly finding bugs that could have been caught by functional tests, you&#39;re testing too late. Track the ratio of defects found at each layer. If most bugs are caught at the top of the pyramid, your investment is inverted. Listen for &quot;if only X were faster&quot;—it often masks a deeper structural problem.</p><p><strong>How to mitigate it:</strong> Shift investment toward functional tests. Every bug caught in an E2E test is a failure of the layers below it. Make it a team goal to push defect detection earlier in the pipeline. Measure success not by E2E speed but by how few issues make it to E2E in the first place. The fastest E2E test is the one that has nothing left to catch.</p><h3>4. Failure to Succeed</h3><p>The group tries to solve the problem, declares victory, but the underlying situation gets worse.</p><p><strong>What this looks like in testing:</strong> Leadership acknowledges that centralised testing isn&#39;t working. The test team is a bottleneck, feedback is too slow, teams are frustrated. The decision is made to decentralise—push testing responsibility out to individual teams. This feels like progress. It&#39;s a real structural change, not just optimising the existing approach.</p><p>Six months later, the centralized team reports success. They&#39;re no longer a bottleneck. Goals met.</p><p>But overall testing now takes longer than before. Defect leakage has increased. Gaps have appeared in coverage between teams—each team tests their own service, but nobody owns the boundaries. The decentralised teams are struggling, but they don&#39;t report it because testing is now &quot;their problem&quot; and admitting failure reflects poorly on them. The former centralized team doesn&#39;t report these issues because they&#39;re no longer responsible and their goals have been met.</p><p>Everyone declares victory. The metrics that get reported look fine. The actual quality of testing has declined, but the organizational structure now hides the failure.</p><p><strong>How to spot it:</strong> Look beyond the metrics each team reports and measure outcomes at the system level. Is overall defect escape rate better or worse? Is total time spent on testing across all teams higher or lower? Ask teams directly whether they feel equipped for the responsibilities they&#39;ve been handed—but ask in a way that makes honesty safe. Watch for coverage gaps at boundaries. If nobody owns cross-team integration testing, it&#39;s probably not happening.</p><p><strong>How to mitigate it:</strong> Define success at the system level, not the team level. A reorganisation that makes one team&#39;s metrics look better while making overall quality worse isn&#39;t success—it&#39;s displacement. Create forums where teams can admit they&#39;re struggling without it being a performance issue. Keep some central accountability for overall quality outcomes, even when execution is decentralized. Someone has to own the whole picture.</p><h3>The Pattern I Keep Seeing</h3><p>The organizations that avoid collapse aren&#39;t the ones with the best engineers or the biggest budgets. They&#39;re the ones where someone is explicitly watching for these failure modes—usually a principal engineer or director who&#39;s seen scale kill an architecture before and knows what the early stages look like.</p><p>But watching isn&#39;t enough. That person also has to be listened to.</p><p>I&#39;ve seen organizations where alarms are raised for years. They wrote the documents, presented the data, escalated through the right channels. Leadership nodded, thanked them for their diligence, and continued prioritising features. The vigilance existed. The authority to act on it didn&#39;t.</p><p>The patterns that worked at 50 people don&#39;t fail immediately at 1200. They fail slowly, invisibly, while everyone insists things are basically fine.</p><p>Success depends heavily on individuals who happen to have the right experience and the right position to act on it.</p><p>I&#39;d be interested to hear how others think about this. What signals have you seen that a scaling organization is starting to collapse?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e743e4cd6fcb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Fallacy of Example-Based Testing]]></title>
            <link>https://medium.com/@eamonn.faherty_58176/the-fallacy-of-example-based-testing-d03f303105a2?source=rss-0021d5c19a0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/d03f303105a2</guid>
            <category><![CDATA[property-based-testing]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[integration-testing]]></category>
            <category><![CDATA[software-testing]]></category>
            <dc:creator><![CDATA[eamonn faherty]]></dc:creator>
            <pubDate>Wed, 17 Dec 2025 14:39:28 GMT</pubDate>
            <atom:updated>2025-12-17T14:39:28.722Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rJ_xXtHi7E3D3xtL6WOUIw.png" /></figure><p>After writing too many integration and end-to-end tests across various projects, I’ve come to an uncomfortable realisation: example-based testing, the way most teams practice it, provides confidence without completeness. We write tests for the scenarios we think of, ship the code, and then discover the scenarios we didn’t think of in production.</p><p>I’ve watched teams add more and more integration tests to catch more edge cases. I’ve seen test suites grow to thousands of tests, with CI pipelines taking hours to complete. I’ve been that engineer who wrote another end-to-end test after a production incident, trying to prevent the same bug from happening again. And I’ve seen the next incident come from a different edge case that also wasn’t covered.</p><p>Something is fundamentally limited in how we approach testing.</p><h3>Understanding What Example-Based Testing Actually Does</h3><p>Before I get into the limitations, let me clarify what I mean by example-based testing.</p><p><strong>Example-based testing</strong> is any testing approach where you specify concrete inputs and verify concrete outputs. This includes unit tests, integration tests, end-to-end tests, and parity tests. You think of a scenario, you encode it as a test, you move on to the next scenario.</p><pre>def test_merge_sorted_lists():<br>    # Arrange<br>    list_a = [1, 3, 5]<br>    list_b = [2, 4, 6]<br>    <br>    # Act<br>    result = merge(list_a, list_b)<br>    <br>    # Assert<br>    assert result == [1, 2, 3, 4, 5, 6]</pre><p>This test verifies one scenario. To verify another scenario, you write another test. Your coverage of the input space is bounded by the number of examples you write.</p><p>The limitation isn’t that these tests are wrong. They’re not. They verify exactly what they claim to verify. The limitation is that we treat a passing test suite as evidence that our code handles all cases, when it’s actually evidence that our code handles the cases we thought to test.</p><h3>The Imagination Bottleneck</h3><p>Here’s something I didn’t fully appreciate until I started analysing test suites more carefully. The coverage of example-based tests is bounded by human imagination and human time.</p><p>Consider a function that validates email addresses. A typical test suite might include:</p><pre>def test_valid_email():<br>    assert is_valid_email(&quot;user@example.com&quot;) == True<br><br>def test_invalid_email_no_at():<br>    assert is_valid_email(&quot;userexample.com&quot;) == False<br><br>def test_invalid_email_no_domain():<br>    assert is_valid_email(&quot;user@&quot;) == False<br><br>def test_empty_string():<br>    assert is_valid_email(&quot;&quot;) == False</pre><p>Four tests. Four scenarios. But the input space for email validation is enormous. What about: — Unicode characters in the local part — Plus addressing (user+tag@example.com) — IP address domains — Quoted strings with special characters — Maximum length boundaries — Consecutive dots — Comments in parentheses (yes, that’s valid in RFC 5321)</p><p>You could write tests for each of these. But you’d need to know they exist first. And you’d need time to write them. Most teams stop when they feel they have “enough” coverage, which is a subjective judgement that often underestimates the actual input space.</p><h3>The Slow Feedback Problem</h3><p>Example-based tests, particularly integration and end-to-end tests, have a scaling problem. Every new scenario you want to cover requires a new test. Every new test adds to the execution time.</p><p>I’ve worked with test suites where: — The full integration test suite took 45 minutes to run — Engineers would run a subset locally and rely on CI to catch issues — CI queues backed up during busy periods — The feedback loop between “I think this works” and “CI confirms this works” stretched to hours</p><p>The response is often to parallelise, to add more build agents, to optimise individual tests. These help. But they’re treating the symptom, not the cause. The cause is that coverage scales linearly with test count, and test count scales linearly with execution time.</p><h3>The False Confidence of Parity Testing</h3><p>Parity testing deserves specific mention because it’s a common pattern that has a subtle failure mode.</p><p>The idea is straightforward: you have a legacy system and a new system, you feed them the same inputs, you verify they produce the same outputs. If they match, the new system is correct.</p><p>The limitation is that parity tests only cover the inputs you feed them. If the legacy system has a bug that the new system also reproduces, parity passes. If both systems handle a certain input incorrectly in the same way, parity passes. Parity testing verifies equivalence, not correctness.</p><p>I’ve seen teams discover, after migration, that both systems had been wrong for years. The parity tests were green the entire time.</p><h3>We Already Know This Doesn’t Work</h3><p>Here’s the thing: we already behave as if we know example-based testing is insufficient. We just don’t talk about it that way.</p><p>Most teams I’ve worked with do some form of exploratory testing before major releases. Some call it a game day. Some call it bug bash. Some just call it “everyone stop what you’re doing and try to break this.”</p><p>The defining characteristic of exploratory testing is that you don’t follow a test plan. You explore freely, trying things that occur to you in the moment, following hunches, combining inputs in ways nobody specified. You’re explicitly not bound by the scenarios someone thought to write down.</p><p>And it works. Exploratory testing regularly finds bugs that the test suite missed. Bugs that survived thousands of automated tests get discovered in an afternoon of unscripted poking around.</p><p>But think about what that implies. If our example-based tests were sufficient, exploratory testing would be redundant. We’d find nothing. The fact that we schedule time for humans to explore outside the test plan is an admission that the test plan doesn’t cover everything. We know the examples we wrote down aren’t complete.</p><p>We’ve institutionalised the workaround without questioning the underlying approach.</p><h3>A Different Approach: Property-Based Testing</h3><p>Property-based testing inverts the example-based model. Instead of specifying inputs and outputs, you specify properties that should hold for all inputs. The framework generates inputs automatically.</p><pre>from hypothesis import given, strategies as st<br><br>@given(st.lists(st.integers()), st.lists(st.integers()))<br>def test_merge_length_preserved(xs, ys):<br>    # Arrange<br>    xs_sorted = sorted(xs)<br>    ys_sorted = sorted(ys)<br>    <br>    # Act<br>    result = merge(xs_sorted, ys_sorted)<br>    <br>    # Assert<br>    assert len(result) == len(xs) + len(ys)<br><br>@given(st.lists(st.integers()), st.lists(st.integers()))<br>def test_merge_result_is_sorted(xs, ys):<br>    # Arrange<br>    xs_sorted = sorted(xs)<br>    ys_sorted = sorted(ys)<br>    <br>    # Act<br>    result = merge(xs_sorted, ys_sorted)<br>    <br>    # Assert<br>    assert result == sorted(result)</pre><p>Two property definitions. Hundreds of generated test cases. The framework explores the input space automatically, including edge cases you wouldn’t think to test: empty lists, single-element lists, lists with duplicates, lists with negative numbers, lists with large values.</p><p>When a property fails, the framework shrinks the failing case to the minimal reproduction. Instead of “test failed with these 500 elements,” you get “test failed with [0] and [1].”</p><p>The shift is from “verify these specific examples work” to “verify this property holds universally.” The feedback loop is faster because you’re not writing individual tests for each edge case. The coverage of the input space is broader because you’re not limited by imagination.</p><h3>A Different Approach: Model Checking</h3><p>For concurrent or distributed systems, there’s another technique worth understanding: model checking.</p><p>Model checking tools like TLA+ let you specify what your system should do, then exhaustively explore every possible execution path within defined bounds. For a lock implementation, the model checker will verify that no two threads can ever hold the lock simultaneously — not by testing examples, but by exploring every possible interleaving.</p><p>I’ve seen race conditions that survived thousands of integration test runs get caught immediately by a model checker. The integration tests happened to never hit the specific interleaving that triggered the bug. The model checker found it because it explores all interleavings systematically.</p><h3>What I’ve Learned</h3><p>I’m not saying example-based tests are useless. They’re not. Integration tests verify that components actually work together. End-to-end tests verify that user journeys complete successfully. These are valuable signals.</p><p>But I’ve learned to recognise the limitations. Example-based tests tell you that specific scenarios work. They don’t tell you that all scenarios work. The difference matters.</p><p>Here’s what I’m experimenting with: — Property-based testing for functions with complex logic, data transformation, parsing, and serialisation — Model checking for concurrent code and state machines where correctness is critical — Reserving integration and end-to-end tests for high-value user journeys rather than edge case coverage — Treating example-based tests as documentation of known scenarios, not proof of completeness</p><p>I haven’t figured out the right balance yet. Property-based testing requires thinking in terms of invariants, which is a different skill than thinking in terms of examples. Model checking has a learning curve and works best for specific types of problems. Every team and codebase has different needs.</p><p>But I’ve become increasingly sceptical of test suites that grow endlessly with each new edge case discovered in production. If your tests only catch the bugs you already know about, they’re not finding bugs — they’re documenting history.</p><p>For those of you managing large test suites: have you experimented with property-based testing? How do you balance the coverage benefits against the learning curve? I’d be interested to hear what’s working for others.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d03f303105a2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Fallacy of Unit Test Coverage]]></title>
            <link>https://medium.com/@eamonn.faherty_58176/the-fallacy-of-unit-test-coverage-9a63c919388b?source=rss-0021d5c19a0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a63c919388b</guid>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[atdd]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[eamonn faherty]]></dc:creator>
            <pubDate>Thu, 11 Dec 2025 14:09:19 GMT</pubDate>
            <atom:updated>2025-12-11T14:09:19.400Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*gMw4tLE0u5_8jJ9Q9vTAxQ.png" /></figure><p>After writing more than a thousand tests across various projects, I’ve noticed something that’s changed how I think about quality metrics: test coverage doesn’t measure what most people assume it measures.</p><p>I’ve watched teams celebrate hitting 80% coverage targets. I’ve sat in meetings where leaders pointed to coverage graphs trending upward as proof of quality improvement. I’ve been that engineer who wrote tests purely to satisfy a coverage gate. And I’ve seen production incidents in codebases with impressive coverage numbers.</p><p>The disconnect between coverage metrics and actual test quality is worth understanding, because I think many teams are optimising for the wrong thing.</p><h3>Understanding What Coverage Actually Measures</h3><p>Before I get into the limitations, let me explain what the metrics actually mean.</p><p><strong>Line coverage</strong> (sometimes called statement coverage) answers a simple question: which lines of code were executed when the tests ran? If you have 100 lines of code and your tests caused 75 of them to execute, you have 75% line coverage. That’s it. Nothing more.</p><p><strong>Branch coverage</strong> goes a step further. It asks: for every decision point in your code (every if, every while, every ternary operator), did the tests exercise both outcomes? A single if statement has two branches: the path where the condition is true and the path where it&#39;s false. Branch coverage requires both to be executed.</p><p>Here’s a concrete example:</p><pre>def calculate_discount(price, is_member):<br>    if is_member:<br>        return price * 0.9<br>    return price</pre><p>A single test with is_member=True gives you 100% line coverage. Every line executes. But you only have 50% branch coverage because you never took the false branch. This distinction matters, and most teams I&#39;ve worked with don&#39;t track it.</p><h3>Why Most Teams Only Track Line Coverage</h3><p>Here’s what I’ve observed: the vast majority of engineers and engineering leaders only track line coverage. Branch coverage is either ignored entirely or treated as a nice-to-have afterthought.</p><p>I think this happens for a few reasons. Line coverage is simpler to understand and explain. It maps directly to “did this code run?” which feels intuitive. Branch coverage requires thinking about control flow, about the invisible paths through your code, and that’s harder to visualise.</p><p>There’s also a tooling problem. Most CI/CD pipelines come preconfigured with line coverage. Getting branch coverage requires additional setup, and when you’re under pressure to ship features, nobody wants to spend time configuring coverage tools.</p><p>The result is that teams optimise for a metric that tells them less than they think. I’ve seen codebases with 90% line coverage where entire logical branches had never been tested. The coverage report was green. The bugs were real.</p><h3>The Exercise vs Testing Problem</h3><p>Here’s something I didn’t fully appreciate until I started auditing test suites. There’s a fundamental difference between exercising code and testing it, and coverage metrics can’t tell the difference.</p><p>Consider this test:</p><pre>def test_calculate_discount():<br>    # Arrange<br>    price = 100<br>    is_member = True<br>    <br>    # Act<br>    result = calculate_discount(price, is_member)<br>    <br>    # Assert<br>    # ... nothing here</pre><p>This test achieves coverage. The code executes. The coverage tool records that the lines ran. But this test doesn’t actually test anything. There’s no assertion. No verification that the result is correct. The code could return None, return 42, throw an exception, or delete your database, and this test would still pass.</p><p>I call this “coverage tourism.” You visit the code. You don’t verify it works.</p><p>I’ve audited test suites where a significant percentage of tests had weak or missing assertions. They were written to satisfy coverage requirements, not to catch bugs. The coverage numbers looked healthy. The actual bug-catching ability was far lower than you’d expect.</p><p>Here’s what a real test looks like:</p><pre>def test_calculate_discount_for_member():<br>    # Arrange<br>    price = 100<br>    is_member = True<br>    <br>    # Act<br>    result = calculate_discount(price, is_member)<br>    <br>    # Assert<br>    assert result == 90</pre><p>Coverage metrics treat both examples identically. They can’t distinguish between code that was merely executed and code that was actually verified.</p><h3>How Exceptions Undermine Branch Coverage</h3><p>This is something that took me a while to fully appreciate. Exceptions create invisible branches in your code that coverage tools often miss or misrepresent.</p><p>Consider this code:</p><pre>def process_order(order_id):<br>    order = database.get_order(order_id)  # Can throw<br>    validated = validator.validate(order)  # Can throw<br>    result = payment.charge(order)         # Can throw<br>    return result</pre><p>How many branches does this function have? If you’re only counting explicit if statements, you might say zero. But in reality, every line that can throw an exception creates an implicit branch: the normal continuation path and the exceptional exit path.</p><p>Your coverage tool might report 100% coverage for this function after a single successful test. But you’ve only tested one path through what is effectively an eight-branch function (each of the four statements can either succeed or fail).</p><p>This gets worse when you consider that exceptions can propagate through multiple stack frames. A test might exercise a line of code, but an exception thrown three calls deep might cause that line to exit early in production in ways your test never explored.</p><p>I’ve found that exception handling is where coverage metrics give the least accurate picture. The happy path gets tested. The error paths, the ones that matter most when things go wrong in production, often have zero effective coverage despite what the metrics say.</p><h3>The Boundaries and Ranges Problem</h3><p>Let’s say you have a function that calculates shipping costs based on weight:</p><pre>def calculate_shipping(weight_kg):<br>    if weight_kg &lt;= 0:<br>        raise ValueError(&quot;Weight must be positive&quot;)<br>    if weight_kg &lt;= 1:<br>        return 5.00<br>    if weight_kg &lt;= 5:<br>        return 10.00<br>    if weight_kg &lt;= 20:<br>        return 20.00<br>    return 50.00</pre><p>You can achieve 100% branch coverage with tests for weights of -1, 0.5, 3, 10, and 25. All branches covered. Coverage report is green.</p><p>But you haven’t tested the boundaries. What happens at exactly 1kg? At exactly 5kg? At exactly 20kg? Off-by-one errors are among the most common bugs, and coverage metrics don’t care about them at all.</p><p>You also haven’t tested extreme values. What about float(&#39;inf&#39;)? What about 0.0000001? What about None? Coverage doesn&#39;t distinguish between testing with representative values and testing with boundary values.</p><p>I’ve found bugs at boundaries in code with 100% coverage more times than I can count. The coverage was achieved with convenient middle-of-the-range values. The edge cases were never explored.</p><h3>The Business Logic Gap</h3><p>This is perhaps the most significant limitation, and it’s one I rarely see discussed.</p><p>Coverage measures code paths. It doesn’t measure requirements. It doesn’t know about your business logic.</p><p>Imagine you’re building a discount system with these requirements: — Members get 10% off — Orders over £100 get free shipping — Members ordering over £100 get an additional 5% off — Discounts don’t apply to sale items — Maximum total discount is 25%</p><p>You could achieve 100% coverage by testing each of these rules in isolation. But have you tested the combinations? What happens when a member orders £150 worth of sale items? What about a non-member ordering £99.99?</p><p>Coverage tools count lines and branches. They don’t count business scenarios. I’ve worked on systems where the combination of valid business states numbered in the hundreds, but the test suite only covered a handful. The coverage was high. The confidence should have been low.</p><p>This isn’t something coverage tools can solve. They measure what they measure. The problem is when we treat their output as a proxy for test completeness.</p><h3>A Different Approach: Acceptance Test Driven Development</h3><p>After years of wrestling with these coverage limitations, I’ve found that Acceptance Test Driven Development (ATDD) addresses many of these shortcomings in ways that unit test coverage simply cannot.</p><p>The core idea is straightforward: instead of measuring how much code your tests execute, you measure how many business scenarios your tests verify. You start by defining acceptance criteria as executable specifications before writing any code. Each feature gets broken down into scenarios that describe expected behaviour from the user’s perspective.</p><p>Here’s what this looks like in practice using a format like Gherkin:</p><pre>Feature: Member Discounts<br>  <br>  Scenario: Members receive 10% discount on regular items<br>    Given a customer is a registered member<br>    And they have a regular item priced at £100 in their basket<br>    When they proceed to checkout<br>    Then the item price should be £90<br>    <br>  Scenario: Member discount does not apply to sale items<br>    Given a customer is a registered member<br>    And they have a sale item priced at £100 in their basket<br>    When they proceed to checkout<br>    Then the item price should be £100<br>    <br>  Scenario: Combined discounts cannot exceed 25%<br>    Given a customer is a registered member<br>    And they have items totalling £200<br>    And they qualify for free shipping<br>    And they have a 20% promotional code<br>    When they proceed to checkout<br>    Then the total discount should be capped at 25%</pre><p>Notice what’s different here. Each scenario is explicitly tied to a business requirement. There’s no ambiguity about what’s being tested or why. The assertions are built into the specification itself.</p><h3>Tracking Progress Through Scenario Completion</h3><p>Here’s where I’ve found ATDD changes the conversation with stakeholders.</p><p>Instead of reporting “we have 82% line coverage,” you report “we have implemented 47 of 63 acceptance scenarios for the checkout feature.” Instead of a coverage percentage that means nothing to product owners, you have a burndown of feature completion that everyone understands.</p><p>I’ve found this changes team behaviour in meaningful ways. When the metric is scenario completion, there’s no incentive to write assertion-free tests that merely exercise code. A scenario either passes with its assertions or it doesn’t. You can’t game it the way you can game coverage.</p><p>The burndown also provides genuine progress visibility during development. When building a new feature, I can show stakeholders that we’ve completed scenarios for happy path ordering, member discounts, and error handling, while scenarios for international shipping and promotional codes are still in progress. This is information they can actually use for planning and prioritisation.</p><p>Compare this to coverage metrics, which might show 60% coverage but tell you nothing about which user-facing functionality works and which doesn’t.</p><h3>Why ATDD Addresses the Coverage Gaps</h3><p>Let me connect this back to the problems I outlined earlier.</p><p>The exercise vs testing problem largely goes away because acceptance tests are defined by their assertions. A scenario without verification isn’t a scenario. The format enforces that you specify expected outcomes.</p><p>The business logic gap shrinks because you’re explicitly mapping tests to requirements. If a business rule exists, there should be a scenario for it. If there isn’t, that gap is visible in your scenario list, not hidden behind a coverage percentage.</p><p>The boundary and edge case problem becomes manageable because you can write scenarios specifically for edge cases:</p><pre>Scenario: Shipping weight at exact boundary<br>    Given an order weighing exactly 5.00 kg<br>    When shipping cost is calculated<br>    Then the cost should be £10.00</pre><pre>Scenario: Shipping weight just above boundary  <br>    Given an order weighing 5.01 kg<br>    When shipping cost is calculated<br>    Then the cost should be £20.00</pre><p>These boundary scenarios are now visible in your test suite as first-class citizens, not buried as implementation details in unit tests that may or may not have adequate assertions.</p><h3>The Trade-offs</h3><p>I’m not suggesting ATDD solves everything. It doesn’t.</p><p>Acceptance tests are slower than unit tests. They typically require more infrastructure. They’re harder to write and maintain. A test suite with hundreds of acceptance scenarios takes time to run, and that affects your feedback loop.</p><p>There’s also a learning curve. Teams used to writing unit tests need to think differently about how they structure scenarios. Getting the right level of abstraction takes practice.</p><p>That said, I’ve noticed AI coding agents are starting to offset some of these drawbacks. Writing step definitions and glue code has always been the tedious part of ATDD. It’s repetitive, it requires understanding both the testing framework and the application, and it’s exactly the kind of boilerplate that AI tools handle well. I’ve been experimenting with having AI agents generate step definitions from scenarios, and the time savings are significant. What used to take an hour of wiring up database fixtures and API calls now takes minutes of review and refinement.</p><p>The maintenance burden also shrinks when AI can help refactor step definitions as the application evolves. Updating a hundred scenarios because the user model changed is no longer a day of tedious work.</p><p>I’ve found the approach works best as a complement to unit testing, not a replacement. Unit tests still have value for testing implementation details, edge cases at the code level, and providing fast feedback during development. But unit test coverage should not be your primary quality metric.</p><p>The real value of ATDD is that it gives you a metric that actually means something: how much of the specified behaviour has been implemented and verified. That’s a question coverage metrics simply cannot answer.</p><h3>What I’ve Learned</h3><p>I’m not saying coverage metrics are useless. They’re not. Zero coverage is a signal. A sudden drop in coverage when code changes is a signal. Coverage can help identify completely untested code paths.</p><p>But I’ve learned to treat coverage as a lower bound, not a target. High coverage is necessary but not sufficient. It tells you what code was executed, not what was verified, not whether boundaries were tested, not whether business requirements were satisfied.</p><p>Here’s what I’m experimenting with instead: Acceptance Test Driven Development with scenario burndown as the primary progress metric. Mutation testing, which measures whether tests actually catch bugs. Property-based testing for boundary and edge case discovery. Combining ATDD for business scenario coverage with unit tests for implementation details. Code review practices that question assertion quality, not just coverage. Treating coverage drops as signals but not coverage increases as achievements</p><p>I haven’t figured out the right balance yet. Every team and every codebase has different needs. But I’ve become increasingly sceptical of coverage targets and coverage gates that create incentives to game the metric rather than improve the testing.</p><p>For those of you managing test suites on large codebases: are you tracking branch coverage or just line coverage? Have you experimented with ATDD or scenario-based progress tracking? How do you ensure your tests actually assert meaningful behaviour rather than just executing code? I’d be interested to hear what’s working for others.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a63c919388b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Performance Testing Django Projects with AI Code Editors: Catching Slowdowns Before Your Users Do]]></title>
            <link>https://medium.com/@eamonn.faherty_58176/performance-testing-django-projects-with-ai-code-editors-catching-slowdowns-before-your-users-do-9e98ef1a02b2?source=rss-0021d5c19a0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e98ef1a02b2</guid>
            <category><![CDATA[software-testing]]></category>
            <category><![CDATA[performance-testing]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[ai-code-assistant]]></category>
            <category><![CDATA[django]]></category>
            <dc:creator><![CDATA[eamonn faherty]]></dc:creator>
            <pubDate>Wed, 01 Oct 2025 12:37:43 GMT</pubDate>
            <atom:updated>2025-10-01T12:37:43.093Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zXxfYXQzwGoc3kq8xs7URA.png" /></figure><p><em>Part of the “</em><a href="https://medium.com/@eamonn.faherty_58176/getting-the-most-fromai-code-editors-1bd7e097acf6"><em>Getting the Most from AI Code Editors</em></a><em>” series</em></p><p>After writing more than a thousand tests with AI code editors for my Django project, I’ve discovered something powerful: the same tools that accelerate feature development can just as easily accelerate performance testing. The key is setting up the right constraints and letting AI agents write tests that catch bottlenecks during development — when they’re easy to fix — instead of in production when root cause analysis is harder and fixes are more expensive.</p><p>This post covers how I’m using AI code editors to write comprehensive performance tests that catch N+1 queries, verify optimal attribute selection, and measure page load times — all while reducing the time investment typically required for this kind of testing.</p><h3>The Recommendation</h3><p>Write automated performance tests that verify:</p><ul><li>Database query counts (to catch N+1 queries)</li><li>Query efficiency (selecting only required attributes — loading entire User objects when you only need the email wastes memory and database bandwidth)</li><li>Page load times (real-world performance)</li></ul><p>Use AI code editors to generate these tests based on your views and models. The speed gains from AI-assisted development give you time to write tests that actually matter to users, catching performance issues during development instead of in production.</p><h3>Why Performance Testing with AI Code Editors Works</h3><p>Traditional performance testing has a problem: it’s time-consuming. When you’re under pressure to ship features, performance tests get deprioritized. You tell yourself you’ll add them later, then production gets slow and users complain.</p><p>AI code editors change this calculation. When I can generate a comprehensive performance test suite in minutes instead of hours, I actually do it. The reduced development time from AI assistance doesn’t just mean faster features — it means time for quality work that prevents user-facing problems.</p><p>Here’s what I mean with a real example from my Django views:</p><pre>@pytest.mark.django_db<br>class TestViewPerformance(TestCase):<br>    &quot;&quot;&quot;Test performance of views&quot;&quot;&quot;<br># Performance thresholds based on project requirements<br>    PAGE_LOAD_TIME_THRESHOLD = 0.8  # 800ms for frontend pages<br>    MAX_QUERIES_PER_REQUEST = 10  # &lt; 10 queries per request<br>    def test_buy_view_get_query_efficiency(self):<br>        &quot;&quot;&quot;Test BuyView GET request uses minimal database queries&quot;&quot;&quot;<br>        url = reverse(&quot;frontend:buy&quot;, kwargs={&quot;product&quot;: &quot;BIN&quot;})<br>        # Currently uses 11 queries (target would be &lt; 10 but this is acceptable for now)<br>        # Queries: session, user, asset, wallet, balance (2x), window, fees,<br>        # user details, something else<br>        # Room for optimization: reduce duplicate balance lookups, use select_related<br>        with self.assertNumQueries(11):  # Document current state<br>            response = self.client.get(url)<br>        # Currently uses 11 queries (target would be &lt; 10 but this is acceptable for now)<br>        self.assertEqual(response.status_code, 200)<br>        self.assertContains(response, &quot;SOMETHING&quot;)<br>        self.assertContains(response, &quot;97.50&quot;)  # price</pre><p>This test does several things:</p><ul><li>Sets an explicit query count threshold</li><li>Documents what each query is doing</li><li>Notes optimization opportunities</li><li>Verifies the page works correctly</li></ul><h3>How I Structure Performance Tests</h3><p>I organize performance tests into three categories: query efficiency, page load times, and transaction performance.</p><h3>Query Efficiency Tests</h3><p>These catch N+1 queries — the classic Django performance killer. An N+1 query happens when you load a list of objects (1 query), then loop through them and access a related object for each one (N additional queries). For example, loading 100 orders and then accessing order.user.email for each one results in 101 queries instead of 1.</p><p>N+1 queries are bad because they scale terribly. Ten orders means 11 queries. A thousand orders means 1,001 queries. What seems fine in development becomes unusably slow in production. Django provides select_related and prefetch_related to solve this, but it&#39;s easy to forget them or miss them in code review.</p><p>python</p><pre>def test_buy_view_get_query_efficiency(self):<br>    &quot;&quot;&quot;Test BuyView GET request uses minimal database queries&quot;&quot;&quot;<br>    url = reverse(&quot;frontend:buy&quot;, kwargs={&quot;product&quot;: &quot;SHOE&quot;})<br>    with self.assertNumQueries(11):<br>        response = self.client.get(url)<br>    self.assertEqual(response.status_code, 200)</pre><p>The magic of assertNumQueries is that it fails loudly when you introduce new queries. Add a field lookup in a template loop? The test breaks. Forget to use select_related? The test breaks.</p><p>I don’t aim for perfection immediately. That 11-query limit documents the current state. The comment explains what’s happening and where optimization could help. When I have time to optimize, I know exactly where to look.</p><h3>Page Load Time Tests</h3><p>Query counts matter, but users care about actual speed. I measure page load times with simple timing:</p><pre>def test_buy_view_load_time_performance(self):<br>    &quot;&quot;&quot;Test BuyView page loads within time threshold&quot;&quot;&quot;<br>    url = reverse(&quot;frontend:buy&quot;, kwargs={&quot;product&quot;: &quot;PANTS&quot;})<br>    start_time = time.time()<br>    response = self.client.get(url)<br>    elapsed_time = time.time() - start_time<br>    self.assertEqual(response.status_code, 200)<br>    self.assertLess(<br>        elapsed_time,<br>        self.PAGE_LOAD_TIME_THRESHOLD,<br>        f&quot;Buy page took {elapsed_time:.3f}s, threshold is {self.PAGE_LOAD_TIME_THRESHOLD}s&quot;,<br>    )</pre><p>This catches regressions that query counts miss. Maybe you’re doing expensive calculations in Python when SQL would be faster. Maybe you’re making external API calls that slow everything down. Load time tests catch these.</p><h3>Transaction Performance Tests</h3><p>For operations that modify data, I test both query efficiency and total time:</p><pre>def test_buy_transaction_performance(self):<br>    &quot;&quot;&quot;Test buy transaction performance with realistic volume&quot;&quot;&quot;<br>    url = reverse(&quot;frontend:buy&quot;, kwargs={&quot;product&quot;: &quot;book&quot;})<br>    start_time = time.time()<br>    response = self.client.post(url, {&quot;quantity&quot;: &quot;10&quot;})<br>    elapsed_time = time.time() - start_time<br>    self.assertEqual(response.status_code, 302)<br>    self.assertLess(<br>        elapsed_time,<br>        self.PAGE_LOAD_TIME_THRESHOLD,<br>        f&quot;Buy transaction took {elapsed_time:.3f}s, threshold is {self.PAGE_LOAD_TIME_THRESHOLD}s&quot;,<br>    )</pre><p>I test realistic volumes. Buying 10 books should be fast. If it’s slow, users will notice. The test catches this before deployment.</p><h3>Setting Performance Thresholds</h3><p>I define thresholds as class constants:</p><pre>PAGE_LOAD_TIME_THRESHOLD = 0.8  # 800ms for frontend pages<br>MAX_QUERIES_PER_REQUEST = 10  # &lt; 10 queries per request</pre><p>These numbers come from project requirements, not arbitrary preferences. Your thresholds might be different based on your application’s needs.</p><p>The important part is making them explicit. When someone changes code and breaks the threshold, they see exactly what the limit is and why the test failed.</p><h3>Testing Edge Cases and Validation</h3><p>Performance matters for error cases too. Users hit validation errors frequently, and those paths should be fast:</p><pre>def test_buy_form_validation_performance(self):<br>    &quot;&quot;&quot;Test buy form validation efficiency&quot;&quot;&quot;<br>    url = reverse(&quot;frontend:buy&quot;, kwargs={&quot;product&quot;: &quot;SOCKS&quot;})<br>    start_time = time.time()<br>    # Invalid amount<br>    response1 = self.client.post(url, {&quot;quantity&quot;: &quot;invalid&quot;})<br>    # Zero amount<br>    response2 = self.client.post(url, {&quot;quantity&quot;: &quot;0&quot;})<br>    # Negative amount<br>    response3 = self.client.post(url, {&quot;quantity&quot;: &quot;-5&quot;})<br>    elapsed_time = time.time() - start_time<br>    self.assertEqual(response1.status_code, 302)<br>    self.assertEqual(response2.status_code, 302)<br>    self.assertEqual(response3.status_code, 302)<br>    # Validation should be very fast<br>    self.assertLess(<br>        elapsed_time,<br>        0.1,  # 100ms for form validation<br>        f&quot;Form validation took {elapsed_time:.3f}s, should be under 0.1s&quot;,<br>    )</pre><p>Three validation checks should complete in under 100ms. If they take longer, something’s wrong — maybe you’re hitting the database when you shouldn’t, or running expensive operations before validation.</p><h3>Testing High-Volume Scenarios</h3><p>Real applications have users with different usage patterns. I test these explicitly:</p><pre>def test_high_volume_performance(self):<br>    &quot;&quot;&quot;Test performance with realistic high-volume scenarios&quot;&quot;&quot;<br>    # Simulate a user with high volume<br>    high_volume_user = User.objects.create_user(<br>        &quot;whale&quot;, &quot;whale@test.com&quot;, &quot;password123&quot;<br>    )<br>    initialize_user_data(high_volume_user)<br>     high_volume_wallet = high_volume_user.wallet<br>    # Give high volume user lots of credits<br>    credits_balance = Balance.objects.get(<br>        wallet=high_volume_wallet, currency=self.credits_currency<br>    )<br>    credits_balance.amount = Decimal(&quot;100000.00&quot;)<br>    credits_balance.save()<br>    self.client.force_login(high_volume_user)<br>    buy_url = reverse(&quot;frontend:buy&quot;, kwargs={&quot;product&quot;: &quot;EXPENSIVE_WATCH&quot;})<br>    # Test large buy transaction<br>    start_time = time.time()<br>    response = self.client.post(buy_url, {&quot;quantity&quot;: &quot;50&quot;})<br>    elapsed_time = time.time() - start_time<br>    self.assertEqual(response.status_code, 302)<br>    self.assertLess(<br>        elapsed_time,<br>        self.PAGE_LOAD_TIME_THRESHOLD,<br>        f&quot;High volume buy transaction took {elapsed_time:.3f}s, threshold is {self.PAGE_LOAD_TIME_THRESHOLD}s&quot;,<br>    )</pre><p>If buying 5 watches is fast but buying 50 is slow, you’ve got an algorithmic problem. Test the edge cases that matter.</p><h3>Where AI Code Editors Excel</h3><p>Here’s where AI code editors become powerful for performance testing:</p><p><strong>Generating test fixtures quickly.</strong> Setting up realistic test data is tedious. I describe what I need (“create a user with 100 items in their basket and sufficient credits”) and the AI generates the setup code. This would take 20 minutes to write manually. With AI, it’s 2 minutes.</p><p><strong>Creating comprehensive test coverage.</strong> I can ask for “performance tests covering all major views” and get a solid starting point. The AI generates tests for GET requests, POST requests, validation failures, and edge cases. I review and adjust, but the bulk of the work is done. When I review and adjust I refine the prompt for next time. I manage a directory of guidance I use when developing that is divided into theoretical guidance on how things should be done and practical guidance on how to do them — eg how should performance testing be done and then how to write tests of different types in pytest. If I change to a Java project I can update my how to section only.</p><p><strong>Documenting query patterns.</strong> Those comments explaining what each query does? The AI writes them by analyzing the view code. This documentation would normally get skipped because it’s time-consuming. With AI, it’s automatic.</p><p><strong>Identifying optimization opportunities.</strong> When I ask the AI to review query efficiency, it spots patterns I might miss: “This could use select_related” or “Consider bulk_create here.” The suggestions aren’t always perfect, but they’re a good starting point.</p><h3>My AI-Assisted Workflow</h3><p>I have developed my own scrum workflow using different agents. When it comes to performance testing I request my performance SDET agent writes the tests using the performance testing guidance once my engineer agent has completed implementing features (denoted by the acceptance tests passing). The guidance looks a little like this:</p><pre>Performance Testing Requirements:<br>1. Add assertNumQueries checks for all views<br>2. Document what each query does in comments<br>3. Add page load time tests with 800ms threshold<br>4. Test validation error performance (should be &lt; 100ms)<br>5. Include high-volume scenarios<br>6. Note optimization opportunities in comments</pre><p>The performance SDET agent workflow looks a little like this:</p><ol><li><strong>Ask AI to generate performance tests</strong> for the view</li><li><strong>Review and adjust thresholds</strong> based on requirements</li><li><strong>Run the tests</strong> to get baseline numbers</li><li><strong>Add to pre-commit hooks</strong> to catch regressions</li></ol><p>The AI gives me tests I might not have written due to time pressure. I review them because the AI sometimes makes mistakes — wrong assumptions about what matters, overly strict thresholds, or missing edge cases. But the first draft is solid enough that I spend minutes refining instead of hours writing from scratch.</p><p>Defining the prompts in reusable guidance ensures consistent test structure across the codebase. I have architectural tests to ensure the agent is following the programming guidelines (yes I have tests to verify the tests are written correctly).</p><h3>Integrating with Development Workflow</h3><p>I run performance tests in pre-commit hooks:</p><pre># .pre-commit-config.yaml<br>repos:<br>  - repo: local<br>    hooks:<br>      - id: performance-tests<br>        name: Performance Tests<br>        entry: uv pytest tests/test_performance_views.py -v<br>        language: system<br>        pass_filenames: false</pre><p>Every commit attempt runs performance tests. If someone introduces a regression, the commit is blocked. This catches problems immediately, when the code is fresh in mind and easy to fix.</p><p>For AI code editors like Claude Code, I use verification hooks:</p><pre># .claude/hooks/verify_performance.py<br>def verify_task_completion():<br>    &quot;&quot;&quot;Check performance tests before marking task complete&quot;&quot;&quot;<br>    result = subprocess.run([&#39;uv&#39;, &#39;pytest&#39;, &#39;tests/test_performance*.py&#39;, &#39;-v&#39;],<br>                          capture_output=True, text=True)<br>    <br>    if result.returncode != 0:<br>        print(&quot;❌ Performance tests failed:&quot;)<br>        print(result.stdout)<br>        return False<br>    <br>    return True</pre><p>The AI validates its own work against performance constraints. If it introduces a regression, it catches and fixes it before I even review the code. If I make a change without the AI code editor the pre commit hook picks up my change and verifies it.</p><h3>What I’ve Learned</h3><p>Performance testing used to be something I did when I had extra time — which meant rarely. AI code editors changed this by making test generation fast enough to be routine.</p><p>The time I save on feature development gives me capacity for quality work that prevents production issues. Instead of debugging slow pages after users complain, I catch problems during development when they’re cheap to fix.</p><p>This isn’t about AI writing perfect tests. It’s about AI removing the friction that prevented me from writing them at all. I still review everything. I still adjust thresholds and add edge cases the AI missed. But I’m starting from 80% complete instead of 0%, and that makes all the difference.</p><p>The combination of automated testing and AI-assisted development creates a workflow where performance is a first-class concern rather than an afterthought. Tests run on every commit. Regressions get caught immediately. The feedback loop stays tight.</p><h3>The Benefits</h3><p>The most immediate benefit: users don’t find your performance bugs. When you catch N+1 queries before deployment, users never see those 5-second page loads. When you verify page load times in tests, you ship fast interfaces from day one.</p><p>Developer experience improves too. Pre-commit hooks catch regressions before code review, saving reviewer time and preventing the frustration of finding performance issues late in the process.</p><p>The tests document performance expectations. New developers see the thresholds and understand what “good enough” means for the project. Comments explain query patterns and optimization opportunities. The test suite becomes documentation that stays current because it’s executable.</p><p>AI-assisted generation makes comprehensive coverage realistic. Without AI, I’d write performance tests for the most critical paths and skip the rest. With AI, I test everything because the time investment is minimal.</p><h3>The Drawbacks</h3><p>This approach adds test maintenance overhead. When you change a view, you might need to update query counts or load time expectations. Tests can become brittle if thresholds are too strict.</p><p>AI-generated tests sometimes miss context. The AI might not know that a particular operation is expected to be slow, or that certain queries are intentional. You need to review and adjust.</p><p>Threshold selection requires judgment. Set them too strict and you’ll have constant false positives. Too loose and they won’t catch real problems. This takes experience.</p><p>Performance tests add to CI time. Running comprehensive performance tests on every commit slows down the feedback loop. I run full suites in pre-commit but use faster smoke tests in CI.</p><p>The tests document current behavior, not always optimal behavior. That 11-query limit I showed earlier? It’s documenting reality, not perfection. There’s a risk of accepting suboptimal performance because “that’s what the test allows.”</p><h3>What’s Next</h3><p>I’m experimenting with AI-generated performance regression reports. When tests fail, the AI analyzes the diff and explains what changed and why it affected performance. Still early, but promising.</p><p>I’m also working on having AI suggest concrete optimizations when performance tests fail. Instead of just saying “too many queries,” it could suggest “add select_related(‘user’, ‘wallet’) to this queryset.” The AI has the context to make specific recommendations.</p><p>The goal remains the same: catch performance issues during development, before they reach users. AI code editors make this realistic by removing the time barrier that prevented comprehensive testing.</p><p>If you’re using AI code editors for feature development, consider using them for performance testing too. The time savings enable quality work that makes the difference between a fast application and a slow one.</p><p>I’d be interested to hear how this approach works for others. What performance metrics matter most in your applications? How do you balance test coverage with maintenance overhead?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e98ef1a02b2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Right-Sizing Your Python Files: The 150–500 Line Sweet Spot for AI Code Editors]]></title>
            <link>https://medium.com/@eamonn.faherty_58176/right-sizing-your-python-files-the-150-500-line-sweet-spot-for-ai-code-editors-340d550dcea4?source=rss-0021d5c19a0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/340d550dcea4</guid>
            <category><![CDATA[ai-code-assistant]]></category>
            <dc:creator><![CDATA[eamonn faherty]]></dc:creator>
            <pubDate>Mon, 29 Sep 2025 18:54:48 GMT</pubDate>
            <atom:updated>2025-09-29T19:31:15.014Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ubQNdRmGfFvUdJ0RmDi6_A.png" /></figure><p>This is part of the “<a href="https://medium.com/@eamonn.faherty_58176/getting-the-most-fromai-code-editors-1bd7e097acf6">Getting the Most from AI Code Editors</a>” series</p><p>After watching AI code editors work across thousands of files in my Django project, I’ve noticed something: file size matters way more than I expected. Not for the usual reasons like readability or maintainability, but for how well AI agents can understand, modify, and verify their work.</p><p>This post covers why keeping Python files between 150–500 lines helps AI code editors perform better and how this constraint has shaped my development workflow.</p><h3>The Recommendation</h3><p>Keep your Python files between 150 and 500 lines of code.</p><p>This isn’t some arbitrary rule I’m enforcing. It’s about optimizing for how AI code editors actually process and modify code. When files fall within this range, AI agents can:</p><ul><li>Hold the entire file context in their working memory without truncation.</li><li>Understand the complete scope of changes they’re making.</li><li>Generate accurate diffs that are easy to review.</li><li>Identify side effects and dependencies more reliably.</li></ul><p>I don’t count blank lines or comments toward this limit. I’m talking about actual code.</p><p>When a file approaches 500 lines, I look for natural split points. Sometimes that’s extracting a class into its own file. Sometimes it’s breaking a large module into smaller, more focused ones. The key is finding logical boundaries rather than forcing arbitrary splits.</p><p>For files under 150 lines, I generally leave them alone unless they’re doing too many unrelated things. Short, focused files are fine. The 150 line floor is more about preventing over-splitting than enforcing minimums.</p><p>The automated test is straightforward. I use pytest to check file sizes across my codebase:</p><pre>&quot;&quot;&quot;<br>Test that Python files don&#39;t exceed 500 lines.Long files are difficult to maintain, understand, and test.<br>Files should be kept concise and focused on a single responsibility.<br>&quot;&quot;&quot;<br>import os<br>from pathlib import Path<br><br>def test_python_files_max_500_lines():<br>    &quot;&quot;&quot;<br>    Ensure all Python files in frontend, domain, and tests are at most 500 lines long.<br>    Files exceeding 500 lines indicate:<br>    - Too many responsibilities in a single module<br>    - Need for refactoring into smaller, focused modules<br>    - Potential violation of single responsibility principle<br>    Exceptions can be made for:<br>    - Generated migration files<br>    - Configuration files with large data structures<br>    &quot;&quot;&quot;<br>    max_lines = 550<br>    project_root = Path(__file__).parent.parent.parent<br>    violations = []<br>    # Directories to check<br>    directories = [&#39;frontend&#39;, &#39;domain&#39;, &#39;tests&#39;]<br>    for directory in directories:<br>        dir_path = project_root / directory<br>        if not dir_path.exists():<br>            continue<br>        # Walk through all Python files<br>        for py_file in dir_path.rglob(&#39;*.py&#39;):<br>            # Skip migration files<br>            if &#39;migrations&#39; in py_file.parts:<br>                continue<br>            # Skip __pycache__ directories<br>            if &#39;__pycache__&#39; in py_file.parts:<br>                continue<br>            # Count lines in the file<br>            try:<br>                with open(py_file, &#39;r&#39;, encoding=&#39;utf-8&#39;) as f:<br>                    line_count = sum(1 for _ in f)<br>                if line_count &gt; max_lines:<br>                    relative_path = py_file.relative_to(project_root)<br>                    violations.append(f&quot;{relative_path}: {line_count} lines (exceeds {max_lines})&quot;)<br>            except Exception as e:<br>                # Skip files that can&#39;t be read<br>                continue<br>    if violations:<br>        violation_list = &#39;\n&#39;.join(violations)<br>        assert False, f&quot;Files exceeding {max_lines} lines:\n{violation_list}&quot;</pre><p>Notice I set max_lines = 550 rather than 500. This gives me some buffer for files that are close to the limit without being strict about every single line.</p><p>I run this test through git pre-commit hooks so violations get caught before code even enters the repository:</p><pre># .pre-commit-config.yaml<br>repos:<br>  - repo: local<br>    hooks:<br>      - id: file-size-tests<br>        name: File Size Tests<br>        entry: uv pytest tests/test_file_size.py -v<br>        language: system<br>        pass_filenames: false</pre><p>For Claude Code, I use verification hooks that run after tasks complete. This catches violations that the AI introduced during code generation:</p><pre># .claude/hooks/verify_file_sizes.py<br>def verify_task_completion():<br>    &quot;&quot;&quot;Check file size constraints before marking task complete&quot;&quot;&quot;<br>    result = subprocess.run([&#39;uv&#39;, &#39;pytest&#39;, &#39;tests/test_file_size.py&#39;, &#39;-v&#39;],<br>                          capture_output=True, text=True)<br>    <br>    if result.returncode != 0:<br>        print(&quot;❌ File size tests failed:&quot;)<br>        print(result.stdout)<br>        return False<br>    <br>    return True</pre><p>I also include file size guidelines in my Claude Code prompts:</p><pre>Keep files between 150-500 lines:<br>1. Run: uv pytest tests/test_file_size.py -v<br>2. If tests fail, refactor large files into smaller modules<br>3. Find natural split points - don&#39;t force arbitrary divisions<br>4. Re-run tests until they pass<br>Current results: {test_output}</pre><p>This three-layer approach catches violations early. The test defines what’s acceptable, pre-commit hooks prevent violations from entering the repository, and Claude Code hooks ensure AI-generated code follows the constraints.</p><p>Beyond just checking file length, I also verify file contents to ensure they maintain single responsibility. For example, I use acceptance testing when developing applications and have a check to ensure each step definition has its own file:</p><pre>&quot;&quot;&quot;<br>Test that each architectural step file contains exactly one function.This enforces the principle of single responsibility for step definitions<br>and makes steps easier to find, understand, and maintain.<br>&quot;&quot;&quot;<br>import ast<br>from pathlib import Path<br><br>def test_architectural_steps_have_single_function():<br>    &quot;&quot;&quot;<br>    Ensure each step file contains exactly one function.<br>    This enforces the principle of single responsibility for step definitions<br>    and makes steps easier to find, understand, and maintain.<br>    Each step file should contain:<br>    - One @given, @when, or @then decorated function<br>    - Helper functions are discouraged - use shared utilities instead<br>    &quot;&quot;&quot;<br>    step_files_with_violations = []<br>    steps_root = Path(&quot;tests/architectural/steps&quot;)<br>    if not steps_root.exists():<br>        # No steps directory found, test passes<br>        return<br>    for step_file in steps_root.rglob(&quot;*.py&quot;):<br>        # Skip __init__.py files and utility files        if (step_file.name == &quot;__init__.py&quot; or<br>            &quot;common&quot; in step_file.name.lower() or<br>            &quot;util&quot; in step_file.name.lower() or<br>            step_file.name.endswith(&quot;_utils.py&quot;)):<br>            continue<br>        try:<br>            with open(step_file, &#39;r&#39;, encoding=&#39;utf-8&#39;) as f:<br>                content = f.read()<br>            # Parse the Python file<br>            tree = ast.parse(content)<br>            # Count top-level function definitions only (not nested functions)<br>            function_count = sum(1 for node in tree.body<br>                               if isinstance(node, ast.FunctionDef))<br>            if function_count != 1:<br>                relative_path = str(step_file)<br>                step_files_with_violations.append(f&quot;{relative_path} (has {function_count} functions)&quot;)<br>        except Exception as e:<br>            # If we can&#39;t parse the file, report it as a violation<br>            relative_path = str(step_file)<br>            step_files_with_violations.append(f&quot;{relative_path} (parse error: {e})&quot;)<br>    if step_files_with_violations:<br>        violations = &quot;\n  - &quot;.join(step_files_with_violations)<br>        raise AssertionError(<br>            f&quot;The following step files violate single function rule:\n  - {violations}\n\n&quot;<br>            f&quot;Each architectural step file should contain exactly one function. &quot;<br>            f&quot;Consider splitting multi-function files or moving helper functions to shared utilities.&quot;<br>        )</pre><p>This combination of length checks and content verification helps maintain both the size constraints and the architectural integrity of my codebase. When AI code editors generate or modify files, these tests catch violations immediately.</p><h3>The Benefits</h3><p>The first benefit hits you immediately: git status tells you everything your AI code editor altered.</p><p>When files are appropriately sized, the list of modified files becomes a readable summary of what the AI changed. When changes are small iterations, this is very effective. I can scan the output and instantly understand the scope of modifications:</p><pre>modified:   domain/orders/services.py<br>modified:   domain/orders/models.py<br>modified:   tests/domain/test_order_service.py<br>modified:   infrastructure/notifications.py</pre><p>This tells me the AI touched order services, the order model, related tests, and notifications. I know exactly where to look. I can also see the AI code editor didn’t touch my architecture tests so I know my rules defining how code should be structured and written is being enforced still.</p><p>Compare this to working with large files. When your codebase has files like services.py with 2000 lines containing multiple service classes, git status becomes useless for understanding what changed. The AI might have modified one small section, but git status just shows &quot;services.py changed&quot; without indicating whether it touched payment processing, order fulfillment, or user management.</p><p>The git diff becomes more valuable too. When AI modifies a 300-line file, I can see exactly what changed. When it modifies a 2000-line file, the diff becomes a wall of text that I’m likely to skim rather than review properly.</p><p>AI code editors understand smaller files better. I’ve noticed they make fewer mistakes when working with files in this range. They’re less likely to:</p><p>Modify the wrong function because they lost context midway through the file. Miss edge cases because they couldn’t see the full scope. Break existing functionality because they didn’t realize a function was called from multiple places. Create duplicate implementations because they forgot what already existed.</p><p>The three-attempt limit I use for AI tasks works better with appropriately-sized files. When an AI agent fails to complete a task correctly, smaller files make it easier to identify what went wrong. The agent can re-read the entire file, understand the problem, and try again. With larger files, context gets lost between attempts.</p><p>Architectural testing becomes more precise. My pytest-archon rules can pinpoint violations more accurately when each file has a clear, focused responsibility. When files grow too large, they tend to accumulate multiple concerns, making architectural rules harder to define and enforce.</p><p>Token efficiency improves. AI code editors consume tokens when reading and modifying files. A 300-line file fits comfortably in context. A 2000-line file might need to be processed in chunks, leading to inconsistencies and mistakes.</p><p>Pre-commit hooks run faster. My architectural integrity tests process smaller files more quickly, giving faster feedback. When hooks take too long, developers start skipping them. Fast tests keep the feedback loop tight.</p><h3>The Drawbacks</h3><p>This approach creates more files. My Django project has thousands of files, and navigating them requires good tooling. I rely heavily on IDE search features and clear naming conventions. If you’re used to scrolling through large files, the cognitive shift takes time as it means I have to update architectural tests that verify the structure of my project.</p><p>The overhead of file management increases. More files mean more imports, more navigation, and more mental overhead tracking where things live. This is real friction, especially when working across multiple related components.</p><p>Finding the right split points takes skill. Bad splits create artificial boundaries that make code harder to understand. I’ve created splits that seemed logical at the time but later realized they separated closely related concepts. Refactoring these mistakes takes time.</p><p>Some patterns become awkward. Large data structures, comprehensive test suites, or complex state machines sometimes feel cramped in 500 lines. I’ve experimented with different approaches here, and honestly, I don’t have perfect answers yet.</p><p>The 150-line floor can be contentious. Some developers prefer even smaller files with single classes or functions. Others find 150 lines too restrictive. I’ve settled on this range through experimentation, but your experience might be different.</p><p>There’s a cost to enforcing this. I’ve spent time refactoring files that worked fine but exceeded the limit. Whether that time investment pays off depends on how actively AI code editors modify those files.</p><h3>What I’ve Learned</h3><p>File size constraints aren’t about code quality in the traditional sense. They’re about optimizing for a new workflow where AI agents do significant code generation and modification.</p><p>The 150–500 line range has made my AI-assisted development more reliable. Diffs are reviewable. Context is manageable. Mistakes are easier to catch and fix. The feedback loop stays tight.</p><p>This has helped me reduce the time it takes to review code generated from AI code editors and it should reduce the time when working with other engineers also.</p><p>I recently moved to Sonnet 4.5 and am more confident than ever with model changes. My unit, integration and acceptance tests enforce the implementation of my service is correct and my architectural tests enforce my prompts are being followed. My performance tests ensure the service runs performantly. The combination of appropriately-sized files and comprehensive testing creates a foundation where I can upgrade models without fear.</p><p>What matters is understanding why file size affects AI code editors and making informed decisions based on your specific context. In codebases where AI does heavy lifting, these constraints have been valuable for me.</p><p>If you’re working with AI code editors on large projects, I’d be interested to hear whether similar constraints help or hinder your workflow.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=340d550dcea4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Keeping Your Django Codebase in Line: Testing Architectural Integrity with pytest-archon]]></title>
            <link>https://medium.com/@eamonn.faherty_58176/keeping-your-django-codebase-in-line-testing-architectural-integrity-with-pytest-archon-7ddc208d28ad?source=rss-0021d5c19a0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ddc208d28ad</guid>
            <category><![CDATA[ai-code-assistant]]></category>
            <dc:creator><![CDATA[eamonn faherty]]></dc:creator>
            <pubDate>Fri, 26 Sep 2025 14:09:36 GMT</pubDate>
            <atom:updated>2025-10-01T13:48:20.370Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Part of the “</em><a href="https://medium.com/@eamonn.faherty_58176/getting-the-most-fromai-code-editors-1bd7e097acf6"><em>Getting the Most from AI Code Editors</em></a><em>” series</em></p><figure><img alt="Architectural integrity automated testing for AI coding agents" src="https://cdn-images-1.medium.com/max/1024/1*CDLwjssAB4ivO5SD1EFtyQ.png" /></figure><p>After hitting 160k lines of code in my experimental Django project, I learned something important: AI code editors accelerate code generation, but they can also accelerate the creation of architectural mess. This post covers how I’ve been using pytest-archon to maintain architectural discipline while letting AI agents do the heavy lifting.</p><h3>What is Architectural Integrity?</h3><p>Architectural integrity includes two principles that I’ve found critical in large codebases: single responsibility and separation of concerns.</p><p>Single responsibility means each module should have one reason to change. Separation of concerns means different aspects of your application should be handled by distinct components that don’t know too much about each other.</p><p>Here’s what I mean with a Django e-commerce example:</p><pre># Clean architecture - what I aim for<br># myproject/domain/orders/models.py<br>class Order:<br>    def calculate_total(self):<br>        return sum(item.price * item.quantity for item in self.items.all())<br><br># myproject/domain/orders/services.py  <br>class OrderService:<br>    def create_order(self, cart_items, user):<br>        order = Order.objects.create(user=user)<br>        # ... order creation logic<br>        return order<br><br># myproject/infrastructure/notifications.py<br>class EmailNotificationService:<br>    def send_order_confirmation(self, order):<br>        # Email sending logic here<br>        pass</pre><p>Compare this to what happens when architectural discipline breaks down:</p><pre># Architectural mess - what I try to avoid<br>class Order:<br>    def calculate_total(self):<br>        return sum(item.price * item.quantity for item in self.items.all())<br>    <br>    def send_confirmation_email(self):<br>        # Domain model handling email? Bad idea.<br>        send_mail(...)<br>        <br>    def process_payment(self):<br>        # Domain model handling payments? Even worse.<br>        stripe.Charge.create(...)</pre><p>When the Order model starts handling emails and payments, you’ve lost. Changes to email templates now require touching your core domain logic. Payment provider changes break your order model. Testing becomes a nightmare.</p><h3>How I Automate Architectural Integrity Testing</h3><p>Manual code reviews don’t scale. I learned this the hard way. <a href="https://github.com/jwbargsten/pytest-archon">pytest-archon</a> lets me codify architectural rules as automated tests. There are other libraries available but I found pytest-archon the best fit for me.</p><p>The cost difference is stark. Manual reviews are expensive, slow, and inconsistent. Automated tests run in seconds and catch violations immediately.</p><p>I use git pre-commit hooks to enforce these rules:</p><pre># .pre-commit-config.yaml<br>repos:<br>  - repo: local<br>    hooks:<br>      - id: architecture-tests<br>        name: Architecture Integrity Tests<br>        entry: uv pytest tests/test_architecture.py -v<br>        language: system<br>        pass_filenames: false</pre><p>Every commit attempt runs my architectural tests. If they fail, the commit is blocked.</p><p>When working with AI code editors, I protect my architectural tests from modification. I use specialized sub-agents with single responsibilities. After the AI finishes its work, I check git diffs to ensure my test files weren’t touched and validate the test report.</p><h3>Creating a Workflow for AI Code Editor Verification</h3><p>I’ve built a feedback loop where AI code editors verify their own work against architectural constraints.</p><p>In Claude Code, I implement verification hooks:</p><pre># .claude/hooks/verify_architecture.py<br>def verify_task_completion():<br>    &quot;&quot;&quot;Check architecture tests before marking task complete&quot;&quot;&quot;<br>    result = subprocess.run([&#39;uv&#39;, &#39;pytest&#39;, &#39;tests/test_architecture.py&#39;, &#39;-v&#39;], <br>                          capture_output=True, text=True)<br>    <br>    if result.returncode != 0:<br>        print(&quot;❌ Architecture tests failed:&quot;)<br>        print(result.stdout)<br>        return False<br>        <br>    return True</pre><p>I also create verification prompts for the AI:</p><pre>Check your work against architectural rules:<br>1. Run: uv pytest tests/test_architecture.py -v<br>2. If tests fail, fix the violations<br>3. Don&#39;t modify the architecture test files<br>4. Re-run tests until they pass<br>Current results: {test_output}</pre><p>This creates a self-correcting system. The AI validates its own work.</p><h3>My Experience with a 160k Line Django Project</h3><p>I’ve been experimenting with this approach on my Django project that currently sits at 162,595 lines of source code. Here’s what I’ve learned.</p><p>I implemented a three-attempt limit for AI code generation. If the architecture tests don’t pass after three tries, I stop and review manually. This prevents wasted tokens and time when I haven’t explained my requirements clearly enough or where my architectural rules are too stringent.</p><p>The feedback loop has been reliable. Key observations:</p><p>Sub-agents respect their boundaries when given clear single responsibilities. The architecture testing agent hasn’t modified my rules inappropriately. Violations are caught within seconds rather than during code review.</p><p>I see potential for using architectural rules to describe coding preferences more concisely:</p><pre># Instead of long prose instructions<br>coding_instructions = &quot;&quot;&quot;<br>Follow architectural rules in tests/test_architecture.py:<br>- Domain purity (test_domain_purity)<br>- App isolation (test_app_isolation)  <br>- Dependency direction (test_dependency_direction)<br>&quot;&quot;&quot;</pre><p>I haven’t had time to fully experiment with this approach yet, but it could make AI instructions more precise.</p><p>Here are the architectural rules I’ve found most effective:</p><pre>from pytest_archon.rule import archrule<br><br><br>def test_no_direct_database_imports_in_steps():<br>    &quot;&quot;&quot;<br>    Ensure acceptance steps don&#39;t import Django ORM or database utilities directly.<br><br>    This prevents steps from bypassing the web interface to manipulate data directly.<br>    All data setup should go through admin/frontend views.<br><br>    Exception: django.contrib.auth.models is allowed for client.force_login setup.<br>    &quot;&quot;&quot;<br>    (archrule(&quot;Acceptance steps must not import Django ORM&quot;,<br>              &quot;Steps should interact with views, not manipulate database directly&quot;)<br>     .match(&quot;tests.acceptance.steps.*&quot;)<br>     .should_not_import(&quot;django.db.*&quot;, &quot;django.core.management.*&quot;, &quot;*.fixtures.*&quot;)<br>     .may_import(&quot;django.contrib.auth.models&quot;)  # Allow auth models for client.force_login<br>     .check(&quot;tests&quot;))</pre><pre>from pytest_archon.rule import archrule<br><br><br>def test_acceptance_steps_do_not_import_domain_code():<br>    &quot;&quot;&quot;<br>    Acceptance steps should only interact with views,<br>    never directly with domain models or business logic.<br><br>    This enforces the BDD principle that tests should simulate real user<br>    interactions through the web interface, not bypass it via direct model access.<br><br>    Violations would indicate:<br>    - Steps bypassing the web interface<br>    - Direct database manipulation in tests<br>    - Breaking the separation between test layers and business logic<br><br>    Exceptions:<br>    - django.contrib.auth.* is allowed for client.force_login test utility<br>    - domain.accounts.utils.initialize_user_data is allowed for user setup<br>      in background steps that create test users (should eventually be replaced<br>      with proper signup flow through frontend)<br>    &quot;&quot;&quot;<br>    (archrule(&quot;Acceptance steps must not import domain code&quot;,<br>              &quot;BDD steps should only interact with views, not domain models directly&quot;)<br>     .match(&quot;tests.acceptance.steps.*&quot;)<br>     .should_not_import(&quot;domain.*&quot;)<br>     .may_import(&quot;django.contrib.auth.*&quot;, &quot;domain.accounts.utils&quot;)<br>     .check(&quot;tests&quot;))</pre><pre>from pytest_archon.rule import archrule<br><br><br>def test_page_objects_do_not_import_domain_code():<br>    &quot;&quot;&quot;<br>    Page objects should also not import domain code directly.<br><br>    Page objects represent the UI layer and should interact with views,<br>    not bypass them to access domain models directly.<br>    &quot;&quot;&quot;<br>    (archrule(&quot;Page objects must not import domain code&quot;,<br>              &quot;Page objects represent UI layer and should not bypass views&quot;)<br>     .match(&quot;tests.pageobjects.*&quot;)<br>     .should_not_import(&quot;domain.*&quot;)<br>     .check(&quot;tests&quot;))</pre><p>Whilst this doesn’t use pytest_archon I found this useful for ensuring python files only have a single function (single test per file and single view per file). Having a single function per file helps me spot where AI is making changes more easily when I perform a git diff.</p><pre>import ast<br>from pathlib import Path<br><br><br>def test_acceptance_steps_have_single_function():<br>    &quot;&quot;&quot;<br>    Ensure each acceptance step file contains exactly one function.<br><br>    This enforces the principle of single responsibility for step definitions<br>    and makes steps easier to find, understand, and maintain.<br><br>    Each step file should contain:<br>    - One @given, @when, or @then decorated function<br>    - Helper functions are discouraged - use shared utilities instead<br>    &quot;&quot;&quot;<br>    step_files_with_violations = []<br>    steps_root = Path(&quot;tests/acceptance/steps&quot;)<br><br>    if not steps_root.exists():<br>        # No steps directory found, test passes<br>        return<br><br>    for step_file in steps_root.rglob(&quot;*.py&quot;):<br>        # Skip __init__.py files and utility files<br>        if (step_file.name == &quot;__init__.py&quot; or<br>            &quot;common&quot; in step_file.name.lower() or<br>            &quot;util&quot; in step_file.name.lower() or<br>            step_file.name.endswith(&quot;_utils.py&quot;)):<br>            continue<br><br>        try:<br>            with open(step_file, &#39;r&#39;, encoding=&#39;utf-8&#39;) as f:<br>                content = f.read()<br><br>            # Parse the Python file<br>            tree = ast.parse(content)<br><br>            # Count top-level function definitions only (not nested functions)<br>            function_count = sum(1 for node in tree.body<br>                               if isinstance(node, ast.FunctionDef))<br><br>            if function_count != 1:<br>                relative_path = str(step_file)<br>                step_files_with_violations.append(f&quot;{relative_path} (has {function_count} functions)&quot;)<br><br>        except Exception as e:<br>            # If we can&#39;t parse the file, report it as a violation<br>            relative_path = str(step_file)<br>            step_files_with_violations.append(f&quot;{relative_path} (parse error: {e})&quot;)<br><br>    if step_files_with_violations:<br>        violations = &quot;\n  - &quot;.join(step_files_with_violations)<br>        raise AssertionError(<br>            f&quot;The following step files violate single function rule:\n  - {violations}\n\n&quot;<br>            f&quot;Each acceptance step file should contain exactly one function. &quot;<br>            f&quot;Consider splitting multi-function files or moving helper functions to shared utilities.&quot;<br>        )</pre><h3>Getting Started</h3><p>Install pytest-archon: uv add pytest-archon</p><p>Start with basic rules for high-level separation. Set up pre-commit hooks to catch violations early. Configure AI workflows with verification feedback loops. Iterate based on what actually happens in your codebase.</p><h3>What I’ve Learned</h3><p>Architectural integrity testing provides a foundation for AI-assisted development. By automating verification of design principles, I can let AI agents generate code while maintaining long-term codebase health.</p><p>The combination of automated testing, git hooks, and AI feedback loops creates a system that’s both flexible and disciplined. Architecture stays intact, AI agents stay productive, and the codebase remains maintainable.</p><p>In an era where AI can generate thousands of lines of code in minutes, these architectural guardrails aren’t optional. They’re essential for keeping large codebases from becoming unmaintainable messes.</p><p>I’m still learning, but this approach has worked well so far. If you’re dealing with similar challenges, I’d be interested to hear how it works for you.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ddc208d28ad" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>