I did a little (informal) experiment this weekend that reminded me how expensive innocent-looking code can be, why the right data structure matters, and why my CS101 professors made such a big deal about Big-O notation.

I’m weird, so every once in a while, when I have a few minutes to kill, I pick a random LeetCode problem and solve it (old school, without AI). These days, AI helps me with the vast majority of the code I write professionally, so I still enjoy doing it the traditional way sometimes, just for fun. As I was waking up and enjoying my morning coffee, I rolled the dice and got 242 Valid Anagram.

This is a classic (easy) LeetCode question: given two lowercase strings, determine whether one is an anagram of the other (an “anagram” is a word or phrase made by mixing up the letters of another word or phrase).

V1: using a map

I deliberately started with a somewhat sloppy but “reasonable-looking” implementation.

It counts how many times each character appears in the first string, then walks through the second string subtracting those counts. If it sees a character that was not in the first string, it returns false. If the second string uses more occurrences of a character than the first string contains, it also returns false. Else, if everything cancels out exactly, the strings are anagrams.

public boolean isAnagram(String s, String t) {

    Map<Character, Integer> count = new HashMap<>();

    for (int i = 0; i < s.length(); i++) {
        char c = s.charAt(i);
        count.put(c, count.getOrDefault(c, 0) + 1);
    }

    for (int i = 0; i < t.length(); i++) {
        char c = t.charAt(i);
        if (!count.containsKey(c)) {
            return false;
        }
        int newCount = count.get(c) - 1;
        if (newCount == 0) {
            count.remove(c);
        } else {
            count.put(c, newCount);
        }
    }

    return count.isEmpty();
}

I knew it wasn’t efficient, but it would get the job done. I was curious, so I submitted it.

It wasn’t just inefficient: 80% of submissions out there in the wild were better!!! The engineer in me was mortified, so I went down the rabbit hole.

My intuition was:

• A map was overkill, I could probably speed this up with an array
• Boxed integers were overkill, I could probably speed this up with primitive data types

V2: using an array

A map is overkill for this problem. The possible characters are a finite set. For this problem, the input is restricted to the 26 lowercase English letters, so I know the entire keyspace ahead of time (Unicode would make this slightly more complicated). Maps are great for dynamic or sparse keyspaces, but when you have a small number of fixed keys that map directly to array indexes, an array is much simpler.

So here’s the nearly identical algorithm, but using an array instead of a map. I took the liberty of making a few more simplifications, like comparing the lengths of the strings to short-circuit any obvious non-anagrams.

public boolean isAnagram(String s, String t) {

    // short-circuit if the strings don't have the same length
    if (s.length() != t.length()) {
        return false;
    }

    Integer[] count = new Integer['z' - 'a' + 1];

    // initialize the counters to 0 because default is null
    for (int i = 0; i < count.length; i++) {
        count[i] = 0;
    }

    for (int i = 0; i < s.length(); i++) {
        int index = s.charAt(i) - 'a';
        count[index]++;
    }

    for (int i = 0; i < t.length(); i++) {
        int index = t.charAt(i) - 'a';
        if (--count[index] < 0) {
            return false;
        }
    }

    return true;

}

One thing you might notice is I’m using Integer, not int. That was deliberate; I wanted to isolate what performance improvements would come from going from map to array, and going from boxed data types to primitive data types, as two different steps.

There’s a couple of other small differences, the short-circuit at the very beginning if the strings aren’t the same size, and also a loop to initialize all counter values to 0 (an Integer array defaults to null values, not zeroes, so I have to initialize it to all zeroes myself).

So that took it from 16 ms to 8 ms. It was a good improvement, but still not amazing. The biggest structural change was replacing the HashMap with direct array indexing, removing hashing, bucket lookups. Since the keyspace was only 26 known characters, direct array indexing was a much cheaper representation.

Next, it was time to test my theory that a lot of the wasted time was coming from the JVM boxing and unboxing integers. Let’s switch to primitive data types!

V3: using primitive data types

Boxed types are less efficient: in this case, an Integer[] stores references to Integer objects instead of primitive int values directly in the array, adding reference indirection plus boxing/unboxing around arithmetic. I was curious just how much of a difference using good old-fashioned ints everywhere would make.

private static final int ALPHABET_SIZE = 'z' - 'a' + 1;
private static final int FIRST_LETTER = 'a';

public boolean isAnagram(String s, String t) {

    if (s.length() != t.length()) {
        return false;
    }

    int[] count = new int[ALPHABET_SIZE];

    for (int i = 0; i < s.length(); i++) {
        count[s.charAt(i) - FIRST_LETTER]++;
    }

    for (int i = 0; i < t.length(); i++) {
        int index = t.charAt(i) - FIRST_LETTER;
        if (--count[index] < 0) {
            return false;
        }
    }

    return true;

}

That’s pretty simple and much easier to read as well. When I ran it, the time went from 8 ms to 4 ms. I reran each of those three versions several times because LeetCode timings can be noisy, and the 16 → 8 → 4 pattern held.

V4: using sort

While looking through the solutions others had submitted, I noticed a sorting solution. I thought it was kinda cheating to just sort the strings and compare them as that didn’t really demonstrate any particularly interesting coding skills, but I figured I’d do it for completeness.

public boolean isAnagram(String s, String t) {

    if (s.length() != t.length()) {
        return false;
    }

    char[] sArr = s.toCharArray();
    Arrays.sort(sArr);

    char[] tArr = t.toCharArray();
    Arrays.sort(tArr);

    return Arrays.equals(sArr, tArr);

}

While this version ran at 4 ms as well, I knew it was asymptotically worse. The time complexity is O(n log n) because sorting each character array dominates execution. And the space is O(n) because toCharArray() creates two new arrays, each proportional to the string length.

But my previous version, using primitive arrays, runs in O(n) and uses O(1) space. That makes the primitive-array version asymptotically better on both dimensions.

Takeaways

Three things jumped out at me.

1️⃣ Use the simplest data structure that actually matches your problem

A HashMap is a fantastic general-purpose abstraction. But generality has a cost. For a domain containing exactly 26 known values, a hash table is doing a lot of work I simply don’t need: hashing, lookups, buckets, objects, references, potentially collisions, and poorer memory locality. An array maps almost perfectly onto the actual problem.

2️⃣ Boxing is not free; use primitives if you can

This was the more interesting result to me. V2 (Integer[26]) and V3 (int[26]) used essentially the same algorithm. Yet moving from boxed integers to primitives cut the runtime roughly in half in this experiment. Please keep in mind that modern JVMs are clever. They cache some boxed values, optimize aggressively, and can eliminate some allocations. So I am NOT claiming that “boxing always makes Java 2x slower.” This is a tiny LeetCode experiment, not a rigorous JMH benchmark. But with Integer, operations that look simple can involve unboxing a reference into an int, doing arithmetic, then boxing the result again. You also replace a compact contiguous array of values with an array of references to objects.

3️⃣ Always understand Big-O notation

If I had just trusted LeetCode’s benchmark, I might have concluded that V3 (the array of primitives) and V4 (sorting) were “comparable” as they both ran in 4 ms. But they aren’t equivalent. V3 runs in O(n) time and O(1) space, while V4 runs in O(n log n) time and O(n) space. LeetCode’s benchmark didn’t expose that scaling difference. But as the input grows, V4’s O(n log n) runtime grows faster than V3’s O(n). A benchmark tells you what happened with the data you tested. Big-O tells you how the algorithm scales as the data grows. You need to understand both and apply some engineering judgment.

I will happily trade a few nanoseconds for readability, maintainability, and flexibility. But when something is in a hot loop, operating at enormous scale, or sitting inside infrastructure that runs billions of times, it is worth remembering what the machine underneath your abstraction is actually doing. That is true even today: don’t just blindly trust the code that your AI assistant generates, review it carefully and own the outcome.

• Carlos Arguelles, Senior Principal Engineer, Amazon
• Adithya Venkatesh, Principal Engineer, Amazon
• Alexander Müller, Principal Engineer, Amazon
• Carlos Puyuelo Rosas, Principal Engineer, Amazon
• Cameron McColl, Principal Engineer, Amazon
• Christopher Magnollay, Principal Engineer, Amazon
• Ashay Raut, Principal Engineer, Amazon
• Steven Mascola, Principal Engineer, Amazon

How do you create a high-fidelity, high-availability 𝐭𝐞𝐬𝐭 𝐞𝐧𝐯𝐢𝐫𝐨𝐧𝐦𝐞𝐧𝐭 for a vast, complex production call graph with layers and layers of thousands of services working together to serve traffic? This is our current challenge at Amazon.

When customers go to our Store on a browser or in their mobile shopping app, a long sequence of service calls spreads throughout a graph like this to serve you the content. In fact, this specific graph was Amazon the day I was hired in 2009… it happens to look quite a bit more complex than that 17 years later:

Building local changes and standing up an ephemeral endpoint for integration and end-to-end tests is not particularly difficult. But in a distributed system, a service never works in isolation. Services have dependencies. I’ve seen 3 common patterns here: [1] use test doubles (fakes, mocks or stubs) to replace real dependencies; [2] call production dependencies, or [3] call test dependencies.

None of these options is great. Your tests can’t safely call a production dependency if that call will change state (e.g. write to a database, trigger an external workflow, or invoke a third-party integration). In that case, you probably want to call the test stack of your dependency. But often test endpoints lack 𝐟𝐢𝐝𝐞𝐥𝐢𝐭𝐲 (they’re missing production-realistic data) and have poor 𝐚𝐯𝐚𝐢𝐥𝐚𝐛𝐢𝐥𝐢𝐭𝐲 (they’re never treated with the same operational rigor as production). You can replace these network calls with local test doubles (fakes, mocks or stubs) but these also often lack fidelity (not guaranteed to behave exactly like the real thing).

Ultimately there’s two foundational limitations:

• 𝐟𝐢𝐝𝐞𝐥𝐢𝐭𝐲. a copy of production is always that: just a copy, not the real thing. I can think of dozens of times that we found a problem only in production because the test environment didn’t accurately reflect production: in data, in config, in hardware, in scale. Configuration drift between environments has caused real incidents across the industry.
• 𝐚𝐯𝐚𝐢𝐥𝐚𝐛𝐢𝐥𝐢𝐭𝐲. running and operating test environments at the same quality bar as production environments is expensive and time consuming, so inevitably these environments end up being flakey. If you are testing a service with just 3 dependencies that have test environments with availability 99%, 95% and 93% (unacceptable for production but common for test environments), your tests will succeed at most 87% (99% x 95% x 93%). In a graph like the Amazon Store, hundreds of services need to reliably work for a single end-to-end test to pass.

There’s only one environment that has production fidelity and production availability: production. Could we 𝐭𝐞𝐬𝐭 𝐢𝐧 𝐩𝐫𝐨𝐝𝐮𝐜𝐭𝐢𝐨𝐧…𝐬𝐚𝐟𝐞𝐥𝐲?

“Testing in production” does not mean “testing unproven code on real customers.” It means safely and responsibly leveraging production infrastructure to validate code before it reaches any real customer, with multiple layers of isolation ensuring that test transactions are sandboxed, side effects are suppressed, and real customers are never exposed to experimental behavior.

The more we think about it, the more excited we are. You need 5 pillars for testing in production to be safe and feasible at scale. [1] some way to mark transactions as tests, something like a test token; [2] reliably propagate across all the system; [3] route test requests to the desired test endpoint; [4] override the required behaviors where needed; [5] safely handle all writes.

Pillar 1: Test Token

You need something to mark a transaction as being a “test” transaction or a “prod” transaction, so that all services in the graph can make appropriate handling choices.

You could use ephemeral customer test accounts, but it’s likely that not every single service in the graph is aware of the concept of “accounts,” so you need an approach that works more broadly. I still very much like using ephemeral test accounts as a blast radius mechanism, but I don’t think I would exclusively depend on it.

In a distributed system, every transaction is a chain of network calls from a service to another. You can have an agreed upon standard http header, you can build the propagation logic into a common RPC library and push it broadly to services in your graph without requiring each service owner to modify their code and write bespoke handling code.

You could initially think about this test token first as a boolean ‘isTest’. That will probably suffice at the beginning, but will lack the flexibility on differentiating what you want to do on each case. Eventually you’ll end up with a collection of specific overrides that you need across the graph to make this safe. It might be long and complex, like “I want you to route calls to ServiceX-Prod to ServiceX-Test, and I want to suppress the side effects of ActionY”.

One of the technical decisions we need is: do we put the entire override payload in the header, or simply a GUID so that you can fetch the payload later, on demand? There’s pros and cons to both. If you were to put the full override in the header you may be increasing payloads for the entire graph, impacting bandwidth in services that have nothing to do with your override. The other option is to simply inject a GUID. The presence of this field tells services that this is a test transaction, and that they need to call an OverrideService providing the GUID to get the actual override they may need to act on.

I like that option better because it keeps the header size constant, but the tradeoff is that you need to incur in an additional network call (increasing latency) to fetch the override. But there’s ways to make this more efficient. For example, if overrides are immutable (which I think is the right thing to do for safety anyways) you can cache them in individual services. Or, part of the test token could be a small Bloom filter that encodes which services the override applies for, and services can decide in constant time if they should call the OverrideService (minus false positives from the Bloom filter).

You also need to think long and hard about security. Tokens must be cryptographically signed and immutable, created only by authorized actors through a controlled issuance process, never modifiable after creation, and rejected if injected from untrusted boundaries.

Pillar 2: Token Propagation

But how do you ensure that the token is always propagated, particularly in the future? If any one of the thousands of services in the graph neglects to propagate the token, the rest of the call graph down from it will treat that test transaction as a production transaction.

One idea is to segment a small percentage of the production fleet and dedicate it exclusively to test traffic. Then, you can have network interceptors on egress just on that sub-fleet to make sure the token is always present for outgoing calls, and if not, short-circuit the entire transaction as it’s no longer safe to continue.

Another reason I like the idea of a sandboxed sub-fleet for test transactions is that if sharing compute with production transactions, test transactions could theoretically cannibalize resources, affecting real production customers. Same infrastructure, but never the same compute.

Keep in mind that most complex distributed systems are not a single synchronous graph, but actually a collection of asynchronous graphs, connected by queues or databases. Token propagation may die at the async boundaries unless you bake it into the asynchronous messages too. For example, you could have an interceptor for Amazon SQS that attaches an attribute to mark a message as a test message when enqueuing it, and later reads the attribute when dequeing to continue propagating the token.

Having an OverrideService has a nice additional benefit: it can operate as a global Andon cord. If you need to cancel overrides for everything all at once (e.g. sev1 affecting the entire ecosystem), or a specific service, or a specific region, or a specific actor, you can control those from the OverrideService, which could simply return an error code to short-circuit the test transaction. And the OverrideService can act as an authorization layer — only providing overrides to actors that are appropriately authorized to act on that override.

Pillar 3: Traffic Routing

Let’s say you have some changes you want to make to your service, in your local workspace. You want to deploy these changes to an ephemeral test environment, and inject it into the production graph. In other words, you need to override specific calls (the ones marked with your token) so that your test endpoint is called instead of the standard production endpoint. To do that, you need a router that sits between calls, inspects the token, and re-routes accordingly.

You can have a network proxy between calls (aka a service mesh) that gives you a layer of indirection and control without changing application code. This approach comes with extra latency (few extra milliseconds between each call), but there are lots of (non-test related) benefits to a service mesh, particularly around traffic control: retries, timeouts, and circuit breaking, load balancing, throttling, etc. And since a network proxy is an independent process, you don’t need to change application code; it’s a sidecar.

One thing I particularly like about having a network proxy in the context of testing is that it can intelligently shed test traffic if a production service is under stress. Prod transactions are always, always more important than test transactions, so it’s necessary to kill them in periods of high customer load.

If you don’t want the overhead and complexity of a service mesh, you can inspect the token before you make the network call (ServiceD would choose to call G-Test instead of G-Prod), or after the network call (in this case, Service D calls Service G Prod, which proxies to G-test). The downside of this is that you need to change application code for every service to enable routing like this, instead of handling it globally via standard mesh technology.

Routing controls require the same security rigor as the token itself. Only authenticated, authorized actors can register a test endpoint for routing, and the system must enforce that only code reviewed and built through standard CI/CD can be injected into the graph. All routing overrides must be fully auditable: who created the override, what code was deployed to the test endpoint, and when.

Pillar 4: Behavioral Overrides

What happens when calls to your downstream dependencies have side effects? Let’s say one of your dependencies is a service that triggers irreversible external workflows, or one that initiates a financial transaction with a third-party provider. Your test needs to call those dependencies, but avoid the actual side effects. This is where behavioral overrides come into place. Non-idempotent calls always make testing more difficult.

In this case, a behavioral override would simply say: “This is a test transaction, so suppress the external side effect and return a synthetic success response” But understand that anytime you override a behavior for testing purposes, you’re losing a little bit of fidelity in your tests. You’re no longer testing exactly the true production path, you’ve deviated ever so-slightly. It’s a necessary tradeoff if you simply can’t accept the side effects.

For this reason, I wouldn’t want to see thousands of overrides popping up just because it’s possible to add them. You may want a central body of governance to ensure every single behavioral override is there for a good reason and is used sensibly. This introduces friction and is hard to scale, but it might be necessary.

I’m also a fan of creating vetted groupings of overrides that can only be used atomically, rather than tests mixing and matching individual overrides. Overrides often have logical dependencies on each other: disabling one behavior without disabling other related behaviors can produce nonsensical or unsafe test scenarios. For example, if you suppress an external action, you should also suppress its downstream workflows; these are logically coupled actions that only make sense when overridden together. Atomic groupings enforce these invariants by design, preventing partial overrides that would compromise the integrity of the test.

Pillar 5: Safely Handling Writes

Under no circumstance do you want a test write to pollute a production database. The idea is really a different flavor of the network routing problem. Have an interceptor before every write, and if it’s a test write, handle it differently. For example, Prod writes can go to a production database and test writes can go to a test database. Or, they could go to a single database with an extra field to mark test rows. There’s more sophisticated things you could do if you have the ability to control the database itself but a single interceptor that follows one of these two patterns can go a long way otherwise.

“Writes to databases” is the most obvious write case, but not the only one. It could be publishing a message to a queue. Or, as your service is operating, it’s writing logs. These logs may be pushed out to other systems for auditing, attribution, and various other reasons. You may want to split your test logs from your production logs. Likewise, your service is emitting metrics (e.g. AWS CloudWatch). Is it ok for the metrics from test traffic to be mixed in with the metrics from production traffic, or do you want to emit separate streams?

Another unexpected place where “writes” show up is in caches. In services that cache data, are you ok with caching test data? My gut feeling is that no, I would generally not want to cache test data, because that is using valuable real estate in a cache that should be used to make the experience of production customers more performant — or worse, a cached value from a test could override a cached value for prod.

Why do all this work?

What I described so far is not an insignificant amount of work, with a lot of “dot the i’s and cross the t’s” and a fair bit of risk. But I’ve been in the space of test environments for 20 years and getting it right has been very challenging with engineers routinely face significant toil from instability in the environments. As AI-driven development accelerates the pace of code generation across the industry, reliable test infrastructure becomes even more critical. AI needs trust-worthy signals to validate that it created a correct implementation for a builder’s intent. Ultimately, testing in production is the only thing that provides high fidelity and high availability.

It also opens the avenue for other interesting capabilities.

• It makes it possible to atomically test changes to entirely unrelated services, which couldn’t be easily tested together before. You simply create an override with multiple network redirects.
• It makes it possible to mock data. For example, what if you’re making a UI change and you need to test how the UI renders a really long product name in the Amazon Store? You could hard-code an ASIN with a really long product name from the real Store in your tests, OR you could simply ask the service in charge of fetching the product name to return a fake long product name just to trigger the code paths you’re trying to test.
• It provides an interesting foundation for running stress and chaos tests safely. You can create an override that requests that calls to service X should fail a percentage of the times (faking a partial or total outage), or that they have an extra 100ms of latency, to see how that would impact the ecosystem. Since the transactions are sandboxed, you’re not affecting the entirety of production, but you are seeing the impact of problems like these in a real production graph. Best of both worlds!

If Medium puts this content behind a paywall, you can also view it here (LinkedIn).

For almost 30 years, I have worked in Engineering Productivity in one form or another. I have spent time at Amazon, Google, and Microsoft, and every large software company has the same underlying problem: at that scale, small inefficiencies compound into developer fatigue, wasted infrastructure, slower delivery, and missed business opportunities.

For almost 30 years, I have worked in Engineering Productivity in one form or another. I have spent time at Amazon, Google, and Microsoft, and every large software company has the same underlying problem: at that scale, small inefficiencies compound into developer fatigue, wasted infrastructure, slower delivery, and missed business opportunities.

Yet those inefficiencies often remain unfixed because the pain is distributed across thousands of people, each inconvenienced only slightly. No single person or team experiences enough pain to take ownership. Fixing the problem requires someone to act as an 𝐎𝐰𝐧𝐞𝐫 by quantifying the aggregate cost, defining a credible solution, convincing a senior leader to fund it, and making sure it gets done.

The waste is spread across people, teams, and budgets, which makes it almost invisible. And the irony is that, when you look at the problem holistically, fixing it often costs the company only a fraction of what it is already wasting by allowing the inefficiency to persist.

I wanted to use a real-world example from this week to illustrate that idea. It also highlights three principles I have learned repeatedly:

1. You do not need to be the domain expert to recognize an inefficient system
2. You do not need perfect data to decide that a problem is worth investigating
3. You do not need a perfect solution, be scrappy and iterative and learn

You need enough understanding to form a hypothesis, enough directional data to estimate whether the impact is meaningful, and enough curiosity to test a simple solution.

On Thursday, I landed at Seattle-Tacoma International Airport on Turkish Airlines flight TK203, returning from a visit to my Amazon Berlin team with a stopover in Istanbul. I reached the baggage carousel and waited. And waited. And waited. It took more than 30 minutes for my suitcase to appear. When I finally grabbed it, hundreds of bags were still circling the carousel, while hundreds of increasingly impatient passengers continued waiting.

As I watched the system operate, I realized it was a perfect example of the kind of problem I see in my day job. This is a small inefficiency that inconveniences hundreds of people on one flight and thousands over the course of a day. For each passenger, the impact is annoying but tolerable. In aggregate, and over time, the human cost is massive (SeaTac handled more than 52 million passengers in 2025).

The pain is distributed. The passengers experiencing it do not own the system. The people operating individual parts of the baggage process might not see the end-to-end impact. And the person with the authority to fix it might never experience the problem, much less quantify it. There is no clear owner, no visible measure of the total cost, and little incentive for any individual person or team to act.

As I watched the bags enter the carousel, I noticed that they were arriving in 𝐝𝐢𝐟𝐟𝐞𝐫𝐞𝐧𝐭 𝐨𝐫𝐢𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧𝐬. Some were lying flat with their long side parallel to the direction of travel, taking up the most space. Others were also lying flat but rotated 90 degrees, so they occupied somewhat less carousel length. A smaller number were positioned on their thin edge, taking up substantially less room.

Near the point where the luggage entered the carousel, there was a sensor that checked whether enough space was available before releasing another bag. When the carousel became too crowded, the feed paused. That safeguard makes sense. You do not want luggage colliding, piling up, or falling off the belt.

The problem was that many of the bags already on the carousel had gone around several times without being claimed. Because they were using much more space than necessary, the system frequently decided that there was not enough room to release another bag. The same bags continued going around and around. There was plenty of activity, but very little throughput.

Meanwhile, hundreds of bags were still waiting behind the entry point, and hundreds of passengers were waiting for those bags.

𝐖𝐡𝐚𝐭 𝐢𝐟 𝐰𝐞 𝐣𝐮𝐬𝐭 𝐭𝐮𝐫𝐧𝐞𝐝 𝐞𝐯𝐞𝐫𝐲 𝐬𝐮𝐢𝐭𝐜𝐚𝐬𝐞 𝐨𝐧𝐭𝐨 𝐢𝐭𝐬 𝐭𝐡𝐢𝐧 𝐞𝐝𝐠𝐞? [Well, not every bag could or should be positioned that way: soft-sided luggage, duffel bags, and unusually shaped or unstable items might require a different orientation. The real goal is to position each bag so that it uses as little carousel as practical while remaining stable, safe, and easy to retrieve.]

Since I was just waiting, I started doing some rough math to keep myself entertained.

Suppose placing a suitcase on its thin edge allows it to occupy 30% less belt length than one flat orientation and 60% less than another flat orientation. If those two inefficient orientations occur equally often, the average reduction would be: (0.5×30%) + (0.5×60%) = 45%.

In this intentionally simplified model, 𝐩𝐫𝐨𝐩𝐞𝐫𝐥𝐲 𝐨𝐫𝐢𝐞𝐧𝐭𝐞𝐝 𝐛𝐚𝐠𝐬 𝐰𝐨𝐮𝐥𝐝 𝐮𝐬𝐞 𝐚𝐛𝐨𝐮𝐭 𝟓𝟓% 𝐨𝐟 𝐭𝐡𝐞 𝐬𝐩𝐚𝐜𝐞, so the 𝐜𝐚𝐫𝐨𝐮𝐬𝐞𝐥 𝐜𝐨𝐮𝐥𝐝 𝐚𝐜𝐜𝐨𝐦𝐦𝐨𝐝𝐚𝐭𝐞 𝟖𝟐% 𝐦𝐨𝐫𝐞 𝐛𝐚𝐠𝐬 (1÷0.55=1.82) before becoming saturated.

Of course, this is only a rough model. Data scientists, please do not bring your pitchforks and torches. It’s not meant to have scientific rigor, just scrappy math. Suitcases come in different shapes and sizes. Bags would not fit together perfectly. Curves in the carousel create gaps, and passengers still need enough room to retrieve their luggage safely. But 𝐝𝐢𝐫𝐞𝐜𝐭𝐢𝐨𝐧𝐚𝐥𝐥𝐲, this back-of-the-napkin calculation hinted that deliberate bag orientation could still make a meaningful difference in throughput.

𝐖𝐚𝐢𝐭𝐢𝐧𝐠 𝟑𝟎 𝐦𝐢𝐧𝐮𝐭𝐞𝐬 𝐟𝐨𝐫 𝐦𝐲 𝐬𝐮𝐢𝐭𝐜𝐚𝐬𝐞 was an annoying but tolerable inconvenience. But that is the individual view. The system-level cost is the delay multiplied across everyone experiencing it.

Could I back-of-the-napkin estimate it? The Airbus A350–900 that brought me home from Istanbul was configured to carry 𝟑𝟐𝟗 𝐩𝐚𝐬𝐬𝐞𝐧𝐠𝐞𝐫𝐬. Suppose the flight was 𝟗𝟎% 𝐟𝐮𝐥𝐥 and 𝟗𝟎% 𝐨𝐟 𝐩𝐚𝐬𝐬𝐞𝐧𝐠𝐞𝐫𝐬 𝐡𝐚𝐝 𝐜𝐡𝐞𝐜𝐤𝐞𝐝 𝐥𝐮𝐠𝐠𝐚𝐠𝐞. That would mean ~𝟐𝟔𝟔 𝐩𝐚𝐬𝐬𝐞𝐧𝐠𝐞𝐫𝐬 𝐰𝐚𝐢𝐭𝐢𝐧𝐠 𝐚𝐭 𝐭𝐡𝐞 𝐜𝐚𝐫𝐨𝐮𝐬𝐞𝐥.

If they waited an average of 30 minutes, that’s 266×30 = 7,980 passenger-minutes, which is 𝟏𝟑𝟑 𝐩𝐚𝐬𝐬𝐞𝐧𝐠𝐞𝐫-𝐡𝐨𝐮𝐫𝐬 waiting for the luggage from a single flight. That’s a lot of waste.

My earlier calculation suggested that orienting the bags more efficiently could increase the carousel’s effective capacity by 82%. If carousel saturation were the dominant bottleneck, and that additional capacity translated into a proportional increase in throughput, the average wait could theoretically fall to: 30÷1.82 = 16.5 minutes. That would save 30−16.5 = 13.5 minutes per passenger. Let’s be conservative and hypothesize that 𝐰𝐞 𝐜𝐨𝐮𝐥𝐝 𝐬𝐚𝐯𝐞 𝟏𝟎 𝐦𝐢𝐧𝐮𝐭𝐞𝐬 𝐩𝐞𝐫 𝐩𝐚𝐬𝐬𝐞𝐧𝐠𝐞𝐫. Across 266 passengers, that would save: 266×10 = 2,660 passenger-minutes. That equals approximately 𝟒𝟒 𝐩𝐚𝐬𝐬𝐞𝐧𝐠𝐞𝐫-𝐡𝐨𝐮𝐫𝐬 on a single flight.

This is why small inefficiencies matter at scale. Ten minutes may not be enough for any one passenger to demand that someone redesign the process. But collectively, it is more than an entire workweek of human time, every time a flight like this arrives.

Of course, simplified efficiency arguments can become misleading. Better orientation would not make other potential bottlenecks faster, such as [1] opening and unloading the aircraft; [2] transporting luggage from the aircraft; [3] sorting and routing the bags; [4] delivering the first bag to the carousel; etc. It would improve only the carousel congestion portion of the process, specifically when it is full and the system is preventing more luggage from entering.

These are obviously just estimates. To calculate the real result, we would need actual data: number of passengers, number of checked bags, passenger arrival times at baggage claim, first-bag and last-bag delivery times, carousel occupancy over time, frequency and duration of feed pauses, bag dimensions and orientations, upstream baggage-handling constraints, …

But this is often how an investigation begins: You observe something, you form a hypothesis, you build a simple model, then you decide whether the potential impact is large enough to justify collecting better data.

So, should we embark on an expensive multi-year redesign of the hardware? No. My immediate solution would be surprisingly scrappy and low-tech. What if one person stood near the entry point and consistently turned each suitcase onto its thinner side?

Swissport, the company that handles luggage at SeaTac, has job listings for baggage handlers for $22/hr. We could add less than a dime to the cost of each ticket to fund the position, and it could potentially save 44 passenger-hours of waiting [the actual employment cost would be somewhat higher once benefits, payroll taxes, training, and overhead were included, but we would still be talking about pennies per ticket, not dollars].

As a customer, I would have gladly paid an extra dime to save ten minutes of my life waiting there.

This illustrates another critical part of Engineering Productivity: always be scrappy and iterative when validating an idea. Start with the simplest experiment that can validate the hypothesis, and learn relentlessly.

We could run an experiment for N months where a baggage handler positions all bags on their thin edge as they come out, and we measure how much time that saves. We could run A/B experiments (with comparable flights) where sometimes we have a baggage handler and sometimes we don’t. That would give you more trustworthy ROI data to help you decide whether or not to fix this long-term, because a manual step is not the most scalable or durable final answer. If the experiment demonstrated meaningful improvement, we could then consider automation, such as a redesigned chute, to orient bags before they enter the carousel.

Maybe the experiment would reveal an 𝐮𝐧𝐢𝐧𝐭𝐞𝐧𝐝𝐞𝐝 𝐜𝐨𝐧𝐬𝐞𝐪𝐮𝐞𝐧𝐜𝐞 of the change. Maybe if the bags are placed on the thin edge it’s harder for passengers to fetch them, so we’ve fixed a problem but introduced a new problem. Sometimes ideas that look good on the surface have side effects you never anticipated. Maybe the entire idea was a bad idea!

But do not begin with the expensive solution. Begin with the cheapest credible experiment.

Engineering Productivity is sometimes misunderstood as “𝐣𝐮𝐬𝐭 𝐛𝐮𝐢𝐥𝐝𝐢𝐧𝐠 𝐝𝐞𝐯𝐞𝐥𝐨𝐩𝐞𝐫 𝐭𝐨𝐨𝐥𝐬.” My real job is improving the performance of the entire system, making it easier for thousands of engineers to build features that delight customers and create business value.

Sometimes improving efficiency really is as simple as turning the suitcase.

If Medium puts this content behind a paywall, you can also view it here (LinkedIn).

On May 22 2026, I spoke at the 19th IEEE International Conference on Software Testing, Verification and Validation (ICST) 2026, Daejeon, Korea on the topic of “𝐀𝐈-𝐍𝐚𝐭𝐢𝐯𝐞 𝐓𝐞𝐬𝐭𝐢𝐧𝐠: 𝐀𝐝𝐚𝐩𝐭𝐢𝐧𝐠 𝐨𝐮𝐫 𝐕𝐚𝐥𝐢𝐝𝐚𝐭𝐢𝐨𝐧 𝐏𝐫𝐚𝐜𝐭𝐢𝐜𝐞𝐬 𝐟𝐨𝐫 𝐀𝐜𝐜𝐞𝐥𝐞𝐫𝐚𝐭𝐞𝐝 𝐃𝐞𝐯𝐞𝐥𝐨𝐩𝐦𝐞𝐧𝐭”

The talk is available here:

Here’s a sanitized transcript with the slides.

==============================

Hi, my name is Carlos. I’m a Senior Principal at Amazon, and I’m going to talk about AI-native development and testing, and particularly testing.

A little bit about me. I’ve been at Amazon since 2009. I did take four years to work at Google, and I came back to Amazon about two and a half ago. So I am a xoogler, and I know a lot about what you guys [Google presenters before me] talked about today as well.

With AI, we’re now able to generate code 10x faster than we could two years ago, and that’s pretty awesome. You just give a prompt and you can get 10,000 lines of code very easily. And everybody’s hyper-focused on how fast can we generate code.

But the real problem, the real bottleneck that I’m concerned about is that 𝐰𝐞 𝐝𝐨 𝐧𝐨𝐭 𝐫𝐞𝐚𝐥𝐥𝐲 𝐡𝐚𝐯𝐞 𝐭𝐡𝐞 𝐚𝐛𝐢𝐥𝐢𝐭𝐲 𝐭𝐨 𝐭𝐞𝐬𝐭 𝐭𝐡𝐚𝐭 𝐜𝐨𝐝𝐞 𝐚𝐭 𝐭𝐡𝐞 𝐬𝐚𝐦𝐞 𝐫𝐚𝐭𝐞 𝐚𝐭 𝐰𝐡𝐢𝐜𝐡 𝐰𝐞 𝐚𝐫𝐞 𝐠𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐧𝐠. Our test infrastructure is not really set up to run a 10x. And most importantly, when that happens, there’s going to be two things in tension, 𝐩𝐫𝐨𝐝𝐮𝐜𝐭 𝐪𝐮𝐚𝐥𝐢𝐭𝐲 and 𝐬𝐩𝐞𝐞𝐝 𝐨𝐟 𝐝𝐞𝐥𝐢𝐯𝐞𝐫𝐲. And guess what? Speed of delivery is always going to win. It’s just what happens.

I think you have sort of this perfect storm where everybody’s focused on the development side of things, and we’re not focused enough on the testing side of things. And we’re also creating this really big problem. So that’s what has been kind of consuming my life for the last few years.

The way that I started doing this is I would go to one team and I would say, “hey, think about this!”. And it kind of felt like going and, realigning their [individual] compass to point a certain direction. Eventually, after doing this many, many times, it kind of felt like herding cats because I got really tired of doing that for every single team. And so I thought, rather than realigning every single team’s compass, can we create some sort of North Star that we can just point at and say: “This is a general direction that we should be heading towards.”

The way that we set out to do this with a bunch of other Principals at Amazon was we were going to paint this destination of where we were going to take the company. Generally, a “destination” [to inspire] is “some beautiful thing.” You’ve got this beautiful scenery, and your destination is this curvy road. And it eventually gets this beautiful village in the valley. And, you know, that’s inspiring and makes people happy. I’ll come back to this image.

Our destination looked like this… You don’t really have to read what’s [specifically] in the slide. Don’t worry. It’s very small, fine print here. But the way that we kind of envision how AI native testing fits in here is kind of this workflow of a bunch of different agents.

I’ll just highlight the parts that are really important here.

The first one is we have the ability to generate specs from code. And we do that pretty aggressively as we have 25 years of legacy code in Amazon. Nobody really understands what some of that code does. But now with LLMs, you can generate much better specs. So we have specs of the present and the current systems. We have specs in terms of what engineers are telling the LLMs they want to build. And so we start with specs. From the specs of what the system does today and the specs of what somebody wants the system to do tomorrow, we can generate some sort of a plan. This is an agent consuming all this data and generating a plan. And then another agent may be critiquing the plan and saying, “I don’t think your design is right, think about this or that.” So there’s sort of a loop there.

Once you have an execution plan, then there’s some sort of agent here that will be a supervisor for the execution of this. So there may be an agent writing code. There may be an agent critiquing the code. And they work out with each other what they think the right code is. There’s an agent writing tests. And there’s potentially other agents, reviewing the tests and so forth. This [what we actually build and converge on] may not necessarily look exactly like this. But the high level bit here is that there’s agents interacting with each other and there’s executing on a plan.

Then there’s a couple escape hatches for real humans to interact. But for the most part, the beautiful thing here is that this is an autonomous system, minus some escape hatches.

Now, 𝐢𝐟 𝐲𝐨𝐮 𝐭𝐫𝐮𝐥𝐲 𝐰𝐚𝐧𝐭 𝐭𝐡𝐢𝐬 𝐭𝐨 𝐛𝐞 𝐚𝐮𝐭𝐨𝐧𝐨𝐦𝐨𝐮𝐬, 𝐭𝐡𝐞𝐧 𝐭𝐞𝐬𝐭𝐢𝐧𝐠 𝐢𝐬 𝐩𝐚𝐫𝐚𝐦𝐨𝐮𝐧𝐭 𝐡𝐞𝐫𝐞. You have to have proper validation for this entire thing to be autonomous. Otherwise, we’re going to be in this situation where we’re just creating slop.

And the high level bit here is that testing is all these little green circles. This is basically all the different places where we can be testing, injecting tests. We can generate testing in the spec; we can be generating testing as we’re reviewing the code; we can test the tests; we can test production metrics. Several places where there’s feedback and data that we can get.

And so this is like the destination that we want to get to. This autonomous development. Remember I told you I was going to come back to that [pretty road]. So this is what it looks like in real life:

There’s a bunch of potholes, there’s a bunch of broken down cars. I don’t know about you guys, but this is what my life looks like. And that is pretty much every company that I’ve ever seen. This is what it looks like in reality once you start diving deeper. Getting to that beautiful valley is going to be a little bit difficult here.

Just to set the same vocabulary here: there’s unit tests, integration tests, and end-to-end tests, and they cover different things.

One of the things that are kind of problematic for us is that traditionally at Amazon we run unit tests in the inner loop, and then we run integration of an end-to-end test in the outer loop, which is quite different from the way Google does it, as they did a much better job shifting testing left. We [Amazon] don’t have a monorepo, so we didn’t “have” to shift left [with the same urgency]. We have tens of thousands of microrepos, so we could be a little bit lazier about that. But as a consequence of that, we have a system where most of our integration end-to-end tests are running in the outer loop. And my assertion here is that if you’re going to have an AI-native systems where LLMs are writing code autonomously, getting feedback here is just too late. My hot take here is that the outer loop is dead. That’s a hot take… you don’t have to agree with it, but that’s what I wanted to somewhat stir the plot here.

The point is we have to figure out how to shift all these signals earlier in the cycle. We have to figure out how to give the LLMs the signals on where the code that they generated is actually working.

I wanted to put a 𝐦𝐞𝐧𝐭𝐚𝐥 𝐟𝐫𝐚𝐦𝐞𝐰𝐨𝐫𝐤 in place to align a bunch of people in the company to move in this direction.

And we came up with 𝐟𝐢𝐯𝐞 𝐝𝐢𝐟𝐟𝐞𝐫𝐞𝐧𝐭 𝐩𝐢𝐥𝐥𝐚𝐫𝐬 to guide our company as we’re thinking about what investments we’ll make in the company, as we’re thinking about how we evolve our toolings. The question that I want to be asking [for everything we do] is “how does it align with these pillars?” I wanted this to be sort of a thought framework for evolution here.

Pillar 1 is Test Coverage. And what I mean here is we have to improve our test coverage. [This side] is beautiful AI slop…it doesn’t actually look like code, but I asked Gemini, sorry, to generate some code. This code will never compile, but it kind of looks like code, so I left it in there. [laughs]

The point is that most of our services have insufficient test coverage, and we know this. And so we can’t get to an AI Native system unless we improve the, line and branch code coverage, and test coverage in general. So it’s one of the directions in which we need to be moving as a company.

This is a place where LLMs actually provide a great opportunity because we have specs, or when we don’t have specs, we have the ability to generate specs. And we can feed these specs to LLMs to auto generate [not just unit but also integration and end-to-end] tests, so we can significantly increase test coverage for everything that we have. So this is a great opportunity. I’m excited about this because I know some of the test coverage that we have in some of our systems, and it’s pretty scary, but we can be fixing this. So that’s one of the pillars that I think we need to evolve to get to this AI native system.

An example of something that we’re doing and something I’ve been [personally] working on is UI testing as it has been pretty difficult for us. There’s a lot of frameworks [e.g. Appium, Selenium, Playright]. UI testing is notoriously flaky just in general. But two years ago, we started investing in an LLM-based system where you just give an LLM instructions: for example “Hey, I want you to go to the Amazon store, search for Harry Potter, make sure Harry Potter and the Sorcerer’s Stone is in the search results added to the cart, make sure it’s in the cart.” I’ve just given you a test in plain English. No Cypress, no Selenium, no Appium, no Playwright, just plain English.

What we found is that we’re able to run much more interesting tests, and there are much less flakiness. We’ve reduced our flakiness significantly by having LLM-based execution of these UI tests. And everything that goes through the Amazon Store starts in the UI, so UI testing is a big deal. That’s an example of a place where we’re using LLMs to increase coverage.

The next pillar is Test Stability.

We’ve talked about stability a lot today [during the conference]. So kind of preaching to the choir here, but I have to explain to a lot of these executives why test stability was a problem, because I needed them to align a bunch of funding and a bunch of investments towards this. Test Stability is the bane of my existence, and the bane of all of your existences, I’m sure. There’s sort of three parts to this. [1] The test code is unstable, and I think it’s pretty low-hanging fruit to fix this. [2] Product Instability, definitely need to be fixing those bugs. [3] But the thing that uses most of my time is this big one here, which is the Test Environment Instability.

The reason this is difficult? This is what Amazon looked like the day I was hired in 2009. Every little dot is a microservice calling other microservices. So creating a test environment for something like this is very difficult. And this was 16 years ago! It looks like much more terrible than this today. We’ve tried to create test environments and there’s two things that we always run into. First is Availability, because we never really maintain our test stacks at the same level of operational rigor as our production stacks. And the second one is Fidelity, because they just simply are not ever going to be an exact copy of Prod. At the end of the day, they’re just copies. This concept of creating test environments and hoping that they will be “just like Prod” is something I’ve been trying for 20 years, and I don’t think it’s going to ever work.

So what we’re moving towards is actually testing in Prod [I’m leading this initiative]. And I don’t mean that in like, you know, a YOLO kind of way, but: how can we do this safely? This is not just my idea. There’s great whitepapers from Uber that started doing this a number of years ago that got me really excited about this. The idea is: Production is a high fidelity, high availability environment. If we can test safely in Production, that’s great for these very complicated, large ecosystems, right?

And so we hit Production, we mark the transaction as being a test transaction. Then here we have routers that proxy test traffic to the test environment, the SUT, the Service Under Test. But other than that, everything else is Production. It’s just rerouting a specific part of the graph to test. The other part of this is: being in the Amazon Store, you always have unintended consequences when you’re doing something. I don’t want to charge a customer credit card if I’m replacing some traffic. So we have to avoid all these like unintended consequences to downstream services. That’s the other part of this is that we have to have ways of modifying certain behaviors like charging credit cards, shipping items and things like that. And the last but not least is, you have to think about how you’re writing data. Because now you have test transactions going through production services. Are they going to be writing [polluting] into production databases? How do you keep those things safe? Non-trivial, lots of work… but this is a direction that we’re moving. And I’m very excited about this and how to make it safe.

Pillar number 3 is Test Adequacy. What I mean by this is: the traditional way of writing things that we’ve been doing forever is: we have a full regression test suite and we run the same tests for every single change, regardless of whether the change is high risk or low risk. We need to be a little bit smarter about test case selection. Google and Meta have been doing this for a while. I think we’re catching up with a lot of this. We need to be analyzing the risk of a particular change and tailoring the tests that we run for that specific change. That’s one place where we run more adequate tests for that particular change.

The other one is an extension of mutation testing. [The idea of] mutation testing is that you introduce synthetic bugs in the code, and you rerun your test. If your tests don’t catch this bug that you’ve just introduced, then your tests are bad. We’re taking this to the next degree. We’re changing code and we’re introducing all kinds of synthetic bugs, and then we’re seeing how many of those bugs are actually caught by any of our validations. This gets into the business of testing the tests, which is something that I’ve been increasingly worried about as we have LLM automatically generating tests, we have to test those tests.

And ideally, we have some sort of metric for every one of the validation pipeline says, “Hey, your current test strategy is letting 20% of these synthetic bugs that we’re trying to introduce go right through!”

[Laughs] In this slide, I love that Gemini named this thing “Bug Injector 3000” That’s pretty cool. I think I’m going to name it that, like when we put this in production, that’s going to be the name.

The next pillar is Test Efficiency. What I mean by this is there’s a lot of our test infrastructure that isn’t necessarily set up to be running at 10x.

I’ll give you a concrete example of something I worked on, which is mobile testing is a big deal to the Amazon Store. I’m a dinosaur, I buy on Amazon on the web. I think I’m the only person left in the world that uses the website to buy from Amazon. Most of our customers are buying from our mobile app. Yet, two and a half years ago when I came back to Amazon, most of our testing was on the web. So I thought, “this is a disaster waiting to happen: we’re not really testing the way that our customers are using us!”

So we had to really double down on properly testing on mobile devices. We had these labs of 30,000 mobile devices and a bunch of lab engineers trying to maintain these devices. And they were absolutely painful to do. And I thought, this is NOT going to scale. As we depend more on LMS and increase the rate of which we’re generating code and invalidating code, this isn’t going to cut it. And it’s not my expertise to manage a lab.

So, Phase Two about two years ago [AWS Device Farm]. AWS does a really good job here. And I’m lazy. If they do a better job, that’s great. AWS Device Farm gives you physical devices, but they manage all that infrastructure. So that’s great. That got us out of the business of having physical labs that we had to take care of.

I still wasn’t going to scale 10x because I can’t just get my hands on 10,000 Pixel phones if I want to. So we started augmenting that with virtual device testing, running emulators and simulators. That’s easy: we just run it on, 10,000 containers, and there’s the 10,000 devices. [That was Phase 3]

This is an example of a place where the way that we had our traditional test infrastructure was just never going to scale. And once you start looking more carefully, there’s all kinds of bottlenecks like that. We just never anticipated that we’d be generating code and validating it at 10x.

And the last pillar is a little bit fuzzy here, but I call it the non-determinism. And what I mean by this is most of the testing that we traditionally do is pretty binary: I run a unit test, or I run an integration test. It either passes or it doesn’t. But now we are generating more and more agentic experiences where things aren’t necessarily clear cut. Deciding whether an agentic change was adequate or correct is much more difficult. You have to understand what does. For example, Rufus if you are using, Rufus is the chatbot in the Amazon store. That’s an example of an agentic experience that we can’t just test with traditional testing. Understanding how to test all these non-deterministic experiences.

And by the way, since we’re generating code at LLMs, like every single time that we’re generating that code, we’re potentially getting an entirely new code base. So that’s also non-deterministic! Surprise! everything is non-deterministic now, not just the things that you naturally think of non-deterministic.

And so we have a plan for how to get from there to here. I think it’s going to take a lot of years. But one of the nice things about articulating this mental framework is that it gave us a way for all of us to communicate in the same way. It gave us a way for us to understand the urgency of needing to move really, really fast. And it gave us a way to align.

We’ve been presenting this to a bunch of the executives. And it’s been successful in the sense that, for example, one of the teams was investing on something. And after we chatted about all this, they’re like,”Great…how do we double the funding on this? How do we move faster on all these things?” Never understand the power of creating a mental framework when you’re trying to move a very, very big ship.

I just want to leave you with that. Now, it’s getting a little late and we’re all getting tired, but I wanted to also have time for questions and debate and ideas as well. Thank you!

I’m going to ask you my favorite part since we’re here with the academics. Let’s talk about the mutation testing. How are you thinking about mutation testing? I think there was actually a good paper here earlier in the week where they were looking at the different kinds of faults, between an LLM mutation versus, I can’t remember which paper was. Versus the traditional mutation tools, which I use like templated mutation operators. How are you thinking about this? Are you trying to be like adversarial with mutation testing? What kind of bugs are you trying to find?

So this is when we just started playing with it. And very good point. I think the traditional mutation testing is things like changing a < sign to a > or something like that. The kinds of bugs that we’re trying to do is they’re a little bit more subtle than that. Because I think the kinds of bugs that LLM end up creating are more subtle than that. One of the things that we realized, for example, is that we have an LLM that critiques code. And unless you’re a little bit sneaky and adversarial about bugs that you’re introducing, the LLM that critiques code will catch it. Particularly if you have certain phrasing. So we have to try and trick the LLM that’s code reviewing. This is all a big experiment; I don’t have any data yet. But the way that we’re going about some of these injections is we’re actually not giving the LLM access to the code initially, but we’re giving it access to the specs. So it thinks in terms of user scenarios more than just like, oh, I’m going to change this greater than sign to less than sign.

You should look at the proceedings of the mutation workshop that was on Monday. There was a couple of different things that were on higher order mutation where they were extracting either pseudocode or a plan or a specification. And then there was a very nice one that used hazard analysis, which was something that I didn’t know about. But they used this hazard idea and to make structured mutations of the specs. And they found that that was much better than the unguided mutation for you just to ask it to make some change. It gave you like sort of like a template of changes, but for specifications.

I think I want data because my gut feeling is that most of the validation is pretty terrible. And it’s letting a bunch of defects through. But unless we have data that we can want a specific pipeline and say, hey, your your defect scape rate is 18 percent or your defect scape is 80 percent, then we can have a better discussion about that.

Yeah, I was wondering if you can talk a little bit more about the specs, like how widely deployed in the spectrum and development at Amazon and what what form do the specs take?

We are very sold on that. So there’s something called Amazon Kiro, which is an IDE that up to recently was the only IDE that we were using. We’re now also using Anthropic Claude. But up to recently, we’re all like 100 percent using Kiro. Now, the interesting thing about Kiro is that basically it has two modes. You do vibe coding like anybody else or you do spec driven development. And most of us were really going for creating the specs, checking in the specs to the point of sometimes not even looking at the code, just just focusing on the specs themselves. We also have millions and millions, millions of lines of code legacy systems. And so we also invested in a system that creates specs from legacy code. It’s called Spec Studio. And there’s a bunch of [external] talks on that as well. It is great. I had to go change a piece of code in that package that had 300,000 lines of code. And I thought, I don’t have time to look at 300,000 lines of code and understand what the hell this thing does. But I do need to make this change. So I was able to generate a spec for that in one minute and then understand where there I need to be making that change. And so we’re investing in basically creating specs for every single service that we have in the entire company.

And these are just natural language, right?

Yes, all of them.

Are you establishing a structured format for storing them? Or are you just taking the markdown files?

That’s a good question. Kiro uses a very specific format that it generates. And so that’s the good news. Basically, it creates three files. It’s a requirements doc first and starts with requirements. And then from that requirements doc, it creates a design doc (as a markdown file). And then from the design doc, it creates a list of tasks for actually executing the thing. And so that’s pretty standard across the company. Kiro enforces a lot of those standards.

Yeah, sometimes like I think many of the tools, the harnesses, like they’ll create those files in the background, but they’re ephemeral. And I think there’s like an industry wide management problem of how to manage the evolution of those and what to do with them and all these kinds of things. It’s like, yeah, that, that, that, that spider’s nest you showed, the whole spider’s nest, it’s like that.

When people say the word spec, they actually mean different things. So I may have a spec on what I want the future to be. This is a prompt, a spec on a feature that I’m trying to add. That’s a future spec, right? And then I have specs for like what the world does today. And the specs for the feature I’m trying to add, those are somewhat ephemeral in that once I add that feature, it becomes part of this as well. So where do we keep those? How do we treat those? And it’s something we haven’t 100% figured out yet.

I know Amazon is like actually really good at formal specifications. And the AWS Automated Reasoning guys got to play TLA plus. And all those people, but like the natural language specs don’t necessarily compose with that. And how do you update it? And how do you check a natural language spec to make sure it’s self-consistent and all these kinds of things.

There’s all kinds of really interesting one papers coming from the Amazon Kiro and Automated Reasoning guys on exactly that. Cause they are fully spec driven development. And so enforcing a format for how do they think about testing the specs and so forth. And this is a big loop, right? When code is checked in, all these specs have to be regenerated because they’ve changed the state of the world. So these are outdated. And so they’re sort of a regenerated loop all the time.

So for teams looking to be more AI friendly in a testing space with test agent, test judge, all these things, what lessons do you think you can share about entering that world a little bit carefully?

I think the thing that we’ve learned is that you have to think differently about your processes and you have to be willing to kind of blow them out whenever they don’t make any sense. And what I mean by that is like, we have this, this horse cart and we got an LLM. So we just added like a 500 horsepower engine to the horse cart. And then we’re like, Oh shit, this doesn’t drive very well. So we try to make the wheels better. And we’re like, Oh, this doesn’t break very well. So we just added brakes. And so like, at what point does it make sense to just scrape that and go create the Ferrari? So these are active debates that we’re having is how much do we evolve systems? How much do we say, well, the SDLC is going to look entirely different. So like, even like the pillars, we have very hot debates about how much we want to invest in some of these things. For me, for example, investing in fixing test stability, that’s durable. No matter which way LLMs are going to end up, we will need stable test environments. So looking at durable investments is a hard one because I don’t 100% know which thing is going to be durable. This is my view [currently], but could never entirely wrong. Two years from now, we might be having an entirely different discussion about like how agents should talk to each other. Anybody that claims they understand AI and how the SDLC will look like two years from now, they’re full of crap. They have to be willing to go very quickly.

Is your hunch that that it’s adversarial? Like you have a spec dev agent. So the judge is like, is the best thing just the adversarial against every single component, right? So you have literally the the test agent, absolutely adversarial against whatever the hell the dev agent is doing.

Kind of like the old SDET role, right? In fact, this is kind of like the way things used to be, right? So Microsoft hired SDETs back in the 90s. I started my career as an SDET. And the reason they hired SDETs is because they wanted a different persona that was independent from the developers creating the tests. And they had different incentives and so forth. And really [with different AI agents] we just recreated SDEs and SDETs. We’re just recreating the things that we used to do in our old SDLC, but like adopting it to agents.

I also found it fascinating your whole testing in production slide. I was wondering if you could talk more about what’s happening with that. Is that ideation phase right now? Are you trying it in some parts of the company?

Yeah. I made a lot of people really nervous when I put that idea forward. I had to take it to a bunch of the Distinguished Engineers in the company, get them to buy into it. Because a few were like, “are you crazy?” I said, “yeah, I am! But we’ve tried to fix this for 20 years with the same way of fixing it, which is, oh, we’re going to make a copy of prod. And we promise that this time we’re going to get it right. All these efforts always start with good intentions and they agree for six months or so. Eventually the person that started the effort of making a reliable test environment leaves the company.” I think that safety is the part that’s most important. [Last year] we pitched the idea, we got buy-in, we got funding, we hired a team, about 30 engineers in Madrid. We have multiple Principal Engineers working on the problem. And we have it working for a subset of Amazon Stores. And we have a goal, VP level goal, to get this working by end of year. So, we’re very serious about this. Honestly, 90% of the effort to get this working is the safety guardrails; the actual tech is pretty simple. It’s a router to route traffic. Yeah, there’s some challenges to doing that at scale. And there’s another router. And there’s another router to write data safely. But, most of our efforts is in how to make it safe.

I’m looking at the Meta people, but they have this, like, digital twin system, the QA environment, right? And that sort of, like, simulates all the Facebook activity instead of being, like, a bunch of little integration environments. So, was that what you guys were trying to do? Or are you trying to do more, like, a bunch of little ones for, like, breaking part of that giant service mesh?

No, one big giant one. So, we’re able to run unit tests and that’s fine. And, we can deploy a system and mock dependencies and we can run integration tests. That’s fine. But at the end of the day, the most interesting bugs that show up are the bugs, like, you’re somewhere in the middle of that ball of services. And you don’t really know when you’re making a change and you don’t really know if you’re going to break anybody or not because there’s hundreds of microservices above you and hundreds of microservices below you, right? So, that’s the thing that’s most interesting to me is how do we get this end-to-end experience? And how do we improve things? Because if LLMs screw up, that’s where they’re going to screw up because they don’t have enough context on the whole world. They’re making things up. They’re guessing. I’m not worried about LLMs writing code that will fail a unit test. That’s a mostly solved problem. I’m more worried about them creating tests that are going to have subtle problems with a misunderstanding of the way the ecosystem works and how services interact with each other. So, that’s what I’m worried about.

In one of your slides, you showed that you just defined prompt as the test. It wasn’t clear whether you’re using prompts to just create a test and then log this to this test as a test, or is the prompt a test?

Oh, okay. This is the test auto generation. We’re debating this one as well. Some people are in the camp that the prompt is the test. Actually, the extreme view here is that the requirements doc itself is the test. Do we even need to create tests? Or do we just give the LLM the requirements doc and say, figure this out? It’s a bit of be pie-in-the-sky, but I don’t think we’re actually that far from it in real life. There’s a lot of places in Testing that we’ve been moving away from generating code. For example, UI testing, we specifically are moving away from generating code. We don’t generate code in Playright. We feed natural language to an LLM and let the LLM navigate the Store. There’s so much more interesting test cases that the LLMs run into. In fact, the very first time we run this, we told the LLM “log in with its username and password” And the username and password was not right. So it failed to log in. So the LLM is like, “Cool I can see there’s a create account button here. So I’m going to create an account.” And it did! And that failed. And then the LLM being creative and being task-driven was like, “oh, cool, I see that there’s a Chat With Agent help button here.” And it started chatting with an agent. And at that point, we’re like, “well this is cool…but we really need some guardrails around how we’re going to have LLMs navigate the store and run tests!” That’s an entirely different talk…

Over my three decade career spanning Microsoft, Amazon, and Google, I have worked on nearly every dimension of Engineering Productivity at scale: developer tools that engineers use to build, review, test and deploy their code, CI/CD infra, integration testing, load, stress, performance and chaos testing, canary monitoring, profiling, code coverage, bug management, and the cultural challenges of getting tens of thousands of engineers to actually use the things you build. The problems I have faced span technical, organizational, and human: the AI revolution, why creating large test environments is hard, how monorepo versus microrepo shapes everything, and why you must go beyond software engineering and understand psychology and behavior change. This collection of articles I’ve written in the last few years is an attempt to share what I have learned, honestly and practically, from someone who has been in the trenches long enough to have both caused a multimillion dollar outage and helped prevent the next one.

The Outer Loop is Dead: Rethinking how we test (2026)

Right now (2026) I am focused on Amazon’s transition to an AI-native development world for tens of thousands of engineers. Specifically, I am driving the evolution of testing techniques to support autonomous validation of AI-generated code at scale

https://www.linkedin.com/pulse/outer-loop-dead-rethinking-how-wetest-carlos-arguelles-8lwac

Abstract: AI-accelerated code generation has exposed a fatal flaw in the traditional software development lifecycle: the “outer loop” (the CI/CD cycle where integration and end-to-end tests run post-commit) is too slow, too flaky, and too late to keep pace with machine-speed development. The solution is mandatory shift-left testing, moving integration and E2E signals into the developer’s inner loop, but doing so requires fixing four foundational problems first: unstable test environments, tests that don’t validate the right things, inefficient test infra that can’t scale to an order of magnitude increase in code volume, and an evaluation mindset still anchored to pass/fail rather than assessing whether AI-generated changes actually achieve what the builder intended.

Why I’m Betting on LLMs for UI Testing (2025)

This was a more tactical win for 2025; evolving UI testing infra (business-critical to the Store) to be AI-native.

https://www.linkedin.com/pulse/why-im-betting-llms-ui-testing-carlos-arguelles-nwptc

Abstract: UI testing has been brittle and expensive for thirty years, and LLMs may finally change that. The article explains how they are well suited for test generation and execution and makes the case for expressing end-to-end tests in natural language and using an GenAI as the execution engine, reducing authoring cost, eliminating framework-specific maintenance, and handling two root causes of flakiness (unexpected dialogs and DOM volatility) through contextual reasoning. An Amazon Store demo illustrates both the power and the failure modes, with proposed mitigations including execution budgets and judge LLMs for step validation. Looking ahead, persona-driven execution unlocks coverage that traditional automation structurally cannot provide. Latency and cost are real today but improving; the savings in human authoring and maintenance already justify the bet.

Amazon’s journey with mobile testing infrastructure (2024)

This was another tactical win in 2024; evolving mobile testing infra (most of our customers use the Store on mobile yet most of our testing was on web so we weren’t testing our product the way our customers were using us)

https://www.linkedin.com/pulse/amazons-journey-mobile-testing-infrastructure-carlos-arguelles-0dzjc

Abstract: This article traces Amazon’s journey through three phases of mobile test infrastructure: from on-campus physical device labs, to managed physical device testing, to the emerging need for large-scale virtual device testing via emulators and simulators. Two converging forces are driving the next inflection point: shift-left testing and AI dramatically lowering the cost to create and maintain UI tests, together projecting a 100x increase in test volume. To meet this demand, Amazon is augmenting physical device testing with virtual device testing running on containers, balancing the scalability, reliability, cost, and speed advantages of virtual devices against the fidelity advantages of physical ones. The proposed strategy is not a replacement but a layered approach: virtual devices for the inner developer loop and early CI/CD stages, physical devices for business-critical and hardware-dependent validation.

How Amazon and Google view CI/CD in an entirely different way (2024)

I worked across multiple dimensions of Engineering Productivity at both Google and Amazon, and one of the things I found most fascinating was how seemingly small decisions had massive downstream impact on the entire way these companies develop, test, and release software.

https://www.linkedin.com/pulse/how-amazon-google-view-cicd-entirely-different-way-carlos-arguelles-jn7uc

Abstract: Google and Amazon represent two fundamentally different philosophies on CI/CD, and the root cause traces to a single architectural decision: monorepo versus microrepos. Google’s monorepo (shared by over 120,000 engineers with no branches) makes the blast radius of a bad checkin catastrophic, forcing massive investment in pre-submit validation through hermetic, ephemeral test environments, smart test selection, and deflaking infrastructure. Amazon’s microrepos contain blast radius by design, making pre-submit testing a nice-to-have rather than a must-have, but requiring best-in-class post-submit pipelines that shepherd each change to production within hours. Neither approach is objectively superior: both carry comparable total complexity, just distributed differently. The real lesson is that pre-submit and post-submit excellence should not be an either/or tradeoff: both are worth deliberate investment, and at the scale of tens of thousands of engineers, the leverage on reduced developer toil more than justifies the work.

How Google uses hermetic, ephemeral test environments to reduce flakiness (2023)

At Google, I was the Tech Lead for Dev Infra for Integration Testing, responsible for how Google engineers ship high-quality software at scale using a company-wide unified ecosystem for defining test environments with standardized integration testing infrastructure covering Functional, Performance, Diff Testing, and Development Environments

https://www.linkedin.com/pulse/shifting-test-left-part-2-how-google-uses-hermetic-reduce-arguelles-k7kuc

Abstract: Integration testing at scale surfaces flakiness from shared state, unreliable dependencies, and network instability. Google addressed these root causes by investing in hermetic, ephemeral Systems Under Test (SUTs), environments spawned on demand for each test run, containing the system and its dependencies in a single isolated container. Ephemeral SUTs eliminate state pollution and concurrency conflicts between test runs; hermetic SUTs remove inter-system network calls to unstable shared dependencies. Together, they improve reproducibility, debugability, and test signal trustworthiness. However, this approach introduces its own challenges: slower test startup times and dependency graphs too large to fit in a single container.

Shifting Test Left: When sharks chew on network cables (2022)

Creating test environments for complex systems is non-trivial. Some reflections on the sources of friction, from my time helping others do this at Amazon and Google.

https://www.linkedin.com/pulse/when-sharks-chew-network-cables-carlos-arguelles-zvumc

Abstract: Integration test flakiness is not primarily a code quality problem; it is an infrastructure problem, and a harder one than most engineers expect. Flakiness sources fall into three buckets: flaky test code, flaky production code, and everything else. That third bucket is far larger than intuition suggests. The core villain is the distributed nature of integration testing itself: unlike unit tests, which run in a single process with minimal moving parts, integration tests deploy a System Under Test across machines, wire it to transitive dependency graphs whose combined reliability compounds multiplicatively, and communicate over a network that is neither reliable nor zero-latency. Three additional failure dimensions pile on top: unstable dependency test environments, shared mutable state that corrupts across test runs, and concurrency conflicts when multiple suites execute against the same long-lived environment. There is no silver bullet, but hermetic, ephemeral test environments directly address the worst of these failure modes, at the cost of their own infrastructure complexity.

How simple telemetry helped me understand complex human behavior (2021)

One of my pet peeves as an owner of company-wide infra is onboarding effort. As an engineer, I want tools that I can adopt quickly and easily. So I obsess with what the onboarding experience is for every piece of software that I own.

https://www.linkedin.com/pulse/how-telemetry-helped-me-understand-complex-carlos-arguelles-wtm0c

Abstract: When your product is a loosely coupled suite of dozens of tools that has evolved organically over two decades, understanding how thousands of engineers actually navigate it is not straightforward. The challenge at Google was measuring onboarding time for an integration testing suite with no unified telemetry, no agreed definition of “onboarding,” and customer journeys that varied wildly across entry points. The solution was a lightweight clickstream approach borrowed from Amazon’s retail analytics: instrument roughly 15 disparate surfaces to emit timestamped customer events, then stitch those events into individual journeys to identify where engineers got stuck, slowed down, or abandoned the product entirely. Imperfect data gathered pragmatically proved far more valuable than waiting for a perfect measurement system. The resulting journeys surfaced concrete bottlenecks, made the business case for investing in onboarding, and established a repeatable baseline for measuring whether improvements actually moved the needle.

Deduplicating software bugs with Machine Learning at Google (2020)

Back at Google, we were swimming in bugs auto-cut from test suites that were running as part of CI/CD. We realized that a lot of these bugs shared simple root causes and started an experiment: could we use machine learning to deduplicate them?

https://www.linkedin.com/pulse/deduplicating-software-bugs-machine-learning-google-carlos-arguelles-75gyc

Abstract: At Google scale, duplicate bugs are not a minor annoyance; they are a multimillion dollar productivity drain. Bug auto-filers reduce toil by automatically logging failures, but when a single root cause triggers hundreds of test failures, each gets its own bug, and the human cost of investigating them compounds rapidly across 100,000 engineers. The solution was machine learning: a binary classification model that compares bug pairs across dimensions like title similarity, stack trace overlap, filing time, and metadata richness, achieving greater than 90% precision. The harder design decisions were not technical but product-shaped: whether to target automated or manually filed bugs, how many teams to onboard and in what sequence, and most consequentially, whether the system should take action or merely suggest it. The team resolved the last question with a configurable risk dial, letting each team choose their own tradeoff between toil reduction and the probability of a legitimate issue escaping to production.

The Great Code Coverage Holy Wars of the 21st Century (2019–2020)

Having been in EngProd for 30 years, I often see Code-Coverage-Good-or-Bad heated debates spark randomly, so I worked with two other Senior Staff / Principal Engineers at Google to articulate our shared vision for where code coverage metrics are useful and where they’re not.

https://www.linkedin.com/pulse/great-code-coverage-holy-wars-21st-century-carlos-arguelles-mvvbc

Abstract: Code coverage is one of the most reliably polarizing topics in software engineering, capable of igniting holy wars in any large engineering organization regardless of whether the opening salvo is pro or against. The reality is that both camps have legitimate points, and the goal should be finding pragmatic common ground rather than winning the argument. Coverage is a lossy, indirect metric that compresses a lot of signal into a single number; it does not guarantee test quality, and chasing 100% breeds false confidence and low-value tests. But low coverage does guarantee that large areas of code go completely untested on every deployment, which is a concrete, actionable risk. The most useful framing is to treat coverage not as a target to hit but as a gap detector: what is not covered matters more than what is. Embedding coverage data directly into the code review process, applying per-commit thresholds rather than project-wide mandates, and reserving human judgment for the specific uncovered lines that actually carry business risk yields far more value than any top-down percentage target. Culture changes driven by coverage discipline tend to improve code quality broadly, well beyond what the metric itself directly measures.

A Tale of Elephants: How I grew a tool to impact thousands of engineers at Amazon (2014–2018)

Back in 2012, I was responsible for a multimillion dollar outage on amazon. This was a traumatic event, but it set me on an entirely different path in life: creating infra for load and performance testing, and educating thousands of amazonians around me on the value and importance of load, performance, stress and chaos testing.

https://www.linkedin.com/pulse/tale-elephants-how-i-grew-tool-impact-thousands-amazon-arguelles-pnric

Abstract: Growing a developer tool from a weekend project to a platform used by tens of thousands of engineers is not primarily a technical problem; it is a human one. This article uses the Switch framework from Chip and Dan Heath to retrospectively explain how an internal load and performance testing platform at Amazon achieved exponential adoption after years of modest linear growth. The framework organizes influence into three dimensions: directing the rational mind by finding bright spots, scripting the critical moves, and pointing to a compelling destination; motivating the emotional mind by finding the feeling, shrinking the change to make adoption feel achievable, and growing people’s identity around the desired behavior; and shaping the path by tweaking the environment, building habits through CI/CD integration, and rallying a community that transformed the tool from something one person built into something the organization collectively owned. The throughline is that technical excellence is necessary but nowhere near sufficient; understanding psychology, influence, and behavior change is what separates a tool that gets used from one that quietly dies.

When the canary stops singing… (2016)

This is the story of how I ended up working on Canary Testing infra at Amazon back in 2016, simply out of a crazy idea during a meeting once.

https://www.linkedin.com/pulse/when-canary-stopssinging-carlos-arguelles-cif2c/

Abstract: A two-hour AWS ops meeting with a Senior VP turned into an unexpected architectural adventure. His demand for better canary monitoring across all critical AWS business scenarios exposed a tooling gap, and an impulsive hand-raise in the final minutes of that meeting committed a single engineer (me!) to fixing it. The insight was that the load and performance testing platform already in use across thousands of Amazon services was, at the right level of abstraction, a canary execution engine too: both required bootstrapping test code, running it at a desired rate across multiple networks, and emitting metrics for dashboards and alarms. What looked like a one-hour proof of concept turned into months of real engineering, because the architectural tradeoffs baked into the load testing platform, optimizing for scalability over precision and for cattle over pets, were precisely the wrong tradeoffs for canaries. The result worked, served thousands of services, and opened doors to senior leadership visibility that shaped a career. The broader lesson is that pragmatism and serendipity sometimes produce better outcomes than deliberate planning, but only if you are willing to do the hard work after you raise your hand.

Bunnies and Bees (1998–2001)

An old story from my time at Microsoft, and how we had to use ingenuity to test spellers and grammar checkers for MS Office back in the late nineties!

https://www.linkedin.com/pulse/bunnies-bees-inventing-microsoft-back-nineties-carlos-arguelles-h9cqc

Abstract: In 1998, shipping spellers and grammar checkers for 40 languages in Microsoft Office required two things that did not yet exist at the necessary scale: enough text in each language to test against, and enough computing power to run it all. Two weekend-born inventions solved both problems. The first, nicknamed Bunny, was a self-feeding web crawler that bootstrapped from a curated seed corpus, extracted text from search results, and continuously refreshed its dataset to avoid overfitting, scaling from 10,000 sentences to hundreds of thousands per hour across dozens of machines. The second, nicknamed Bee, was a distributed computing daemon that commandeered idle developer workstations overnight, multiplying available computing capacity eightfold without buying a single additional machine. Together they took a testing operation that processed 10,000 sentences per day in one language and scaled it to one million sentences per day across 40 languages. Both ideas read as obvious today, which is precisely the point: the best inventions age into common sense, and the human stories behind them are worth telling before that happens.

If you’ve built software in a distributed system, you know the rhythm: write code, run unit tests locally, put changes up for code review, eventually merge, wait for CI/CD tests to run, get a failure an hour, either rollback your changes or if you’re feeling lucky just re-run the tests, spend an hour debugging whether it’s your fault or a flaky test, fix it, push again, wait again. Repeat.

That rhythm has defined our Software Development Life Cycle for decades. We’ve optimized around it, built tooling for it, and accepted it as the cost of doing business.

𝐀𝐈 𝐡𝐚𝐬 𝐜𝐡𝐚𝐧𝐠𝐞𝐝 𝐭𝐡𝐞 𝐞𝐪𝐮𝐚𝐭𝐢𝐨𝐧. It is now generating code at a pace that makes the toil from this rhythm unsustainable. The part that’s bottlenecking isn’t the code generation, it’s validating the code.

To get to AI autonomous development, we need to trust the output from the AI. To do that, we need the test signals available earlier. To do that, we need to shift testing left. But before we do that, we need to fix the sloppiness and toil we’ve learned to tolerate with our current practices (flaky tests, poor test coverage, slow feedback loops). That’s critical path.

The idea of 𝐬𝐡𝐢𝐟𝐭 𝐭𝐞𝐬𝐭 𝐥𝐞𝐟𝐭 is not novel, but there has been a significant emphasis on it recently, becoming a rallying cry from Engineering Productivity thought leaders. A quick search on Google Trends corroborates it: the following graph shows the popularity of web searches for the term, 2016–2025 (a value of 100 is the peak popularity for the term):

The Three Types of Tests

To understand where we are, we need some context. Software testing happens at three levels, each serving a distinct purpose.

• Unit tests validate individual components in isolation: a single function, a method, a class. Dependencies are generally replaced with mocks, so the test exercises only the code under test. Unit tests run in milliseconds, require no infrastructure, and when one fails, it points directly at the broken component. They’re fast, cheap, and precise.
• Integration tests validate the interactions between components. They exercise real dependencies within a controlled environment: actual databases, actual queues, actual service calls. They’re slower (seconds to minutes), require test infrastructure, and their failures are less precise. But they catch an entire class of defects that unit tests cannot: interface mismatches, contract violations, data serialization errors. The bugs that live in the seams between components.
• End-to-end (E2E) tests validate complete user workflows across the full system stack, from the UI through backend services to data persistence. They exercise the system the way a customer experiences it. Their value is both comprehensive coverage and realistic validation like your customers use your product. For example, in the case of the Amazon Store, it means using the Shopping mobile app’s UI to walk through like a real customer would. They’re also the most expensive: they require a full ecosystem deployment, take minutes to hours to run, and produce ambiguous failures that are notoriously hard to debug.

The Inner and Outer Loops

The pros and cons of these three types of tests led us to adopt the concept of a 𝐭𝐞𝐬𝐭 𝐩𝐲𝐫𝐚𝐦𝐢𝐝: wide at the bottom with unit tests, narrow at the top with E2E tests.

The shape is driven by 4 trade-offs:

• feedback speed: sub-second for unit tests, minutes to hours for E2E
• failure precision: unit test failure pinpoints the broken function; E2E failure could be any of fifty services
• maintenance cost: unit tests break only when their component changes; E2E tests break when anything in the workflow changes, or often since they’re in a distributed system, because of infrastructure problems
• environmental complexity: unit tests need zero infrastructure; E2E tests need a full production-like deployment

These trade-offs don’t just determine how many tests you write at each level. They determine when and where those tests run.

• The inner loop is the developer’s local machine: write code, run tests, read results, refine. This loop needs to complete in seconds to keep you in flow state, so it can only accommodate unit tests and lightweight integration tests.
• The outer loop is everything after code leaves your machine: CI/CD, integration test suites running against shared environments, E2E test execution, pre-production verification. This loop takes minutes to hours. It’s where the heavier integration tests and all E2E tests live, because they’re too slow and too infrastructure-dependent for the inner loop.

But with the inner/outer split, important signals arrive late. The outer loop validates how your service interacts with other services. Whether your API change breaks a downstream consumer. Whether data flowing through three services and two queues arrives in the right shape. These are the interactions where the most catastrophic bugs live, not in isolated components, but in the spaces between them. You don’t find out about these defects until your code has been committed, deployed to a test environment, and run through a CI/CD cycle that takes possibly hours. By then, you’ve moved on. You’ve lost context. Worse, you now need to rollback, and other engineers who have already started building on top of your changes need to rollback as well. The blast radius of a single late-caught bug compounds across the team.

Why AI Breaks the Outer Loop Completely

AI coding assistants and code-generation agents have dramatically accelerated the inner loop. Engineers generate code at an order of magnitude faster speed.

But the outer loop didn’t get faster or better. CI/CD still takes hours. The tests are still flaky. The only thing that changed is that you’re now pushing code through CI/CD far more frequently. More changes per day means more CI/CD runs against the same fragile infrastructure, more flaky failures, more retries, more erosion of trust.

The problem goes deeper than throughput. AI-generated code often works in isolation but fails at the seams. The code is based on the context the AI has been given, typically a single file, a single service, a single component. It doesn’t have visibility into the broader ecosystem: the downstream services that consume your API, the implicit contracts between services, the production configuration that differs from the test environment. It passes unit tests, because they validate components in isolation, which is exactly the scope the model had when generating the code, but these failures show up in integration.

The outer loop was designed for a world where humans wrote code slowly enough that a long CI/CD cycle was an acceptable feedback loop. In a world where AI generates code at machine speed, that cycle is a bottleneck that negates the entire value proposition.

𝐓𝐡𝐞 𝐨𝐮𝐭𝐞𝐫 𝐥𝐨𝐨𝐩, 𝐚𝐬 𝐰𝐞 𝐤𝐧𝐨𝐰 𝐢𝐭, 𝐢𝐬 𝐝𝐞𝐚𝐝.

[Side note — instead of moving tests to the inner loop, why not keep them in the outer loop but fix the test friction, and then give AI agents access to check in code, merge, and monitor CI/CD on their own? In theory, the latency of the outer loop matters less if hundreds of tireless AI agents are working 24x7. But this breaks down at scale: the shared repository itself becomes a contention point. Agents will constantly step on each other’s changes, triggering cascading rollbacks, and competing for CI/CD resources. You’ve replaced human toil with machine thrashing. The outer loop doesn’t just need to be faster or more autonomous. It needs to stop being the place where critical validation happens for the first time.]

If the outer loop is dead, what replaces it? The answer is simple: shift test left is now not just nice-to-have, it’s mandatory. Move integration and E2E tests into the inner loop so builders (both human and AI) get those critical signals while they are working on the code. But here’s the catch: if you naively move today’s outer loop tests into the inner loop, you haven’t solved anything. You’ve just made the inner loop as slow, flaky, and unreliable as the outer loop is today. Before you can move them, you have to fundamentally fix them. That’s what the pillars are about.

Pillar 1: Test infrastructure needs to run reliably

Test environments are never treated with the same operational rigor as prod so they’re often down, lack fidelity, and suffer resource contention from dozens of teams running tests simultaneously. When a test fails, the first question is always “is this a real failure or an environment issue?” I’ve written extensively about this problem and the toll it takes on engineers.

Different companies have resolved or mitigated this in different ways. Google, for example, went the route of hermetic, ephemeral test environments: fully isolated, short-lived environments spun up per test run, sometimes combining test doubles (fakes, mocks or stubs). I was a Tech Lead of that team for years. Uber took a different approach, safely injecting test traffic into their production graph, which gave them the availability and fidelity of production without the risk of affecting real customers. Both approaches solve the same core problem: making test environments stable.

You might think what’s the big deal about creating a test environment for my service? This is what amazon.com looked like the day I was hired, ~17 years ago (every dot is some sort of service), and it hasn’t gotten any simpler to serve traffic since:

Creating a stable, high-availability, high-fidelity end-to-end test environment for this massive graph is nearly impossible.

Pillar 2: Tests need to validate the right things

Tests that validate the wrong thing give you false confidence, which is worse than no confidence at all. This problem is getting more acute as AI generates more of our tests — they tend to be plausible-looking: exercising the obvious paths, asserting the obvious, and achieving respectable coverage numbers. But line and branch code coverage measures what was touched, not what was validated. A test can execute a line of code and assert nothing meaningful about it.

The technique worth doubling down on is mutation testing: intentionally introduce defects into your code, run your tests, and measure what gets caught. A defect that survives reveals a validation gap. Done continuously, this gives you evidence-based confidence your test gates are adequate. Beyond mutation testing, automated reasoning can mathematically prove properties about code behavior, eliminating entire categories of uncertainty.

With spec-driven code generation (like in Amazon Kiro where you and the AI collaborate on creating a requirements doc, then a design doc, then a tasks doc, then the code itself), we have a massive opportunity: auto-generating tests from specs to validate the intent of the builder. This will also help us increase the test coverage in areas that traditionally have lacked testing.

Pillar 3: Tests need to run efficiently and scalably

If we’re going to be moving at 10x speed, that also means we need to run 10x more tests. Is the infrastructure ready?

Running all tests for every change creates unnecessary overhead and expense and makes the inner loop too slow as we move integration and e2e signals earlier in the dev cycle. But you don’t need to run the full suite on every change. Smart test selection analyzes what changed and runs only relevant tests. Risk-based scoring tunes this further: high-risk changes trigger comprehensive suites; low-risk changes trigger minimal validation. AI can analyze the blast radius of a change and select the appropriate test set that provides confidence. Meta provides an industry example of both: their test selection catches 99.9% of faulty changes while reducing infra cost by 2x and test executions by 3x, and their risk scoring helped eliminate code freezes during sensitive periods, unblocking 10k low-risk changes per year.

Once you start looking, you see efficiency opportunities everywhere. For example, 18 months ago we predicted the shift in the Amazon Store, and we started augmenting all the physical device testing that we do with virtual device testing. We still very much run test the Amazon Shopping app on physical phones, but it’s much simpler, cheaper and faster to spin off 100,000 containers and run emulators than to get our hands on 100,000 phones.

Pillar 4: Shifting our mindset to “evaluation”

The first three pillars give you reliable, comprehensive, and efficient test signals. That’s what makes shifting left possible. But as AI agents generate and deploy changes autonomously, the question itself changes. It’s no longer just “does this code work?” It’s “does this change achieve what the builder intended?” That’s a fundamentally different question, and traditional pass/fail testing isn’t equipped to answer it.

This shift becomes urgent as experiences grow more agentic and non-deterministic. Amazon Rufus is a good example of this. When a system generates responses dynamically based on user intent and live context, the same input can legitimately produce different outputs. There’s no single correct answer to match against. You need judges that can assess whether a response is appropriate and reasonable, not just whether it matches a well-defined expected output.

Equally important is knowing when the system can’t make the call. Multidimensional evaluation will surface cases where trade-offs are genuine and human judgment is still required: ambiguous intent, conflicting constraints, policy violations, confidence below threshold. We need to establish the mechanisms so that when an agent hits one of these conditions, it knows to escalate to a human.

In Summary

The outer loop was never a feature. It was a compromise, a concession to the fact that important tests were too slow, too expensive, and too unreliable to run anywhere else. AI-accelerated development didn’t create this problem. It exposed it. It made it more acute and urgent to fix. The outer loop is dead. What replaces it is a continuous, AI validation loop that runs everywhere, all the time, giving builders the signals that matter, when they matter. The bottleneck was never writing code. It was knowing whether the code works. It’s time to fix that.

Recently, I spent some time building a computer from scratch with my 15-yr old, and it was a great experience. I’m going to try to convince you that you should too.

First of all, we need to get our kids more 𝐢𝐧𝐭𝐞𝐫𝐞𝐬𝐭𝐞𝐝 𝐢𝐧 𝐒𝐓𝐄𝐌 in general. Our high schools have systemically failed at doing so. For the last five years, I’ve been driving 30 minutes out of my way every day so that my kids could go to a high school that specifically has STEM tracks, and my 18-yr old is now studying Astrophysics at CU Boulder thanks to that exposure. I believe knowledge and passion for STEM is valuable for a lifetime. Secondly, we need to teach our kids to be 𝐛𝐮𝐢𝐥𝐝𝐞𝐫𝐬. Don’t just buy stuff that some company built. Get your hands dirty, tinker with it. Understand how electricity flows. Every time the dishwasher or the refrigerator broke, I opened things up and forced my kids to participate (not always willingly). Sometimes it went horribly wrong (and I ended up calling a proper professional), but many times I was able to get things working again (maybe with a little luck). The point is — don’t look at things as if they’re black-boxes, the way subcomponents interact in a system is always fascinating.

The idea of building a computer from scratch though did NOT come from me. I’m a software guy, not a hardware guy. I’m no stranger to hardware per say… During college I worked at our Campus Computing labs, fixing all kinds of hardware problems with thousands of computers. But that was… over thirty years ago. I messed around with some hardware as-needed when I was working at Microsoft in the nineties (I worked there for 12 years in Office and Windows). But (and this is a strange admission) the last time I ever used a Windows machine was my last day at Microsoft, back in 2009. I’ve always been a Mac guy for fun and Unix guy for work, but hid it successfully during my tenure at Microsoft.

As so, one day my son came to me and said, “𝐰𝐡𝐚𝐭 𝐢𝐟 𝐰𝐞 𝐛𝐮𝐢𝐥𝐭 𝐚 𝐜𝐨𝐦𝐩𝐮𝐭𝐞𝐫?” I was surprised, but hey You Only Live Once, so I said “Sure! But… YOU have to do the research.” Off he went for a little while, and I assumed that was the end of the computer building business.

Little while later I was looking at the Amazon cart and I noticed

• ASRock B760M PG Riptide (a 𝐦𝐨𝐭𝐡𝐞𝐫𝐛𝐨𝐚𝐫𝐝)
• Intel Core i5–12600KF Desktop Processor (a 𝐂𝐏𝐔)
• Crucial Pro 32GB DDR5 RAM Kit (2x16GB) (the 𝐦𝐞𝐦𝐨𝐫𝐲)
• Biwin Black Opal NV7400 1TB SSD (the 𝐬𝐨𝐥𝐢𝐝-𝐬𝐭𝐚𝐭𝐞 𝐝𝐫𝐢𝐯𝐞)
• Gigabyte GeForce RTX 5050 Windforce OC 8G (the 𝐠𝐫𝐚𝐩𝐡𝐢𝐜𝐬 𝐜𝐚𝐫𝐝)
• Montech AIR 903 MAX, E-ATX Mid Tower Case (the 𝐜𝐚𝐬𝐞)
• PCCooler RZ400 V2 CPU Cooler (the CPU 𝐟𝐚𝐧)
• Corsair CX-M Series CX650M (the 𝐩𝐨𝐰𝐞𝐫 𝐬𝐮𝐩𝐩𝐥𝐲)

I was impressed and amused. The kid had clearly done a lot of research. And also a little bit worried. Now I was on the hook for actually building this thing, and I had no idea how to do it.

And so two days later, things showed up, courtesy of Amazon Prime 2-day delivery. It was a beautiful summer day in the Seattle area, and we didn’t have a lot of those left, so we spread things out on the deck.

And so our adventure started with the 𝐦𝐨𝐭𝐡𝐞𝐫𝐛𝐨𝐚𝐫𝐝.

We identified the places where the 𝐂𝐏𝐔, the 𝐦𝐞𝐦𝐨𝐫𝐲, the 𝐒𝐒𝐃 and the 𝐠𝐫𝐚𝐩𝐡𝐢𝐜𝐬 𝐜𝐚𝐫𝐝 were going to go, and made a game plan.

First we opened up the box where the 𝐂𝐏𝐔 came in…

And popped it in into the 𝐦𝐨𝐭𝐡𝐞𝐫𝐛𝐨𝐚𝐫𝐝. That was surprisingly easy.

Next was the CPU cooler:

This is what it looked like once we opened the box:

At this point, the cat came to help. This was the most dubious part of the entire process. I understand you questioning the wisdom of having cat hair in delicate computer electronics. But the cat was determined to help, so after a number of attempts to discourage said kitty, we begrudgingly accepted her as part of the team. I remain unconvinced she actually added any value to the effort.

The next part was a bit unnerving: we had to apply 𝐭𝐡𝐞𝐫𝐦𝐚𝐥 𝐩𝐚𝐬𝐭𝐞 between the CPU and the cooler. The surfaces of a CPU and a heatsink aren’t perfectly smooth — they have tiny microscopic gaps. If you just pressed metal on metal, lots of little air pockets would form, and air is a terrible conductor of heat. Thermal paste fills those gaps and lets heat transfer much more efficiently from the CPU into the cooler. This was the Point of No Return. But the kiddo had done his research and he did it confidently — a lot more confidently than I would have.

And so now we were ready to actually attach the CPU fan:

Next was the 𝐦𝐞𝐦𝐨𝐫𝐲 and the 𝐒𝐒𝐃…

And so memory went first!

Then the solid state drive…

Time to mount the motherboard + CPU + memory + SSD onto the case…

I found 𝐜𝐨𝐧𝐧𝐞𝐜𝐭𝐢𝐧𝐠 𝐭𝐡𝐞 𝐩𝐨𝐰𝐞𝐫 to all the places where it needed to go very confusing. In fact we got this wrong (we didn’t initially connect the case fans correctly). That was a LOT of cables and it took a while to figure out where they all needed to go. We watched many youtube tutorials here. Thank God for YouTube!

In fact we had to remove the memory chips to connect some of the smaller power wires, our fingers were too big otherwise. Then we put the memory back.

Time to install the 𝐠𝐫𝐚𝐩𝐡𝐢𝐜𝐬 𝐜𝐚𝐫𝐝…

By now the thing was beginning to look somewhat finished… It was dusk so we moved our project indoors.

Now for the scariest moment of it all… Turning it on! To our amazement, the thing actually booted. I had about 50% confidence that it would!

Lastly, of course, the final test was installing Windows (an operating system I had literally worked on shipping back at Microsoft but had not touched in 16 years), and installing the first game (which is of course why the kid wanted to build a PC to begin with…). Gotta love those big blue fans!

All in all, it was a fun way to do something together. I really did take a backseat here: it was HIS project and HIS leadership. He gained skills, knowledge and confidence — and some pride too in having built something from scratch.

It also got me thinking about what a generational leap we’ve made. My first computer, when I was 15, had 2KB of memory — his has 32GB of memory. That’s 16 million times more memory. Something that would take my 1980s CPU one full day to compute takes his modern CPU less than a second to compute. My computer didn’t have a hard drive, I had programs on tape (a cassette tape held about 1MB of data and took a few minutes to load) — his is a blazing fast SSD with a terabyte of storage. That’s roughly a million times more storage, a million times faster transfer rates. Yet that humble computer from the 80s got me hooked on a life-long love for computer science.

What kind of live-long passions could you help your kid develop? We need more Builders in this world.

Every day, thousands of engineers run millions of tests to validate the Amazon Store works as expected. We aspirationally follow the test pyramid, having the bulk of our test coverage coming from unit tests, followed by a strong layer of component and integration tests for each service.

However there are some things that we can only test end-to-end (“E2E”), at the system level, in the UI. Additionally, in large distributed systems, the most interesting bugs tend to live in the way components interact with each other, particularly at scale, which makes E2E even more critical. This is what amazon.com looked like the day I was hired, sixteen years ago (every dot is some sort of service), and it hasn’t gotten any simpler to serve traffic since:

Since most of our customers are accessing the Store via our mobile app, that means running all our critical end-to-end tests on mobile devices. Our testing has to reflect the way our customers are using us.

Amazon Store engineers have always tested local changes both via their own physical devices, and via virtual devices (by spinning off an emulator or simulator in their dev environment). And this works well for a single developer. But how do we evolve our device test infrastructure to be able to do this efficiently and at scale: thousands of engineers, thousands of commits, thousands of tests, hundreds of devices?

First: We created labs of physical devices

To run all our critical tests on mobile devices, we maintained vast labs of physical devices (iOS, Android) throughout the Amazon campus.

Maintaining these labs is cumbersome and expensive. Our engineers need to spend their time keeping track of the health of the devices, and fixing them when they have issues. Unlike commercial servers, consumer devices are not built to have the kind of reliability required to be running 24x7 in labs, so keeping hundreds or thousands of phones healthy becomes a scaling bottleneck. Lab management is non-differentiated work for the majority of us: we want to spend our time writing great tests to ensure Amazon products provide our customers with the highest quality experience, not maintaining devices in labs.

There are other problems too. It isn’t always easy to get our hands on devices for all the configurations we want. As more versions of iOS and Android devices reach the market, that matrix of must-have devices grows very large. These physical labs also can be wasteful, as most of the devices can sit idle for the majority of the time, only to run a test occasionally.

Despite all these challenges, physical device labs continue to be critical to us in many business (and will always be). How does Amazon test “Living Room” devices? Smart TVs (like Samsung, Sony, LG, etc); streaming devices (like Fire TV sticks, Rokus, etc); game consoles (Playstation, XBox); set top boxes, blu-ray players, etc? How do you test prototype devices that will reach the market years from now? A few months ago, I visited our Prime Video London offices and posted about it here.

But for off-the-shelf mobile devices, we needed to think differently about our test infrastructure, and so we reached our first inflection point.

Then: We moved to AWS-managed physical device infrastructure

This is when we started shifting a lot of our testing to run on AWS Device Farm. This allowed us to run our tests across an extensive range of desktop browsers and real mobile devices, without having to provision and manage any testing infrastructure. We were able to offload the non-differentiated lab management work to AWS, which freed our engineers to spend their time improving the actual tests. Additionally, by using Device Farm we were able to only pay for the devices that we used, instead of having giant labs with mostly idle devices, and benefit from their extensive device coverage.

AWS Device Farm gave us the best of both worlds: we were able to run our tests against physical devices, without the burden of having to have physical devices. By leveraging AWS, were able to decommission a number of labs we owned and scale our testing significantly. Reliability of our tests improved: Device Farm runs with an SLA of 99.9%, which was prohibitively expensive for us to achieve when we operated our own device labs.

Amazon at large still does have tens of thousands of devices in private labs for testing purposes, particularly for Kindle e-readers, Alexa devices, Fire Tablets, Fire Sticks, Ring devices, etc. But for the Amazon Store, the vast majority of our device testing happens via AWS Device Farm, although we still operate a number of private device labs for very specific reasons (such as needing access to hardware).

Today, AWS Device Farm is business-critical for ensuring that changes to the Amazon Store are high quality and don’t negatively impact our customers. They have been great partners in our quality journey, and I expect that will continue to hold true indefinitely.

Now: One more inflection point ahead

At this point, we’re facing the next inflection point: there are two trends that are going to increase the scale at which we run our tests, easily by 100x.

[1] Shift-left testing

This means moving testing activities earlier in the development process, focusing on identifying and resolving issues at the earliest possible stage to improve quality and reduce costs.

Traditionally, because UI tests were expensive to run and flakey, we only ran most of them after a piece of code was submitted and merged, as part of the outer developer loop. This led us to identifying errors too late in the process. Amazon, and the industry at large, is making significant investment to shift testing left, so that we can run those same tests as part of the inner developer loop instead, and find the errors much earlier in the process, reducing the blast radius.

There’s something that makes shift-test-left an order of magnitude more important today than a few years ago:

As we increasingly use AI to generate code, providing LLMs with ways to rapidly and iteratively validate how local changes impact interconnected services and complex distributed applications, before commit, is essential.

The idea of shifting left has existed in the industry for as long as I can remember, but there has been a significant emphasis on it in the last few years. A quick search on Google trends corroborates it: the following graph shows the popularity of searches for “shift-left testing” on Google between 2016 and 2025 (a value of 100 is the peak popularity for the term):

[2] LLM-based testing

Traditionally, creating UI tests has been expensive, because it required engineers with domain expertise in various UI testing frameworks (e.g. Selenium, Puppeteer, Cypress, WebdriverIO, Cucumber, Appium, Espresso, XCTest, …) manually writing these tests. They were also cumbersome to maintain, as the tests were subject to failing if the UI changed and required to be constantly modified to reflect front-end changes.

LLMs provide a way to author tests that is more natural. Engineers can write their expectations for the behavior of a system in plain English, and an LLM can interpret that and decide how to drive the UI to accomplish that desired goal. LLMs introduce the ability to analyze images and text, which makes validation simpler and more natural too. No need to couple your tests to a specific framework. A test can simply be expressed as “Go to the Amazon Store, search for Harry Potter books, and verify the books are present in the search results.” Maintenance is simpler too: because the LLMs can navigate some amount of ambiguity, they can be resilient to simple UI changes that in the past would have broken thousands of tests (eg. moving a button to a different location or renaming a UI element).

Tests being significantly easier to create and maintain frees our engineers (and LLMs) to create them at a rate that was simply unthinkable just a couple of years ago.

Combine [1] our commitment to shift-left testing broadly for humans and LLMs, with [2] a reduced cost for creating and maintaining tests, and now you understand how that inflection point is upon us: our testing infrastructure will need to run 100x more tests than we do today. With that kind of scale, cost and speed considerations, running these tests against physical devices only becomes challenging.

Augmenting (NOT replacing) physical device testing with virtual device testing

Scaling by 100x requires different thinking. This inflection point means that we need to invest in being able to run these tests at scale on virtual devices, via emulators and simulators, running on containers (e.g. Amazon ECS, Amazon EKS).

Using virtual devices is not a particularly novel concept; engineers have been testing in their inner loop with an emulator running on their developer desktop for a decade. But running emulators and simulators at scale is a new shift.

Virtual devices have the following benefits:

• Scalable: It’s a lot more feasible to spin off 10,000 emulators on containers on AWS than it is to get my hands on 10,000 physical phones
• Reliable: Virtual devices tend to be more reliable than physical devices, because they’re running on highly available servers. Physical devices are susceptible to hardware failures like getting disconnected from the wifi, a bad USB cable, etc.
• Cheaper: running a test against an Android virtual device is about 10% of the cost of running a test against a physical device — simply because these can run on containers on inexpensive AWS servers (vs. on expensive consumer devices).
• Faster: the actual runtime of a test doesn’t change whether it’s running on a physical device or a virtual device, but the setup time for the test (as well as the teardown time) can be significantly faster. (There are exceptions: physical devices are actually faster than emulators in cases where hardware matters, such as games that require and use dedicated GPUs, or the 3D scan/video playback feature on Amazon Store app).

Given that virtual devices can be more scalable, more reliable, cheaper and faster — why would we continue running on physical devices at all?

The answer is simple but critical: Fidelity.

You should use the same devices your customers use. Unlike virtual devices, physical devices give you a more accurate understanding of the way users interact with your application by taking into account factors like memory, CPU usage, battery draw, time to first frame, location, and modifications made by manufacturers and carriers to the firmware and software. Emulators and simulators may lack that fidelity, leading to issues escaping to production because they did not surface on virtual devices, but will most certainly surface on physical devices.

Hereby lies a tradeoff. Do you want scalability/reliability/cost/latency, or do you want fidelity?

My assertion is that this is not a “OR” choice, it’s an “AND” choice.

My personal view is to have a strategy that leans towards the first for some cases, and leans towards the second for other cases:

• Use virtual devices when scalability/reliability/cost/latency matter most: [i] Anything running in the inner developer loop should default to running on virtual devices; [ii] Earlier stages of CI/CD (e.g. Beta stages) should default to running against virtual devices
• Use physical devices when fidelity matters most: [i] All business critical test cases must run on physical devices; [ii] All hardware-dependent test cases, and performance/latency measurements; [iii] Final stages of CI/CD (e.g. last safety net before things are pushed to production) must run on physical devices

Here’s some more thoughts on strategy:

• Your tests should generally not hard-code or care whether they’re running on a physical device or a virtual device. In fact they should not care about what device they’re running on at all.
• In reality some tests specifically validate a particular platform, but as a general rule, decouple the “what” (the test) from the “where” (the device). This allows you to dynamically run any one test on multiple platforms without making changes.
• Specific test cases may only be able to run on physical devices (e.g. they require a specific hardware to run on), regardless of where in the software development cycle they are executed.
• Service owners may override this behavior for situations where the fidelity of physical devices trumps scalability/reliability/cost/latency concerns.

In that sense, virtual device testing is augmenting/enhancing your testing capabilities but in no way replacing the physical device testing you do. They are both part of a holistic test strategy.

The test pyramid ends up looking more like this:

Notice the percentage of E2E tests increases in this model, as a consequence of shifting testing left and LLMs simplifying the authoring and execution of E2E tests. This is positive, because it helps us prevent the harder-to-find bugs that lurk in the interaction between our large distributed systems.

I’m extremely excited about a future where testing happens ubiquitously in the inner developer loop as it does in the outer developer loop, therefore preventing issues proactively. I’m also extremely excited about how GenAI will help us increase test coverage of our products. And by scaling both our physical and virtual device testing infrastructure, I believe we’re ready for that future!

If Medium puts this content behind a paywall, you can also view it here (LinkedIn).

Right now, we have two GenAI camps — the All-In-Folks and the Skeptics. Some people view GenAI as the solution to every problem in the world; others are absolutely refusing to use it or trust it and are clinging to the traditional ways of doing things. I tend to be a pragmatic dude, somewhere in the middle. I am excited to use it (every day), but I also try to understand the limitations and be realistic. The more I use it, the more excited I get about its future. Like any other new technology, I try to look ahead — a lot of the current limitations will eventually be worked out. Skate to where the puck is going, not where it has been.

For all the time we spend talking about GenAI generating code, we spend very little time talking about GenAI generating and executing tests. I believe they are so much better at the latter than the former.

Specifically, I have been playing with GenAI for UI testing. There’s much literature about GenAI for generating unit tests, but not nearly as much for integration and end-to-end testing. I had theorized that LLMs would be pretty good at that.

The reason UI Testing is business critical is that it tends to be the last line of defense before something is pushed to production. If your test processes are following the Test Pyramid, you’ll have a healthy amount of unit tests, followed by integration tests. But the UI tends to be the place where all functionality comes together and you often find those pesky problems that escaped through the cracks of unit and integration tests.

My first exposure to UI Testing was in 1997. I was working at Microsoft and testing Microsoft Office 2000 at the time. Two things I immediately noticed: the frameworks for driving UIs were clunky, and the tests themselves were brittle. Not that much has changed about the space in the last 30 years. Recently, as I am leading testing the Amazon Store, this has been on my mind a lot.

There’s lots of inputs you could feed an LLM to have it generate tests: specs, design docs, production code, pre-existing tests, code coverage information (so that it can aim at generating code that tests paths that haven’t been exercised yet), operational issues (to give it some hints as to where problems may lurk), etc.

The main question for me was: in what format should those auto-generated integration tests be?

• Path#1 is having the LLM generate tests as code (e.g. as Selenium, Appium, etc), then executing those tests as usual.
• Path#2 is taking a bigger leap — what if we used an LLM for generating the tests as natural language, then used another LLM to execute them?

There are pros and cons to both paths.

• Using LLMs to generate test code means you can inspect the code, and once it’s generated it becomes “deterministic” — meaning you know exactly what gets executed on every single test run. This is within people’s comfort zone. [Not to be pedantic but I would challenge that if you are testing a distributed system with retries, your tests are already not deterministic, but I digress…]
• But if we’ll live on a world where both LLMs and humans write integration tests, it’s so much easier for a human to write a test in natural language than it is to write it using a traditional UI framework like Selenium or Appium. So test authoring in natural language wins.
• Generating tests as code in a traditional framework also means that whoever is inspecting that code needs to be knowledgeable in that particular framework. In contrast, anybody can inspect a natural language test and immediately understand what it’s doing. We all speak natural languages!
• Generating tests as code means you are now on the hook for maintaining and evolving that code for the rest of your life, like any other piece of code. Software upgrade? Your problem. Production code changes, so you have to change the test code to reflect that? Also your problem.
• Most interestingly though, generating tests as natural language and having them executed by an LLM creates an opportunity to think differently. UI tests are notoriously flakey. Two of the reasons are: [1] unexpected message boxes can pop up randomly; [2] often tests drive a UI by navigating the DOM and looking for element ids, if those change, the automation doesn’t know what to do. Both of those are easily navigable by an LLM.

To illustrate this, I wrote a simple UI test and fed it to an LLM for execution. As I walk you through this example, I encourage you to not just focus on the Action that the LLM took, but focus on the Reasoning of the LLM. The little thinking bubbles in my graphs are verbatim what the LLM reasoning field was, which gives you a great insight into how LLMs reason through their world.

Step 1: It typed “Harry Potter book” in the search result.

Step 2: It clicked on the Search button

Step 3: It verified that the right book was present in the search results.

Step 4: It clicked on the correct book. Ironically, it did that while trying to save me money, since it picked that version because “it was the most economical option!”

Step 5: This is where it goes off the rails a bit. The “Add to cart” button is below the fold here, so it’s not visible in the current screenshot. The bot found an “Add prime to get fast, free delivery” button and it theorized that if it clicked it, it would add it to the cart. It was not correct, but to be fair, it was not a terrible guess, as the bot was trying to find a path.

Step 6: It figured out on its own that that was not correct, and it theorized that if it scrolled down, it would find it, which is correct.

Step 7: It found the “Add to Cart” button, it’s happy now!

Step 8: It’s validating that it added it to the cart. Notice it validates in four different ways, which is better than how I would have validated if I was writing this test!

This simple test illustrates the power of LLMs in the domain of UI Testing.

The elephant in the room is non-determinism. How do you guarantee that the LLM takes the correct path navigating through your app as it is testing it? In my little example, the bot clicking on the “Add Prime” button was a hint that when navigating a UI, a bot could definitely get sidetracked.

More amusingly, in another execution, the bot couldn’t login with the given username and password, so it attempted to create a brand new account all on its own. When that too failed, it actually attempted to chat with customer service to work things out. This is actually really impressive — that was one determined little bot!

To address that, one idea we’ve been kicking around among some friends is giving each test an “execution budget.” It took 8 steps to execute my little example. So if a bot is still trying to accomplish the task after 20 steps, it probably veered off course and it’s doing something it isn’t supposed to be doing. So tests could have a budget as a guardrail.

More broadly, as we shift more towards depending on LLMs for test execution, we need to spend a lot more time thinking about guardrails — what actions should the bot simply never take?

Another idea is to have a “judge LLM” that analyzes the steps the “executing LLM” took and its reasoning and decides whether that was correct or not. I see the LLM-as-a-Judge pattern being more used these days on a lot of tasks.

We also have the ability to gather data objectively. We have hundreds of thousands of legacy tests written in frameworks like Appium, Selenium, etc. Could we auto-generate a natural language test suite with the exact same tests, and run both suites in parallel for a while, comparing their results? Ideally, at some point we would have concrete evidence that we can replace the traditional tests with natural languages tests.

Will it also replace human testers? I don’t like to use the word “replace,” but it will definitely shift the work they do. In some orgs, we employ armies of manual testers that perform the same repetitive tasks to certify every release candidate. Sometimes because writing automation is too expensive; sometimes because the product changes too quickly to even write test automation; sometimes it’s lack of forwarding thinking and unwillingness to invest in engineering excellence. But any situation where a release is gated by a human being is not scalable or sustainable. It is also non-deterministic… humans are notoriously non-deterministic and make mistakes too. I would like GenAI to perform those repetitive tests, so that these testers can focus on applying their intuition and hard-earned experience in how products can fail to explore the surface more freely and more creatively.

One more consideration is that LLMs are slower to run these tests than traditional automation, and GPUs are expensive. So bots loose in both latency and cost today. But that will not always be the case. We’re looking into ways to cache things, to process in different ways, etc, to tackle both of those. I don’t want to wait until latency and cost are fully solved to take a bet on this, because if we do that, we’ll simply get started way too late. Another way of thinking about this is that the savings in human authoring and maintenance are worth the latency/cost even as of right now.

Lastly, LLM-based test execution opens up doors — for us to do things we simply couldn’t do before. A fascinating whitepaper from some researchers at Amazon describes how they created agents with diverse personas and goals for testing purposes. You can have a test with a set of instructions that can be interpreted differently depending on the persona — which allows you to discover bugs that you may not otherwise find. For example, what if one of your personas was blind? We don’t need to write special tests to validate whether the Amazon Store is accessible to people with disabilities; we simply should run all our tests, but with personas that reflect different disabilities. This detaches the test steps from the execution behavior — that is something we could never truly do before. We can add all kinds of non-functional guardrails to our already existing functional tests.

To me, the advancement of LLM-based UI-testing is not a matter of IF, it’s a matter of WHEN.

My first experience testing professional software was working at Microsoft in 1997. I installed a veeeery early debug build of Microsoft Office 2000. Excitedly, I double-clicked on the Word icon. The hard-drive frantically spun for 5 minutes, then I got a message box that it had crashed. The thing didn’t even start. It was unimpressive. The next day, I installed that morning’s build, and it opened, but crashed within 3 minutes of me playing with the app. Yet somehow, a couple of years later, we shipped that codebase (much improved) to millions of households. It took a lot of very determined engineers to try things out, learn, iterate, improve, relentlessly. Today’s world is no different.

I am both excited and terrified. But I think it’s time to jump in.

If you put code in production and no one uses it, does it still have an impact?

If Medium puts this content behind a paywall, you can also view it here (LinkedIn).

Back at Google, during a 1:1 with one of my engineers, they mentioned that they had launched a feature into production a couple of months prior. I asked if the feature had been successful. This engineer gave me a confused look and started describing the technical complexity. That was great, but I wanted to know if it had made a difference to a real human being. Turns out that only 2 googlers were (sparingly) using it. It was a great feature, it just hadn’t been properly landed.

There is a huge difference between Launching something and Landing something. As an engineer, the work you do to launch is “fun” and “shiny” — you design a feature and write code, which is what got you into software engineering to begin with. Every code review provides a tiny little hit of dopamine and is a concrete artifact to show others how intelligent you are. The work you do to land is boring and often non-technical. A lot of it has no immediate reward. You don’t control 100% of your destiny. It’s fuzzy and dissatisfying.

But without Landing, your Launch is useless.In the software industry, the old “If a tree falls in a forest and no one is around to hear it, does it make a sound?” translates to “If you put code in production and no one uses it, does it still have an impact?”

In fact, I would argue it has a negative impact. You have increased the complexity and surface of your product and created additional operational load, for no value.

After that wake up call, I started looking closer at all our launches. The majority lacked the most basic discipline around landing. People put code in production and happily moved on to the next intellectual challenge that would sound good on their performance review.

Determined to drive some cultural change, I put together some slides on “Launching v. Landing” and gave a presentation to the entire org, recorded it, and shared it broadly. A couple of hundred people watched it, which I thought was positive.

But there was no behavioral change whatsoever.

I later understood why. I was new to the team (and fairly new to Google), and I had not Earned Trust from the engineers yet. I was just a random dude that had shown up one day as a Senior Staff Engineer without any artifacts. I had written 100k+ lines of production code at Amazon and launched a dozen products used by tens of thousands of amazonians every day, and spoken at dozens of internal conferences, but at Google I was a Nobody.

I needed credibility to inspire behavioral change. And I needed technical artifacts to get credibility. I decided to embed myself in my teams as a part-time engineer. This would help me gain a deeper understanding of the nuances of the products I was leading, and earn trust from my engineers.

I started looking for code I could write that spanned multiple teams, and found a perfect little problem. It was not particularly complex, but it could potentially help a lot of customers, nobody was doing anything about it, and required changing code in three different codebases and convincing multiple teams.

At the time, I was a Tech Lead for company-wide Infrastructure for Integration Testing at Google (I’ve written about this here and here). My product launched ephemeral, hermetic test environments for Google’s CI/CD system (we called them “SUT” for “System Under Test”). Launching these SUTs could take an extremely long time, but where that time was actually being spent was a black box to frustrated customers that wanted to improve it.

Here is an example of a test run that was part of CI/CD, some text redacted. In this case, launching the SUT took almost 28 minutes, and it became the long pole for the entire release process. But you didn’t know why it took 28m.

Here’s an example of the more granular metrics that I envisioned we could expose to our customers, zooming in on what is happening during those 28 minutes. When you stand up an SUT, you first need to build a bunch of components, then you often create a database (Spanner) to hold whatever data you want your system to return, then you need to data-seed that database, and finally you can start your server(s).

In a lot of cases, you could just visually spot the component that was the long-pole. In the example below, it’s easy to see that one component takes 2x to build as the others, and another component takes 2x to start as the others. Now you know what to go optimize.

Also, if I emitted these metrics, not only could I surface them per-customer-run on the UI, but also, I could aggregate them at scale in our own operational dashboards, for my team to understand our boot latency percentiles for the millions of SUTs we were starting every day, identify regressions, and quantify improvements. Here’s an example of our aggregated metrics over a period of a couple of weeks.

So I talked to the teams, got buy-in, did some of the work myself, and convinced others to do the rest. The last step, surfacing the metrics on the UI, turned into an intern project. And sure enough, once the code was pushed to prod and the internship was over, I noticed the team called the thing “done” and moved on to the next thing. It was the same behavioral pattern I had seen before.

Only that, of course, we were NOT “done.” It was in production but nobody was using it. Here was an opportunity for me to lead by example. I came up with a list of things-to-do to guide my process.

1. Was the feature discoverable?

A new feature must be easily discoverable.

I put on my “new customer” hat and navigated the UI. Once you clicked on the SUT, an “Operations History” tab would open up, and you were greeted with a Wall-Of-Text containing the massive command lines that had been executed. Underneath each, a tiny little tab called “Event Timings” was collapsed and easy to miss.

Very few people would want to scrutinize a 25-line command line, so I collapsed it so that the event timings tab was easier to see. I had the theory that more people would want to see event timings than they would the command line — and I actually proved that theory by emitting counters for how many times our customers expanded each one.

This trivial little change (about ten lines of code) increased customer adoption of event timings by 10x. You don’t have to do a lot of hard work to make something more discoverable, you just need to put on your customer hat and view things from their perspective. Most products have massive amounts of low-hanging fruit.

2. Was the feature usable?

I thought our UI was good for a first-launch, but again there was a lot of low-hanging fruit. A few brave customers tried it and gave feedback like this:

I was excited to see this feedback. This individual was motivated enough to use my little feature and cared enough to take the time to write down feedback. So I pinged him and we chatted 1:1 for a bit. Always, always, take the chance to attentively listen to your early customers. Their feedback and perspective is invaluable.

Based on my conversations with him and a few others, I made a bunch of little improvements. Each one of them was a trivial code change, but it improved the usability of the product significantly.

For example, the original bars weren’t labeled, so it was hard to see what was what, so I added labels. It took me 5 minutes to do and it made my customer’s lives better.

Some of the bigger SUT had hundreds of components which made the Event Timing pages long, and customers were scrolling up and down trying to figure out timing for the events of a particular component. So I grouped all the events for a specific component in the same row, which made the graph a lot more compact and made it very easy to compare timing for specific event types across components.

Again: none of these things was particularly difficult to do from a coding perspective. Most changes were ten lines of code here or there. But they collectively improved the usability of the product by a lot. To me, as an engineer, it’s polish and attention to detail.

For example, there was a field “Duration” which just had a double, i.e. “101.12639.” Putting on my customer hat: what unit is this? Probably seconds? Why do I need so much precision in the seconds for something that generally takes minutes? So I changed it to 2 digits followed by an English readable version, i.e. “101.13 seconds (1m41s)”

3. Was the feature useful?

Usable and useful are two different things. I monitored tickets, emails and forums, to see if anybody mentioned the feature. I had dashboards with metrics (more on this later), but I wanted to augment those with subjective anecdotes from actual customers to understand whether this thing had “landed” or not.

Getting emails like these was encouraging:

Or this one:

The takeaway here is look at both hard-data and anecdotes. Both should tell a consistent story.

4. Did we have good documentation?

This was another pet peeve of mine. A lot of our features were either undocumented or poorly documented.

I needed to lead by example. I created a page in our documentation to explain not only how to use the Event Timings pane, but also some best practices and real-world examples of how you could use it to find components to optimize.

I also like to be able to find the corresponding documentation when I’m using the product itself, so I added a little “For tips on how to understand this pane, go <here>” link in the Event Timings pane UI itself.

Lastly, our documentation had to be discoverable, so I made sure it was the first hit when you did common searches like “slow SUT” or “optimize SUT”, etc. After all, Google is a search company, so there was ways to tweak the internal search to bubble up specific urls for specific terms.

5. Did I have usage metrics?

You should never launch a feature without having at least a scrappy way to understand who is using it.

There were all kinds of sophisticated metrics I envisioned, but I didn’t want perfection to be the enemy of good. For my initial landing, the UI emitted a counter that tracked how often the Event Timing pane was displayed. Since it was collapsed by default, I figured a Customer deliberately clicking on the little triangle that expanded the pane was a good signal that they had noticed it and were at least curious enough to click on it. I added another counter to track if a customer had hovered inside the expanded pane and interacted with the bars. Roughly, the “Display” metric gave me an upper bound and the “Hover” metric gave me a lower bound. It wasn’t perfect, but it was a decent initial signal. If the feature took off and was used by thousands of googlers, I could invest in more sophisticated metrics, but until then, I needed to be pragmatic and scrappy. I saw between 150 and 200 unique people interacting with it every day, which was a decent landing for a little feature. This is a 3-week period:

I also tracked metrics on how many people went to the documentation page I had created — it was getting about 100 reads per day.

6. Had I socialized this sufficiently?

I had been working on improving our customer-facing announcements (I wrote about this here: “Injecting Customer Obsession into a foreign culture”). To land my feature, I needed to socialize it, so I leveraged that.

The first paragraph of an announcement is critical. People are very busy and if you don’t get their attention they’ll move on to their next email. You have to clearly articulate the exact customer painpoint that your feature is addressing. Why should your customers keep reading?

My email announcement had a little tracker too, to see how many people actually opened and read the email. This gave me the insight that 1400 googlers read my announcement that first week. Just because somebody read my email didn’t mean they wanted to use the feature, but it did tell me they cared enough to at least open the email, so it was another imperfect yet useful metric.

Lastly, my email had a special link to the documentation with another counter, so I could see how many people not only opened the email but actually clicked on the “Learn more” link — this was a higher fidelity signal than the first one.

Conclusion

Always be mindful that launching is not the end, it’s just the beginning. To land it, make sure your feature is discoverable, usable, useful, that you have documentation and metrics, that you socialize it properly, and that you value and learn from your early customers.