Most people don’t call software slow because they’ve measured them against a benchmark. They call it slow because it feels slow.

Users experience this in a few familiar ways. It could be a button that doesn’t respond. Or the page reacts a beat late when you’re typing, scrolling, or clicking around.

The catch is, performance tests can miss that experience, because it has traditionally focused on the backend. Traditional load testing, for example, simulates users by sending requests, then measures response times and error rates from the server. For modern web apps, that often misses a lot of what happens in the browser.

Modern web apps don’t always “finish” when the server responds. After the first response arrives, the browser is often still working: loading assets, running scripts, fetching more data, and rendering the interface. In many apps, this browser work is where the delays are. The API can be fast, while the UI still feels stuck.

If performance is experienced in the browser, a complete performance test needs to measure the browser too.

Where traditional load tests fall short

Most performance testing starts with a simple idea: simulate real usage at scale, and see how the application responds.

In practice, that usually translates to load testing. We’d ramp up backend calls as we increased the number of users, and watch what happens to response times and error rates. This would give us a clear view of server capacity. It also tends to be repeatable, which makes it useful for baselining and comparing changes over time.

The catch is that modern web apps aren’t defined by a single request and response. A single user action can trigger a chain of work across the client (the browser) and the server. The browser loads assets, runs scripts, fetches more data, updates the UI, and only then becomes properly usable.

A lot of this work happens asynchronously. The app doesn’t do one step, wait, then do the next. It kicks off multiple bits of work in the background, and the page updates as each piece finishes. Some tasks run in parallel. Some depend on earlier results and some only happen after the first render.

The first animation below shows what this looks like in practice: one user action, like a button click, can quickly cascade into work across the client and server. In this example, the time to usability is ~2.5 seconds, and time to stability is ~3.5 seconds.

If we try to model the above user action faithfully with load tests, the task becomes complex very quickly. You may need to simulate dozens, sometimes hundreds, of requests, and link and parameterise them realistically so the chain behaves realistically. In practice, that means accounting for asset loading, incidental browser traffic, and third-party activity as well.

This is why traditional performance testing tends to rely on abstraction.

The next animation shows what that abstraction looks like in practice. It starts with recording the full cascade of activity, then strips it back by discarding browser-side work and any requests the test isn’t modelling. What’s left is a smaller set of backend interactions that can be executed reliably under load.

In our example, the abstracted load test might report the time taken as under 1 second, because it’s only measuring the selected backend interactions.

So traditional performance tests often narrow the focus to a small set of “important” endpoints. This requires judgement calls about what to keep and what to discard. And often, it is easy to overlook and discard something that’s discovered to be useful later.

But this over-simplification is also deliberate for a few more reasons:

• Stability — fewer moving parts means fewer flaky results caused by background calls, caching, or client-side timing.
• Interpretability — it’s easier to explain a spike in one endpoint than to untangle fifty small calls that all moved a bit.
• Cost — running and maintaining large, browser-faithful scripts takes time, and at scale it can be expensive.

For a long time, that trade-off made sense. You got fast tests, clean metrics, and a reliable view of server behaviour under load.

But the abstraction also hides things users actually see and feel. API-level tests won’t see client-side execution, rendering delays, long tasks, and blocked interactions. The UI can feel slow even when no single API call looks terrible. A test can report a fast response time while users are still waiting for the interface to become usable. In short, we’ve already abstracted away what users experience in the browser.

Seen this way, traditional performance testing isn’t wrong. It’s just incomplete. It tells you how the server behaves under load, but not how the application behaves for the user. As more execution shifts into the browser, that distinction matters more.

So what does performance testing look like when we stop abstracting away what users experience in the browser?

A broader view of performance testing

If API-driven performance testing tells us how the server behaves under load, it says far less about how the system behaves for the user. Closing that gap means expanding performance testing beyond the API layer and into the browser.

Browser-driven performance testing starts from the user’s point of view. First, we have to script user interactions through the UI as end-to-end tests do: navigate, click, submit, and wait for state. Then we scale those user flows under load, so we can measure performance across the full path, including the client and the server, rather than isolating individual endpoints.

This changes the kinds of issues performance testing can surface. Instead of focusing only on response times, we can see how long it takes for a page to become usable, how responsive the interface feels under load, and where delays actually emerge. Problems that are invisible at the API layer — like rendering stalls, long-running scripts, chained requests, and blocked interactions — become visible once the browser is part of the test.

The point isn’t that backend testing is useless. It’s that modern applications do meaningful work in the browser, and that work shapes the experience. If we don’t measure it, we’re not accounting for a crucial part of the system.

Browser-driven performance testing brings measurement closer to what it’s meant to reflect: how the product behaves from the user’s point of view. And by leveraging end-to-end tests performed in the browser, it also solves for the complex tasks of modelling the entire cascade of work performed in the client and the server.

But at this point you might be thinking: this sounds good in theory, but does it actually work at scale?

Browser-driven performance testing in practice

Moving performance testing into the browser has becoming increasingly simple with modern tooling. One practical approach is to use an end-to-end automation framework like Playwright to describe real user interactions, then run those scenarios at scale using a load engine like Artillery.

We’d start by capturing a user flow with a Playwright script. For example, a simple login and dashboard load might look like this:

export async function loginAndLoadDashboard({ page }) {
  await page.goto("https://example.app/login");
  await page.fill("#username", userEmail);
  await page.fill("#password", userPassword);
  await page.click("button[type=submit]");
  await page.waitForSelector("#dashboard");
}

To turn that journey into a performance test, it needs to be executed repeatedly under controlled conditions. This is where load generation with Artillery comes in.

Artillery orchestrates and measures the Playwright tests at scale. It controls how many browser sessions run, how users arrive over time, and how timings are collected and aggregated. Because browser sessions are compute-heavy, Artillery spreads execution across multiple machines. This ensures the test is limited by the system under test, not the machine running the test.

Once a Playwright test has been written, configuring load generation is surprisingly simple with Artillery. Here’s what an Artillery configuration YAML might look like:

config:
  target: "https://ourAwesome.app"
  phases:
    - duration: 300
      arrivalRate: 5
  engines:
    playwright: {}

scenarios:
  - engine: playwright
    testFunction: "loginAndLoadDashboard"

The responsibilities are deliberately split. The browser automation defines what the user does. The load generator defines how often it happens, at what scale, and how results are aggregated and measured. This separation keeps user scenarios readable and reusable, while allowing load shape, scale, and distribution to be tuned independently.

There are other alternatives to this tool stack as well. Some teams extend existing load testing tools with browser execution, while others combine different automation frameworks and runners. The specific tools matter less than the approach. What defines browser-driven performance testing is the decision to measure performance where users actually experience it: in the browser, executing real behaviour, under realistic load.

But of course, nothing comes for free. The next obvious question is what you gain, and what you trade off, by doing performance testing in the browser.

The trade-offs

Browser-driven performance testing is more realistic, because it simulates the end-to-end flows users run. But in return, scaling browser tests requires more compute.

In theory this should be more expensive. But in reality, this option is far more accessible in the current tooling landscape. With modern headless browsers, better orchestration, and the option to run on self-managed infrastructure, browser-based tests can be run with predictable cost. In many cases, this ends up being cost-competitive with (and, in my experience, even cheaper than) traditional load testing tools.

The developer experience has improved too. Tools like Artillery + Playwright let you keep scenarios close to the functional tests teams already know, while keeping load shape, scale, and distribution in a separate configuration that’s easy to tune.

Wrapping up

Performance testing has always aimed to answer a simple question: is the app usable at real-world scale?

For a long time, the best way to answer that was to stress the server and measure response times. That’s still useful as it tells you a lot about the application capacity and stability. But it’s only one part of the picture.

Traditional load testing tools tended to rely on abstraction. They simplify real user behaviour down to a smaller set of backend interactions. That makes tests repeatable, but it can miss issues that only show up in the browser.

Browser-driven performance tests close that gap. They run real user flows in the browser under load, capturing both client-side behaviour and server-side signals.

The end result is a more complete view of performance: how the backend holds up under load, and how the product behaves for users at scale.

Note: This post is independent. It is not endorsed or sponsored by any company and product mentioned.

If you enjoyed this article, please join my substack newsletter — refluent. You’ll also be notified when the next article is published.

Refluent shares sharp insights on quality engineering, performance testing, and the future of intelligent automation.

Rethinking performance testing for the modern web was originally published in refluent on Medium, where people are continuing the conversation by highlighting and responding to this story.

Self-healing has long been one of the toughest challenges in software test automation. Automated tests are meant to catch real issues, but they often break when an application changes unexpectedly. Over time, these failures pile up, shifting teams from building new tests to endlessly patching old ones.

This is the paradox of test automation: it promises stability, yet it demands constant repair.

In theory, self-healing can ease this burden by allowing tests to adapt automatically to application changes. In practice, though, self-healing has been difficult to achieve. Traditional self-healing approaches rely on rigid, rule-based logic that struggles to keep pace with evolving applications and often requires intrusive adjustments to the codebase itself.

But recent advances in artificial intelligence are changing that. Large language models (LLMs), such as the ones powering ChatGPT, offer a way to reason about context and intent, making adaptive, context-aware self-healing a realistic goal.

This article will explore what self-healing is, why it matters, and how LLMs can enable it. We’ll also work through a high-level example of implementing self-healing from scratch. We’ll break down the challenges of self-healing and discuss the fundamental building blocks of an AI-assisted self-healing system.

What exactly is self-healing?

One of the most frustrating parts of test automation is when a test fails for the wrong reason. An updated feature may be working perfectly, but the test breaks because something minor changed, such as a label, an ID, or the shape of a response.

Consider a simple case. Suppose you have a test that expects a button labelled Save. A developer renames it to Submit, but everything else still works exactly the same. This causes the test to fail, not because the feature is broken, but because it was written to look for the old label.

This brittleness is everywhere. Automated tests are often tightly coupled to specific implementation details, so harmless changes like this can trigger failures. The Save→Submit example shows a UI change, but the same problem appears at every layer: a renamed input in unit tests, a restructured API response in integration tests, or an altered DOM layout in end-to-end tests.

Self-healing, or auto-healing, is the idea of breaking this cycle. A self-healing system should detect a harmless change and adjust the test automatically. In the Save→Submit case, it would recognise the new label, update the selector, and continue running without human intervention. Tests stay in sync with the application, and the burden of maintenance is reduced.

That may sound simple, but in practice, it isn’t. Let’s look at why.

Why is self-healing tough?

Self-healing is challenging because not every test failure means the same thing. Some failures signal a genuine defect in the application, while others reflect a test that’s simply out of date. To tell the difference, you need to understand what the test is verifying, how it’s written, and whether the application’s new behaviour still aligns with that intent. In short: you need context.

To understand why context matters, let’s look at the kinds of changes that typically break tests:

• Semantic changes: updates to naming or labels of objects, like renaming a button from Save to Submit.
• Structural changes: shifts in layout or placement, like moving a button to another part of a form.
• Behavioural changes: modifications to the logic or flow, like altering when a button can be clicked or what action it performs.

Any one of these changes, or a combination of them, can cause a cascade of test failures even when the application still works as expected. Writing rules to handle all these cases in a clean and flexible way is difficult. Without the contextual understanding that LLMs bring, the result is often rigid with overly complex logic. In some cases, the application ends up being adapted to fit the tests, rather than the tests adapting to the application.

This is where LLMs can help. By interpreting context and meaning more flexibly, they offer a more adaptable way to keep tests aligned with the application as it evolves. With this in mind, let’s quickly examine the advantages and limitations of using LLMs for self-healing.

Self healing with LLMs

Large language models, such as the GPT family, make self-healing easier by reasoning about the intent and context of a test, rather than relying on complex hardcoded rules. At their core, LLMs are trained on vast amounts of text and code, which gives them the ability to “fill in the blanks”. They can infer meaning from partial information and suggest likely continuations. In practice, that means they can often detect what a test is trying to achieve and propose a fix when it breaks.

But LLMs bring their own challenges. The first challenge is context management. With too little context, an AI model doesn’t have enough information to make a good decision. With too much, the output becomes unreliable, slow, or costly. Giving the model just enough detail to be useful without overwhelming it, is a key design consideration when building a reliable self-healing system.

The second challenge is hallucination. A model may suggest selectors, attributes, or code changes that don’t exist. Worse, it has no real way of knowing it’s wrong. Left unchecked, these “fixes” can create tests that pass for the wrong reasons or fail in new ways — the very problem we’re trying to solve.

We can mitigate these challenges by adding guardrails in a well-designed self-healing system. By grounding model suggestions in real application data and carefully managing context, we can take advantage of what LLMs do well while reducing the risk of noisy context or hallucinations. But, the real question is how to bring these ideas together into a system that actually works. That’s what we’ll look at next.

Designing a good self-healing system

A reliable LLM-powered self-healing system rests on three core stages: Context Composition, LLM Evaluation, and Validation. Together these stages form a feedback loop that diagnoses the cause of a test failure, proposes a fix, and validates the fix before accepting it. This loop is designed to address the two main challenges of working with LLMs: managing context effectively and reducing the risk of hallucinations.

The diagram below shows a simplified “happy path” workflow. It illustrates how a harmless application change, such as a renamed button label, is detected and corrected with the help of an LLM. While real-world self-healing systems will require more logical layers (such as handling a true positive or handling incorrect fixes), this example captures the foundational building blocks you‘ll need before developing for more complex cases.

What’s striking about this three-stage loop is its generality. The pattern isn’t limited to self-healing tests; it’s a practical approach to making AI-incorporated systems more reliable. LLMs remain probabilistic, but the loop of context, evaluation, and validation forces their outputs into a process we can check and control.

Practice example

To see how the above flow works in practice, let’s use a simple example with the TodoMVC app and Playwright (but you won’t need to be deeply familiar with Playwright to follow along).

Here’s a test that tries to add a new todo item by filling an input placeholder called What needs to be done:

test('adds a new todo item', async ({ page }) => {
  await page.goto('https://todomvc.com/examples/react/dist/');
  await page.getByPlaceholder('What needs to be done?').fill('Buy milk');
  await page.getByPlaceholder('What needs to be done?').press('Enter');
  await expect(page.getByText('Buy milk')).toBeVisible();
});

Now imagine the placeholder changes to Add new task. The functionality is the same, but the test fails because it was written for the old label. This is where our three-stage loop comes in: Context Composition → LLM Evaluation → Validation.

1. Context Composition

The process begins when a test fails in a continuous integration (CI) pipeline. In this case, Playwright throws an error indicating the label it was looking for could not be found:

Error: Strict mode violation: locator("placeholder=What needs to be done?") resolved to 0 elements.

That failure triggers the self-healing loop. The pipeline captures relevant details about the failure:

{
  "testName": "adds a new todo item",
  "testFile": "tests/todomvc.spec.ts",
  "failingStep": "page.getByPlaceholder('What needs to be done?')",
  "errorMessage": "Strict mode violation: locator('placeholder=What needs to be done?') resolved to 0 elements",
  "errorLocation": 50
  "domSnapshot": "<input placeholder='Add a new task' class='new-todo'> ...",
  "testFile": "test('adds a new todo item', async ({ page }) => { ..."
}

Good context filters out noise so the LLM focuses only on high-value information. This directly addresses the challenge of managing context effectively.

2. LLM Evaluation

Next, the above context is sent to an LLM, such as openAI’s GPT4, via an API. The request will have two parts:

• A system prompt with instructions directing an AI model how to behave:

{
  "role": "system",
  "content": "You are an automated test repair assistant for Playwright (TypeScript). Only adjust selectors - no business logic changes."
}

• A user prompt with the failure context:

{
  "role": "user",
  "content": {
    "context": {
      "testName": "adds a new todo item",
      "failingStep": "page.getByPlaceholder('What needs to be done?')",
      "errorMessage": "Strict mode violation: locator('placeholder=What needs to be done?') resolved to 0 elements",
      "domSnapshot": "<input placeholder='Add a new task' class='new-todo'> ...",
      "testFile": "// tests/todo.spec.ts ..."
     },
    "task": "Propose minimal edits to fix the selector while preserving test intent. Respond as JSON with `rationale` and `changes`."
  }
}

The LLM analyses the context and proposes a targeted fix:

{
  "analysis": "The test expects the placeholder 'What needs to be done?', but the DOM shows 'Add a new task'.",
  "fix": {
    "file": "tests/todo.spec.ts",
    "line": 50,
    "column": 7,
    "oldCode": "await page.getByPlaceholder('What needs to be done?').fill('Buy milk');",
    "newCode": "await page.getByPlaceholder('Add a new task').fill('Buy milk');",
    "reason": "The selector needs to be updated to reflect the new placeholder."
  }
}

3. Validation

Finally, the proposed fix is validated in the CI pipeline:

1. Apply the changes on a temporary branch.
2. Re-run the failing test.
3. Confirm that it now passes.

If validation succeeds, the system raises a pull request for human review:

{
  "title": "Self-healing update - fix placeholder selector in todo.spec.ts",
  "body": "Updated placeholder from 'What needs to be done?' to 'Add a new task' based on DOM snapshot analysis. Validated in CI.",
  "branch": "self-heal/update-placeholder",
  "base": "main"
}

Together, the whole workflow shows how the self-healing loop takes a brittle test, diagnoses the cause, proposes a fix, and ensures the change is safe before it is accepted.

Wrapping up

Thanks for following along. If some of these ideas feel new, they’ll quickly become familiar as you spend more time with AI-driven automation.

We looked at what self-healing is, why it has been so hard to achieve, and how LLMs can change the equation. We sketched out a three-stage design that lays the groundwork for moving into implementation: Context Composition, LLM Evaluation, and Validation.

If you enjoyed this article, please join my substack newsletter — refluent. You’ll also be notified when the next article is published.

Refluent shares sharp insights on quality engineering, performance testing, and the future of intelligent automation.

How to implement self-healing tests with AI was originally published in refluent on Medium, where people are continuing the conversation by highlighting and responding to this story.

Conditional testing, at its core, seems like a straightforward and logical approach to automating tests: write conditions, and based on those conditions, execute specific actions. This method promises a level of adaptability and intelligence in handling various scenarios in automated tests. However, this common test pattern often conceals a tangled web of challenges.

Despite the apparent simplicity of conditional testing they often lead to unpredictable outcomes and complex test maintenance. What’s more, conditional tests can mask these problems until they appear much later, leading to costly and time-consuming fixes.

In this article, we’ll peel back the layers of conditional testing in test automation, revealing why it’s often considered an anti-pattern and how steering clear of it can lead to more resilient and maintainable test suites.

What are conditional tests?

Conditional tests use conditional logic to test diverging pathways in software applications. You will know them by the use of familiar if-else statements:

if x then y else z

In test automation, they are commonly applied to initialise the state of the application, as well as for testing diverging pathways. But as we’ll soon discover, they can also lead to some unintuitive consequences.

Shown below is a basic end-to-end test scenario that uses conditional testing to validate an app’s dark mode. This is an oversimplified example, but it should help us understand some key aspects of conditional testing:

1. Select a random user from the test database.
2. Navigate to the 'Settings' page.
3. Check that 'Dark mode' is enabled:
  A: If dark mode is enabled, skip to step 4.
  B: If dark mode is disabled, toggle on dark mode.
4. Verify dark mode.

The test design is also illustrated below. The decision point to check whether dark mode is enabled is depicted in yellow, and the two logical pathways are depicted in blue and green. This design helps ensure 1) the required software state is initialised for each path to execute predictably, and 2) diverging pathways for either state can be tested.

At first glance, this design seems effective for initialising the application state and testing diverging pathways. However, the very nature of diverging logic can produce some unintuitive issues which we’ll turn to next.

Problems with conditional tests

The diverging logic in conditional testing can be particularly problematic when paired with randomised data to drive the test. In most applications, the shape and distribution of each dataset will likely be different, which can lead to the following:

• Unpredictable test frequency: The number of times each pathway is tested is dependent on the test data distribution, which can vary greatly. For example, if only 1 in 5 users have dark mode disabled as illustrated below, the blue path for enabling dark mode would only execute 20% of the time.
• Flaky edge cases: The unpredictable schedule of tests can naturally lead to flaky tests. If changes in the application result in test failures for any conditional pathways, the test failures might show up as infrequent edge cases, leading to unreliable test results. This is particularly problematic for test failures caused by bugs, which may not be picked up immediately.

While the above issues tend to occur with randomised data, conditional tests can introduce additional layers of complexity even with non-randomised data:

• Synchronisation and awaits: Conditional tests often require additional handling to ensure that the application is in the expected state before proceeding. If not carefully implemented, this can lead to increased test execution time or race conditions that cause intermittent failures and flaky tests.
• Difficult to debug: Due to the tangling logic of multiple pathways and diverging application states, conditional tests tend to be unintuitive and difficult to interpret. This can make debugging tedious and time-consuming.
• Reproducibility: A conditional test may take a different pathway each time it is run. Hence, reproducing errors in a test can require multiple test runs, manual manipulation of application states or mocking of variables to trigger specific test pathways.

These pitfalls can produce unreliable results and excessive maintenance overheads. What’s more, due to the unpredictable test frequencies and edge cases, these problems could appear much later, leading to costly and time-consuming fixes.

Now that we have an understanding of the problems that conditional tests introduce, let’s look at some solutions for navigating conditional tests.

Solving for conditional tests

To avoid the pitfalls of conditional testing we have to write more deterministic tests. In practice, this means designing tests with setups for initialising application states while using linear logic.

Let’s revisit our dark mode test as an example. In the dark mode test, we started by fetching a random user from the database, which left us at the whim of random application states (either dark mode or light mode). This is what the query for the fetch would have looked like:

SELECT user_id FROM users
ORDER BY RANDOM()
LIMIT 1;

Instead of using conditional logic to set the application state via the UI, we can initialise predictable application states right at the start of our tests. We’ll ensure only data for users that have dark mode disabled are fetched at the start of the test (alternatively, we could set the dark mode for any user to true):

SELECT u.user_id, us.notifications_enabled
FROM users u
JOIN user_settings us ON u.user_id = us.user_id
WHERE us.notifications_enabled = true
ORDER BY RAND()
LIMIT 1;

Now that we know that the dark mode toggle will always be off, we can remove the conditional pathways in our script. This allows for a linear test flow:

It is worth noting that we could use other setup methods to initialise the application state early, such as seeding the application with data before each test. In either case, removing conditional statements in test design simplifies test logic, making tests more consistent, more reliable, and easier to maintain. However, it’s important to recognise that conditional statements aren’t inherently problematic and can be useful in some occasions.

When should you use conditional statements?

Of course, conditional statements can be useful in automation tests but their use should be carefully considered. When needed, it’s crucial to implement them in a way that minimises their impact on test predictability and maintainability:

• Use conditional logic sparingly: Reserve conditional logic for scenarios where it’s absolutely necessary. For example, handling different user roles that have access to different features in the application, or for use in polling functions that wait for an application state.
• Isolate conditional logic: When possible, isolate conditional logic to specific parts of the test, particularly the setup phase. This should make it easier to identify and manage conditional logic.
• Document thoroughly: Ensure that any conditional logic within your tests is well-documented to explain why it’s needed and how it affects the test flow. This documentation will be invaluable for maintenance and debugging.

With test automation, and especially UI automation, conditional statements should be the exception, not the rule. The key concern is not to trade off determinism in a test unnecessarily.

Final Thoughts

Conditional testing presents a paradox: while seemingly useful and flexible, it introduces complexity, unpredictability, and maintenance challenges. By recognizing the hidden costs associated with conditional tests, automation testers can take proactive steps to minimize their use, simplifying test logic and improving reliability.

Adopting more deterministic testing practices, such as writing linear tests and initializing the application state early, can significantly reduce the reliance on conditional logic. When conditions are necessary, applying best practices to manage their complexity ensures that tests remain as predictable and maintainable as possible.

Ultimately, the goal of any testing strategy should be to create a suite of tests that are reliable, easy to understand, and straightforward to maintain. By carefully considering the role of conditional logic in your tests, you can achieve a balance between flexibility and predictability, leading to a more reliable testing process.

If you enjoyed this article, please join my substack newsletter — refluent.
We’ll notify you when the next article is published.

Refluent shares sharp insights on quality engineering, performance testing, and the future of intelligent automation.

You can also find me on X and LinkedIn.

References

I found the following resources on conditional testing invaluable:

1. An excellent explainer on the pitfalls of conditional testing on Cypress docs: Conditional Testing.
2. A good short article by Ethan Carlsson: Don’t write Conditional Logic in Tests.

The hidden costs of conditional testing in test automation was originally published in refluent on Medium, where people are continuing the conversation by highlighting and responding to this story.

Small tests can be surprisingly powerful in test automation, especially when compared to traditionally lengthier end-to-end tests. End-to-end tests, while invaluable for real-world test coverage, often has significant downsides. These tests tend to be slow, prone to errors, and often create scaling bottlenecks. To improve test performance, reliability, and scalability, we have to move beyond the constraints of end-to-end test automation.

This is where a shift towards smaller, more targeted tests becomes crucial. Atomic UI tests, which are defined by their focused approach and isolated scope, challenge the constraints of end-to-end tests whilst retaining their benefits. They are a powerful response to end-to-end tests, but before we delve into Atomic tests further, let’s take a closer look at the problems they help solve.

The problems with end-to-end tests

End-to-end tests are useful for simulating real user scenarios, primarily through the UI, to validate the complete flow of an application from start to finish. Testing an application through the UI is invaluable for ensuring all application components and dependencies are working in sync. But automating end-to-end tests is also precisely problematic because of their over-reliance on the UI for testing.

Each additional UI interaction in an end-to-end test increases the test complexity and the likelihood of generating a test error. These errors can stem from various factors, including dynamic page content, timing and synchronisation issues, and unpredictable internal or external dependencies. Put simply, the longer an end-to-end test, the more likely it will break.

The likelihood of test errors due to the UI is also compounded by other common issues in end-to-end tests. We’ll outline some key problems that Atomic UI tests can solve below with the help of an example. Let’s start by considering an online store with some end-to-end test scenarios. We can experience the following problems with these tests:

• Lengthy execution time: End-to-end tests often take a long time to run due to their comprehensive coverage of the entire application flow. In the test below called AddItemToCart, each setup step in the workflow (blue boxes) needs to be executed to perform the key action and validation steps (yellow boxes). The extensive workflow, which includes setup and cleanup steps performed via the UI, prolongs the total execution time of the test.

• Sensitive to changes: End-to-end workflows can be highly sensitive to changes in more than one area of the application, often triggering a chain of failing steps. For example, a test failure in the ‘Select Product’ step below would prevent the key steps (in yellow) from executing. This makes the test needlessly brittle and prone to signalling false negative results.

• Lack of test isolation: Often many end-to-end tests share common setup steps which can trigger multiple test failures. For example, a test failure in the shared ‘Select Product’ step below would not only cause AddItemToCart to fail, but it will also fail unrelated tests as shown. This produces excessive noise, instead of accurately identifying the source of failures.

Summing up, end-to-end tests have a wider surface area for UI interaction, resulting in tests that are slower to execute, more prone to failure, and often with inaccurate results. Though invaluable for confirming real-world functionality, these tests evidently require a more considered balance between coverage and reliability. Let’s unpack how Atomic tests help us achieve that.

Shifting to Atomic UI tests

Atomic UI tests are focused tests that cover a very specific software function. The term Atomic here implies the smallest sensible UI test possible to validate a piece of functionality. Atomic tests are somewhat contrary to end-to-end tests because Atomic tests are designed to be functionality-focused, minimal, and independent.

These three points can be regarded as the guiding principles for designing Atomic UI tests, but the specific techniques to implement them can vary; including the use of session cookies, browser javascript functions, application API calls, and backend/database operations. Let’s expand on the three key terms a little further with the previous AddItemToCart example:

• Focused functionality: Each test is intended to be isolated and limited in scope to assess only the functionality under test. For example, The AddItemToCart test should be limited to the key actions and validations only, while abstracting away any other steps as much as possible. This should make test execution more efficient.

• Minimal setup and teardown: Atomic UI tests use minimised setups and teardowns, particularly by reducing UI interactions to focused areas as much as possible. For instance, we can reduce the UI interactions for AddItemToCart to key steps (in yellow) by abstracting away setup and teardown components with methods and operations that aren’t dependent on the UI (depicted by the grey boxes). The goal is to meaningfully isolate the UI test only to what is necessary and nothing more, resulting in more reliable tests.

• Test isolation: Atomic UI tests are intended to be run independently of each other. This ensures that the failure of one test does not impact the results of another. For example, if we were to split our earlier end-to-end test AddItemToCart, we’d have (at least) two independent tests with minimal external impacts. This also helps us identify the source of an error more efficiently, making test results more accurate.

We can achieve greater test atomicity by abstracting away setup and teardown components with methods that do not depend on the UI. And by reducing the UI automation surface area, we can achieve faster execution, greater reliability, and higher accuracy. These gains also set the foundation for expanding our test coverage in a more scalable manner. Now that we understand the mechanisms and benefits of Atomic testing, let’s quickly tackle the question of how much atomicity is needed for a test to be considered Atomic.

How atomic is atomic?

Traditional end-to-end testing strategies often link comprehensive coverage with extensive, multi-step tests. Atomic testing disrupts this norm by dissecting complex feature sets into smaller units. Each subsequent unit is converted to individual standalone tests.

The process of breaking down extensive tests into concentrated units offers an additional advantage: it compels us to thoughtfully consider each functional component of an application before isolating it. This exercise requires a delicate balance — achieving the minimal scope necessary while ensuring that no critical aspect, such as testing key dependencies, is missed. This is in line with my earlier description of Atomicity as the smallest sensible unit of a test designed to verify a specific piece of functionality.

The definition of what constitutes the ‘smallest sensible unit’ could be debatable. Nicolay Advolodkin suggests that Atomic tests should ideally have just two, or at most three, assertions. Simplifying your tests to this degree might sound challenging, but even small steps towards this goal could enrich your tests with the core characteristics of atomicity: focused functionality, minimalism, and independence. Adopting these can significantly boost your test suite’s overall effectiveness.

Final thoughts

Shifting towards atomicity can have a powerful impact on testing. By embracing the principles of focused functionality, minimal setup, and independence, we’ve seen how Atomic UI tests can potentially transform not just individual tests, but entire automation test suites.

The concept of Atomic testing is not just a theoretical ideal but a powerful practical approach that delivers tangible benefits. Faster execution times, improved accuracy, and simpler maintenance are not just aspirations but achievable realities with Atomic UI tests. Whether it’s a small feature or a complex system, the principles of atomicity are a powerful guide for crafting more efficient, effective, and reliable testing practices.

If you enjoyed this article, please join my substack newsletter — refluent.
We’ll notify you when the next article is published.

Refluent shares sharp insights on quality engineering, performance testing, and the future of intelligent automation.

You can also find me on X and LinkedIn.

References

I found the following resources invaluable for my understanding of atomic testing. If you’d like to learn more about Atomic tests, be sure to check them out:

1. An excellent resource with code examples by Nikolay Advolodkin: Automated Atomic Tests (Definitive Guide)
2. This excellent podcast episode by the inimitable Joe Colantonio with Nikolay Advolodkin: Why Automated Tests Should Be Atomic (The Atomic Punk)

Micro tests, macro impacts: the benefits of switching to Atomic testing was originally published in refluent on Medium, where people are continuing the conversation by highlighting and responding to this story.