Test Observability, For Free

You already generate the data a $20k/year tool would sell you back. Here’s how to stop throwing it away.

Every test automation team hits the same wall eventually.

You’ve got a solid Selenium/TestNG (or Playwright, or Cypress) suite. It runs in CI. It produces a report. And the report is useless the moment you close the tab, because it only ever answers one question: how did this one run go?

The questions that actually matter live one level up:

• Is module X getting flakier over the last two weeks, or was that just a bad Tuesday?
• Which failure category is eating the most engineering time — real bugs, or infrastructure noise?
• Are our slowest 10 tests the same 10 tests they were last sprint?
• Did the API layer get slower this release, or does it just feel that way?

None of that lives in a single HTML report. It lives across runs. And the moment a team notices this gap, the reflex is to reach for a vendor: ReportPortal, Allure TestOps, a hosted analytics SaaS. Which is a fine choice — but it’s worth knowing what you’re actually buying, because most of it is a rollup of data your framework is already producing and discarding after every run.

I found this out by building the free version first.

The reframe

The insight that unlocks this isn’t clever architecture. It’s a one-sentence reframe:

A test report is a snapshot. Test observability is what you get when you stop deleting the snapshots.

That’s it. If every run already writes a structured result somewhere, and you keep those files instead of overwriting them, you have a time series. A time series of pass/fail data is a dashboard away from being a trend chart, a flaky-test detector, and a failure-pattern analyzer — all without a database, a backend service, or a monthly invoice.

The rest of this is just plumbing that respects that idea.

The four layers

Strip away the Selenium-specific plumbing and any test framework’s reporting pipeline decomposes into four layers that barely know about each other:

COLLECTION        →  something taps into the test lifecycle and records events
AGGREGATION       →  events get folded into one in-memory model, per run
PERSISTENCE       →  the model gets written to disk as a durable snapshot
PRESENTATION      →  a dashboard reads every snapshot on disk, not just today's

The trick that turns “reporting” into “observability” happens entirely in the last layer, and it’s almost embarrassingly simple: the dashboard-builder doesn’t read one file. It globs the whole snapshot directory.

Everything before that is just making sure the snapshot is trustworthy. Let’s go layer by layer.

1. Collection — tap the lifecycle you already have

Every test framework worth using exposes lifecycle hooks: onTestStart, onTestSuccess, onTestFailure, beforeEach, afterEach. You don't need custom instrumentation — you need one listener that's a good citizen of the events already firing.

public class ObservabilityListener implements ITestListener, ISuiteListener {

    @Override
    public void onTestStart(ITestResult result) {
        TestRunModel model = TestRunModel.from(result);   // module, tags, id, etc.
        ThreadLocalTestContext.init(model);                // one model per thread
    }
    @Override
    public void onTestSuccess(ITestResult result) {
        ThreadLocalTestContext.get().markStatus(PASSED);
        RunAggregator.getInstance().register(ThreadLocalTestContext.get());
        ThreadLocalTestContext.clear();
    }
    @Override
    public void onTestFailure(ITestResult result) {
        var ctx = ThreadLocalTestContext.get();
        ctx.markStatus(FAILED);
        ctx.setFailureCategory(FailureClassifier.classify(result.getThrowable()));
        RunAggregator.getInstance().register(ctx);
        ThreadLocalTestContext.clear();
    }
}

Two design choices here matter more than they look:

Thread-local, not shared state. If your suite runs in parallel — and it should — a shared mutable “current test” object is a race condition waiting for a Tuesday demo to fail. One model per thread, cleared at the end of every test, is non-negotiable.

Status is never assumed. Notice there’s no default. If a listener callback never fires for some reason, the test’s status should render as UNKNOWN, not silently inherit PASSED. A reporting pipeline that fails safe into a false positive is worse than no reporting at all — it's actively lying to you with a green checkmark. Make the failure mode loud.

2. Aggregation — one model, recomputed, not incremented

Every test that finishes needs to land somewhere shared. The naive version is a counter you increment: passedCount++. Don't do this. In a parallel suite, incrementing counters from multiple threads is a subtle bug generator, and worse, it drifts silently — a miscount doesn't crash, it just quietly under-reports.

The sturdier pattern: keep the individual test results in a ConcurrentHashMap, and compute the summary fresh, every time it's requested, by iterating the map.

public class RunAggregator {
    private static final RunAggregator INSTANCE = new RunAggregator();
    private final Map<String, TestRunModel> tests = new ConcurrentHashMap<>();

    public void register(TestRunModel test) {
        tests.put(test.getId(), test);
    }
    public RunSummary summary() {
        // recomputed from the live map on every call — never trusted from a counter
        long passed = tests.values().stream().filter(TestRunModel::isPassed).count();
        long failed = tests.values().stream().filter(TestRunModel::isFailed).count();
        long skipped = tests.values().stream().filter(TestRunModel::isSkipped).count();
        return new RunSummary(tests.size(), passed, failed, skipped);
    }
}

It’s a little more CPU per call. It’s completely immune to a whole category of concurrency bugs. That trade is almost always worth it at test-suite scale (hundreds to low thousands of tests) — re-evaluate only if you’re aggregating tens of thousands of results per run.

3. Persistence — snapshot atomically, or don’t bother

This is the layer people skip, and it’s the one that actually earns you the word “observability.” Two rules:

Get Kritagya’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Write atomically. Never write directly to the file a reader might be looking at. Write to a temp file, then rename. A rename is atomic on every OS that matters; a partial write is a corrupted JSON file that breaks your dashboard the one time you need it most — usually right after a CI box got killed.

public void writeSnapshot(RunModel run) {
    Path tmp = snapshotDir.resolve(run.getRunId() + ".json.tmp");
    Path finalPath = snapshotDir.resolve(run.getRunId() + ".json");

    objectMapper.writeValue(tmp.toFile(), run);
    Files.move(tmp, finalPath, StandardCopyOption.ATOMIC_MOVE,
                                StandardCopyOption.REPLACE_EXISTING);
}

Snapshot on a timer and on every event, not just at the end. A run that finishes cleanly should have near-real-time JSON on disk the whole way through — flush after every test result, not only at suite completion. Then add a periodic flush (every 10 seconds is a reasonable default) as a safety net, and a JVM shutdown hook that forces one last write. The combination means a process that gets killed mid-suite — OOM, a CI timeout, someone closing their laptop — still leaves you the last-known-good state instead of nothing.

Runtime.getRuntime().addShutdownHook(new Thread(() -> {
    aggregator.summary().markHalted();     // be honest: this run did NOT finish
    writer.writeSnapshot(aggregator.getRunModel());
}));

That markHalted() line matters as much as the atomic write. A snapshot should tell you when it's incomplete. A dashboard that can't distinguish "this run finished and everything passed" from "this run got killed before most tests even started" will eventually cost someone a very confusing debugging session.

4. Presentation — the one line that makes it observability

Here’s the actual unlock, and it’s almost anticlimactic:

public class InsightsGenerator {

    public void generateDashboard(Path snapshotDir, Path outputHtml) {
        List<RunModel> allRuns = Files.list(snapshotDir)
            .filter(p -> p.toString().endsWith(".json"))
            .map(this::parseSnapshot)          // <-- reads EVERY run on disk
            .collect(Collectors.toList());
        renderDashboard(allRuns, outputHtml);
    }
}

That’s the whole trick. Not “read the JSON file this run just wrote.” Read every JSON file that’s ever been written, and aggregate across all of them. One extra line of code is the entire difference between a single-run report and a multi-run observability dashboard. Everything downstream — trend lines, flaky-test detection, failure-pattern frequency — is just different views over that same list of historical runs.

From there, a genuinely useful dashboard needs a handful of views, and they’re all cheap once you have the historical list:

• Stat tiles + a pass/fail/skip donut for the latest run — the “is anything on fire right now” glance.
• A duration trend line across runs — this is the one people underrate. A test that silently goes from 4s to 40s over three weeks is a leading indicator of a flaky test or a real performance regression, and nobody notices until it’s a timeout. A line chart over historical durations catches it for free.
• Breakdowns by module / severity / category — stacked bars, one per dimension, built off whatever metadata tag your team already puts on tests (you’re already tagging tests as smoke/regression/critical for CI filtering — reuse the same tags here instead of inventing new ones).
• Failure root-cause distribution — a pie chart over failure categories, not raw stack traces. This only works if collection classified the failure at capture time (see below).
• A halted-run banner — if any scanned snapshot has status: HALTED, say so loudly at the top. Silence here is how a team stops trusting the dashboard.

The failure classifier: cheap, and worth it

One layer that punches well above its size: a pattern table that turns a raw exception into a category.

public enum FailureCategory {
    ELEMENT_NOT_FOUND, STALE_ELEMENT, WAIT_TIMEOUT,
    ASSERTION_FAILURE, NETWORK_ISSUE, CONNECTION_REFUSED,
    AUTH_EXPIRED, INFRASTRUCTURE_FAILURE, UNKNOWN
}

public class FailureClassifier {
    public static FailureCategory classify(Throwable t) {
        String msg = t == null ? "" : String.valueOf(t.getMessage());
        if (t instanceof StaleElementReferenceException) return STALE_ELEMENT;
        if (msg.contains("Connection refused"))          return CONNECTION_REFUSED;
        if (msg.contains("Timed out waiting"))            return WAIT_TIMEOUT;
        if (t instanceof AssertionError)                   return ASSERTION_FAILURE;
        // ...a dozen more patterns, tuned to what actually shows up in YOUR suite
        return UNKNOWN;
    }
}

This is the difference between a dashboard that says “37 tests failed” and one that says “37 tests failed: 22 were a shared staging outage, 9 were a real regression in checkout, 6 are flaky waits.” The first number is noise. The second is a Monday-morning triage list. And it costs about an hour to write a pattern table tuned to your own suite’s actual exception shapes — this is not a machine-learning problem, it’s a switch statement that gets more accurate every time you add a new pattern you recognize.

The bug I’d tell you to avoid before you write it

I want to be honest about a mistake in an earlier version of this design, because it’s the kind of thing that looks completely reasonable until you measure it.

The original model tried to correlate captured network calls to individual test steps — “which API calls happened during step 3.” The implementation took a shortcut: instead of real timestamp-windowing, it attached the test’s entire list of API calls to every single step.

It worked, in the sense that the data was technically present everywhere someone might look for it. It was also a silent multiplier: for a test with 9 steps and 6 API calls, that’s 54 stored objects instead of 6. Scaled across a few hundred tests, that’s the difference between a report file measured in megabytes and one measured in hundreds of megabytes — and nobody notices until the dashboard starts taking twenty seconds to load and someone finally opens the JSON to see why.

The fix wasn’t clever — it was store the list once, at the level it actually belongs to (the test, not the step), and derive any step-level view at render time instead of persisting the duplication. Rule of thumb that generalizes past this specific bug: if you’re about to attach the same collection to N different parents “for convenience,” you’re not correlating data, you’re multiplying it. Either do the real correlation (timestamps, windows) or store it once and join at read time.

What this buys you, and what it doesn’t

To be fair to the vendors: this pattern gets you 80% of what a paid test-observability tool sells, for the cost of a few hundred lines of code you already half-have in your reporting listener. It does not get you:

• Team-wide auth, RBAC, and shareable hosted links out of the box
• Native CI-system integrations beyond “read a folder of JSON”
• A support contract when something breaks at 2am
• Long-term storage/retention policies, backups, and query performance once you’re past thousands of runs

If any of those are the actual bottleneck for your team, buy the tool — that’s a legitimate trade of engineering time for a solved problem. But if what’s stopping you from having any trend visibility is “we don’t have budget approved yet,” you don’t need to wait. You need four files: a listener, an aggregator, an atomic writer, and an HTML generator that globs a directory instead of reading one file.

Try it this weekend

If you want the smallest possible version of this to exist in your suite by Monday:

1. Find your existing report listener. You have one — Extent, a custom TestNG listener, a Playwright reporter. It already knows when a test starts, passes, and fails.
2. Add one write: serialize a minimal {testId, status, durationMs, module, failureCategory} per test to a JSON file per run, named by timestamp, in a folder you don't .gitignore.
3. Don’t delete old ones. That’s the whole “strategy” — retention is the feature.
4. Write forty lines of HTML + Chart.js that globs the folder and plots pass rate and duration over the last N runs.

You’ll have something worse than ReportPortal and better than nothing by the end of a Saturday. Keep iterating on it only as fast as it actually pays for itself — that’s the same discipline you’d apply to any other piece of internal tooling, and test reporting has no business being exempt from it.

The patterns above are generalized from a production Selenium/TestNG framework — details have been abstracted to focus on the architecture rather than any specific implementation.