AI Agent Test Data: What to Fake and What to Keep Real

Fake the inputs that make a test safe and repeatable, while keeping the behavior under examination real. If an agent is supposed to save a report, a test that replaces the save operation with a success message can verify the agent’s response to that message. It cannot establish that a report was stored.

Use this reference to ask what each passing test demonstrates. “Real” means that the relevant implementation runs in a controlled environment; it does not require customer data or live commercial effects.

A mock is a controlled replacement for part of the system. A fixture is a prepared input or starting state. Both are useful. The table identifies what must remain exercised for the named test to mean anything, then names the claim that still needs separate evidence. Rows are choices for test design, not a ranking from bad to good.

Playwright’s testing guidance recommends user-visible checks, isolated tests and controlled data. It also recommends mocking third-party dependencies. These are useful ways to test the application you own. They do not turn a substituted dependency into evidence of a live integration.

In Playwright’s API examples (read September 9, 2026), route.fulfill returns a chosen response without contacting the intercepted API. The contrasting example calls route.fetch, changes the returned data and then fulfills the request with the patched response. The first tests how the page handles a replacement; the second also exercises the API request, while the displayed data is still modified.

Label that difference in the result: “page behavior checked against a replacement response” or “API response obtained and patched before display.” Neither proves an unmodified live workflow. Keep an independent destination check for any write or delivery claim.

The delivery-state reference helps describe the resulting boundary. The file-output acceptance reference supplies the checks for the actual copy. A passing local test should not silently acquire a claim about recipient access.

Synthetic data becomes misleading when it removes the difficulty the tool must handle. A parser expected to accept names with accents should not be tested only with short plain-English names. A calculation that handles missing values needs deliberate missing cases, not a spreadsheet where every cell is populated.

Preserve relationships as well as individual values. If two records are meant to refer to the same entity, give them a consistent synthetic identifier. If a duplicate is intentional, label why it is present. Otherwise a test can pass because the fixture accidentally avoids the business rule under examination.

Our own-traffic evaluation guide addresses whether tasks represent intended use. This reference addresses what gets replaced within those tasks. Representative examples still need clear substitution boundaries; realistic-looking records alone do not make an integration real.

For each test, write: claim, fixture, substituted boundaries, implementation exercised, observed result and remaining check. Include the test version so someone can tell whether a later run used the same setup. This note is more useful than a single “all tests passed” message when the reader must decide whether work is ready.

Do not invent a universal minimum number of examples. Choose cases from the behavior you promise and the failures you need to distinguish. A controlled expiry response can examine recovery logic; an authorized expired credential in a test environment examines whether the real boundary produces the expected denial. Neither alone estimates how often expiry happens.

Keep expected outcomes separate from the agent’s own assertions. For a transformation, use an independently specified expected value or a property the output must satisfy. For a write, inspect the destination through a separate read path where practical. The test should have a way to reject a plausible explanation of a result that never occurred.