Regression Testing in Python: Open-Source Tools for Catching API Changes
Python teams have a funny relationship with regression testing. Ask around and most will tell you they’re covered, because there’s a big pytest suite sitting right there. And it does catch plenty. What it tends to miss is the quieter stuff. Somebody bumps a dependency, or cleans up a module, and suddenly an endpoint returns something slightly different. CI stays green. A week later, a frontend dev pings the channel asking where a field went.
The gap isn’t a lack of testing. It’s that ordinary tests check what someone thought to assert, while regressions tend to show up in everything else. Luckily the Python ecosystem has some genuinely good open-source tools for exactly this problem. They just come at it from very different directions, and picking the right one depends a lot on what kind of changes you’re trying to catch.
What regression testing in Python really needs to catch
Before looking at tools, it helps to be specific. A regression in a Python API rarely crashes anything. A Pydantic upgrade serializes a value slightly differently. A refactor drops an optional field when it’s empty. A query change returns results in a new order. A float that used to come back rounded now has extra decimals.
None of those trip a test that only checks the status code and one or two fields. So whatever tool you pick, the question is the same: how much of the output does it actually watch, and where do its expectations come from?
Open-source regression testing tools, compared
Keploy
Keploy works at the level of the running service rather than inside your test code. It records the real API traffic your Python app handles, using eBPF at the kernel level, so there’s nothing to add to the code, and turns those calls into regression tests with the actual responses as the baseline. Its AI can also generate tests from an OpenAPI spec, a Postman collection or cURL commands, running each one against the live API before keeping it. Calls the app makes to databases and third-party APIs get saved as mocks, which keeps replays steady in CI, and timestamps and IDs are flagged as noise automatically. Because it sits outside the code, it doesn’t matter whether the service is FastAPI, Flask or something else entirely.
Best for: Python APIs that need broad regression coverage quickly, especially ones with real dependencies and not many existing tests.
pytest-regressions
pytest-regressions adds fixtures to pytest that save a test’s output to a file the first time it runs, then compare against that file every run after. There are fixtures for structured data, plain files, numbers and even dataframes. In practice it feels like normal pytest. Inside a test, you grab whatever the endpoint gave back and pass it to one of those fixtures. If anything differs from the saved copy next time, the test goes red. Meant to change it? Regenerate the stored files and commit them with your code.
Best for: pytest-heavy teams who’d rather pin down an entire response than write an assertion for every single field.
syrupy
syrupy is a snapshot testing plugin for pytest. You compare a value against a snapshot fixture, syrupy stores the snapshot on the first run, and later runs fail if the value changes. Updating snapshots after an intended change takes a single command-line flag. It’s lightweight, and the snapshots are readable files that show up nicely in pull request diffs.
Get Sancharini Panda’s stories in your inbox
Join Medium for free to get updates from this writer.
Best for: quick snapshot checks on API responses or serialized objects inside an existing pytest suite.
Tavern
Tavern lets you describe API tests in YAML files: the request to send and what the response should look like. It runs on top of pytest, so the tests sit right alongside everything else and use the same runner. It’s handy for multi-step flows, too, since values from one response can be saved and reused in the next request. Expectations are written by hand, so it catches exactly what you tell it to look for.
Best for: teams that prefer readable, declarative API tests over Python code.
Schemathesis
Schemathesis is written in Python and generates tests from an OpenAPI or GraphQL schema. It throws a wide range of inputs at the API, including the odd ones nobody writes by hand, and checks for crashes and responses that don’t match the schema. It works as a pytest plugin or a command-line tool. Its expectations come from the schema, so it catches contract breaks well, though not changes that still fit the schema.
Best for: Python APIs with a solid OpenAPI spec that need their edge cases explored.
Where each tool’s expectations come from
Lay the tools side by side and the real difference shows up quickly. It’s all about where the “correct” answer comes from.
Snapshot-style tools like pytest-regressions and syrupy take whatever your code returns today and freeze it. That’s powerful, because it watches the whole output, not just the fields someone picked. But the snapshots only cover the calls your tests make.
Tavern and plain pytest assertions depend on what a person wrote down. Precise, but only as broad as the person’s imagination on the day they wrote it.
Schemathesis trusts the schema, which is great for contract breaks and blind to anything the schema allows.
Keploy flips the source around by recording real traffic. Its baseline comes from what clients actually send and what the API actually answered, so it covers the weird requests nobody would have scripted, without anyone writing them up first. And because its AI can also build tests from a spec or a few cURL commands, teams that don’t have much traffic to record yet still get a starting point.
Mixing tools in one Python project
Nothing says you have to pick only one, and plenty of teams don’t. A common pattern pairs recorded traffic for broad coverage of real behavior with snapshot tests around a few critical functions, plus a Schemathesis run on a schedule to keep poking at edge cases. Each catches regressions the others would miss.
Whatever the mix, two habits keep regression testing in Python from turning into busywork. Commit the baselines, whether they’re snapshots, recordings or regression files, so changes show up in pull request reviews. And treat every baseline update as a decision someone made on purpose, not a button clicked to make CI go green.
Conclusion
Regression testing in Python isn’t short on tools. It’s short on clarity about what each tool actually watches. Snapshot plugins freeze outputs, YAML tests check what you specify, schema tools enforce the contract, and traffic recording captures how the API really behaves for its clients. So it mostly comes down to which kind of change keeps getting past you.
Think about the last regression that reached users. Odds are it wasn’t dramatic. A missing field, maybe, or a number with an extra decimal. Those sneak past hand-written assertions all the time, and they’re exactly what a broader baseline is good at noticing while the change is still sitting in a pull request.
