Member-only story
Reproducibility Isn’t a Specification Problem. It’s a Verification Problem.
Scientific Python solved environment specification. Verification still runs on volunteers. Three production agent patterns for the last mile.
In 2019, researchers pulled 1.4 million Jupyter notebooks off GitHub and tried to run them. About 24% executed without errors. Roughly 4% produced the same results the authors had committed. The top causes of death: missing dependencies, hidden notebook state, and inaccessible data.
That study is old enough to drive in some countries, so you’d hope things improved. They didn’t. A follow-up on biomedical notebooks from PubMed Central found a similar wasteland — only a sliver re-executed, and a sliver of that sliver reproduced. A 2026 study of scientific notebooks found that even with manual troubleshooting, 2 of 19 reproduced.
Meanwhile, the most credible human fix we have CODECHECK, which pairs papers with an independent reviewer who actually reruns the code has issued roughly 116 certificates in six years. Not a typo. Six years of operation by dedicated, skilled people, and the throughput is about 19 papers a year.
Here is the thesis: we spent a decade treating reproducibility as a specification problem. It was always a verification problem. Specification got world-class tooling conda-lock, pixi, uv…
