Open-Weight OCR Models Have Finally Caught Up with Cloud Services — Here’s the Proof
A hands-on benchmark of 10 OCR methods on real-world documents.
A large share of documents fed to AI systems today is not machine-readable. PDFs are the most prominent example. We never know in advance whether their full content can be extracted as text, only parts of it can (e.g., due to embedded graphics), or none of it at all (pure scans). To make them machine-readable, we use Optical Character Recognition (OCR).
The quality of the OCR output directly impacts the decision of the AI system. Imagine an insurance company automating claims processing. A scanned form arrives. The OCR engine reads it and reports that the claimant checked “No pre-existing conditions.” Downstream, an AI agent approves the claim automatically. The problem? The claimant actually checked “Yes.” The OCR engine misread a checkbox — and nobody caught it.
This is not a hypothetical. In my consulting work building document-AI pipelines, I see variants of this failure frequently. Checkboxes, handwriting, nested tables, mixed layouts — these are the places where OCR quietly breaks, and where broken OCR becomes broken decisions.
For years, the safe choice was a managed cloud service: Azure Document Intelligence, AWS Textract, or similar. They offered reliable quality behind a simple API, at roughly $0.01 per page. The trade-off was clear: pay for accuracy and speed, or gamble on open-source tools that weren’t quite ready.
That trade-off just changed.
The New Contenders
Two open-weight models released recently — LightOnOCR-2–1B and Chandra OCR 2 — currently lead olmOCR-bench, an open-source OCR benchmark. Both claim production-grade performance and can run entirely on your local computer, eliminating the data-privacy concerns that block many companies from using cloud OCR at all.
But how good are these novel open-weight alternatives, really? Have they become truly competitive with proprietary services?
Answering this question with public benchmarks like olmOCR-bench alone, is challenging: Many major models and services are missing in the comparison, and it is hard to tell from aggregate performance metrics alone whether a method is good enough to be used in ones own production environment. To address this, I ran my own evaluation — less like an academic benchmark, more like a product-magazine road test.
The Test: Four Documents, Ten Methods
I selected four documents that represent the challenges I encounter most often in production:
1. Coca-Cola Volume Growth (scanned form) — Noise artifacts, checkboxes, tables, body text, and a trademark logo. The checkbox recognition alone is a dealbreaker test: get it wrong, and any downstream automation is compromised.
2. NIST Special Database 19 (handwriting sample) — Handwritten digits and text inside a structured form. The challenge is twofold: read the handwriting and align each entry to its correct form label.
3. 2020 Annual Report, Midwest Food Bank (20 pages) — Tables, infographics, alternating one- and two-column layouts. This tests whether a model can maintain coherent structure across a complex, multi-page document.
4. RKI Epidemiologisches Bulletin (German) — Similar layout complexity, but in German — a language with less OCR training data than English, exposing models that overfit to English-centric datasets.
I tested ten OCR methods against these documents: Azure Document Intelligence as the proprietary baseline, the two new open-weight models (LightOnOCR-2–1B and Chandra OCR 2), five methods available through Docling (Tesseract, EasyOCR, RapidOCR, suryaocr, Granite), plus MinerU and marker — all popular open-source projects with 30,000+ GitHub stars each.
Benchmark Result: Open-Weight Models Win on Quality
For each method, I recorded how well it handled the specific challenges posed by each document, arriving at a final rating from poor to excellent. I also measured processing time on a mid-tier GPU (NVIDIA RTX 5090 at $0.89/hour on RunPod), which allowed me to compute the cost per page for each method. The results are summarized in the table below:
For the first time in years, open-weight models delivered the best extraction quality across the board — matching or outperforming Azure Document Intelligence on every test document.
Get Jonas Wacker’s stories in your inbox
Join Medium for free to get updates from this writer.
The differences were starkest on the two hardest tasks.
Checkboxes. On the Coca-Cola form, every method except LightOnOCR and Chandra misidentified which boxes were ticked. Document Intelligence selected the wrong checkbox. Weaker models failed to detect checkboxes entirely. In a production pipeline, either failure leads to the same outcome: a wrong automated decision that no one reviews.
Handwriting with form structure. On the NIST sample, only LightOnOCR and Chandra both read the handwriting correctly and linked each entry to its form label. Document Intelligence read the handwriting but misaligned the labels — meaning the extracted data looked plausible but was silently wrong.
The remaining documents revealed subtler issues. Weaker models confused table entries, lost column alignment, or dropped information from infographics. The two new open-weight models handled all of these consistently well.
The Catch: Speed and Infrastructure
If open-weight models win on quality, Azure Document Intelligence still dominates on speed — by one to two orders of magnitude. In my tests, the cost-per-page advantage of open-weight models was real but narrow for recent models due to a lack of optimized hardware.
Closing this speed gap would require faster GPUs (at least an H100 at $2.69/hour), optimized serving frameworks like vLLM, and careful compilation for your specific CUDA environment. In practice, that means a dedicated person managing inference infrastructure — a significant commitment.
But here’s the thing: many document-processing workflows don’t need real-time speed. Overnight batch jobs, asynchronous pipelines, backfill operations — these are all cases where a cheaper, slower model is the better economic choice. And for organizations that cannot send documents to external cloud services at all, speed is secondary to the fact that these models exist and work.
One more option worth mentioning: if your documents are relatively straightforward — clean scans, standard layouts, no handwriting — RapidOCR delivered acceptable quality at more than 15× lower cost than Document Intelligence. It’s not the best, but it might be good enough for your use case.
What This Means for Production Systems
This benchmark captures a genuine inflection point. For years, the advice was simple: if OCR quality matters, use a cloud service. That advice no longer holds unconditionally.
Here’s how I’d frame the decision today:
Use LightOnOCR-2–1B or Chandra OCR 2 if you need top-tier extraction quality, your documents include challenging elements (forms, handwriting, complex layouts), you have data-privacy constraints that rule out cloud services, and you can tolerate batch-style latency or are willing to invest in GPU infrastructure.
Stick with Azure Document Intelligence if you need real-time or near-real-time throughput, you don’t want to manage inference infrastructure, and your documents are standard enough that the quality gap doesn’t bite you.
Consider RapidOCR if your documents are clean and simple, and you want to minimize cost above all else.
Looking Ahead
It’s worth noting how small these new models are. LightOnOCR-2–1B has only 2 GB of weights — tiny compared to the large language models that dominate headlines. This matters because smaller models are easier to optimize, faster to serve, and cheaper to run. As inference tooling catches up (and it will — the economic incentive is enormous), the speed gap will narrow.
I expect that within a year, deploying a high-quality open-weight OCR model will be as straightforward as spinning up any other microservice. When that happens, the $0.01-per-page cloud OCR model starts looking like an unnecessary expense for a growing number of organizations.
The code framework and detailed evaluation results behind this benchmark are available in my GitHub repository, where all experiments can be replicated through a single command-line interface.
