Local Accurate and Fast OCR with Ollama
The new GLM-OCR model is light, fast, accurate
The Problem with Sending Financial Documents to the Cloud
If you have ever built a document processing pipeline that handles bank statements, you have probably run into the same fork in the road: use a cloud OCR service and pay per page, or use something like Tesseract and spend two weeks wrestling with layout parsing that still misreads half the numbers.
Neither option is great. Cloud APIs — AWS Textract, Google Document AI, Azure Form Recognizer — work well, but they require you to send sensitive financial data to a third party’s infrastructure. For personal finance tools, small business bookkeeping automation, or anything touching a client’s banking records, that creates real privacy exposure. You are now subject to that vendor’s data retention policies, their breach surface, and increasingly, their compliance requirements.
Tesseract, the open-source default, runs locally — but it is a traditional computer vision pipeline. It does not understand document structure. It does not know that a row in a table represents a transaction. It outputs raw text in reading order and leaves you to figure out what belongs where. For a simple scanned letter, that is fine. For a multi-column bank statement with varying layouts across issuers, it is a significant amount of post-processing work that you end up owning forever.
There is a better option now, and it runs entirely on your own machine.
GLM-OCR: A Purpose-Built Document Understanding Model
GLM-OCR is a multimodal vision model released in early 2026 by the team at ZAI. It was built specifically for document understanding — not as a general-purpose vision model that happens to read text, but as a model whose entire architecture is oriented around extracting structured information from documents.
The architecture combines three components. The visual encoder is CogViT, pre-trained on large-scale image-text data, which means it has seen enormous variety in how documents look before it ever touches your input. The middle layer is a lightweight cross-modal connector that uses token downsampling to keep the representation compact without losing detail. The language decoder is GLM-0.5B, a small but capable language model that produces the final text output.
What makes this practically interesting is the parameter count. The entire model is 0.9 billion parameters. That is small enough to run on a machine with a mid-range GPU or even a modern CPU with enough RAM, while still scoring 94.62 on OmniDocBench v1.5 — the top score on that benchmark at the time of writing. It ranks first on table recognition, formula recognition, and general document understanding. Those are exactly the capabilities that matter for financial documents, which are dense with structured tabular data.
The model is released under the MIT License, so there are no commercial use restrictions to navigate.
Why Not Just Use What You Already Know?
Before getting into the implementation, it is worth being honest about the alternatives, because the choice matters.
Tesseract is free and mature, but it is fundamentally a character recognition system, not a document understanding system. It produces raw character sequences. Extracting a transaction table from a bank statement with Tesseract means you are writing a layout parser on top of an OCR engine — and that parser needs to handle every bank’s slightly different column arrangement. That is a maintenance burden you take on permanently.
LLaMA 3.2 Vision and similar general vision models can read documents, but they were not trained to be OCR engines. They are conversational models that can also see. Their output format is inconsistent for structured extraction — they might narrate what they see rather than extract it, or they might drop rows, especially in long tables. Getting reliable JSON out of a 3.2 Vision model for financial documents requires prompt engineering and output validation that adds significant complexity.
Cloud OCR APIs are the most capable option for complex layouts, but they introduce per-page costs that add up fast in any meaningful volume scenario, and they require you to make peace with transmitting financial records off your infrastructure. For a solo developer running a few hundred statements, the cost is manageable. For anything at scale, or for any client-facing use case, the privacy implications require real legal and compliance consideration.
GLM-OCR’s position is specific: it is a small, purpose-trained model that understands document layout, runs locally, and produces structured output without needing extensive prompt coaxing. That combination is what makes it worth using here instead of the alternatives.
Setting up the Model: What the Modelfile Actually Does.
Ollama lets you run GLM-OCR with a single pull command, but the base configuration is generic. For bank statement processing, you want the model to behave deterministically, avoid the repetition loops that can occur with long JSON outputs, and have enough context to handle a full page image plus a verbose transaction list.
The Modelfile in this repo creates a new named model — glm-ocr-optimized — layered on top of the base glm-ocr. Here is the full file:
FROM glm-ocrPARAMETER temperature 0.1
PARAMETER top_k 20
PARAMETER top_p 0.9
PARAMETER repeat_penalty 1.2
PARAMETER repeat_last_n 128PARAMETER num_ctx 8192
PARAMETER num_predict 4096SYSTEM """You are a precise OCR engine specializing in financial documents.
When given an image of a bank statement, extract all visible text and structured data.
Always output a single, valid JSON object. Never repeat fields. Never add commentary.
Include every transaction row without omission."""
Temperature 0.1 keeps the model near-deterministic. OCR is not a creative task. You want the same input to produce the same output every time, and you want the model to commit to what it sees rather than generating plausible-sounding alternatives.
top_k 20 with top_p 0.9 narrows the sampling pool considerably. The model is choosing from a much smaller set of likely next tokens at each step, which makes the output more consistent and less prone to wandering.
repeat_penalty 1.2 with repeat_last_n 128 is the practical fix for a known failure mode: long JSON outputs from vision models can enter repetition loops, especially when the transaction list is long. The repeat penalty makes the model less likely to regenerate text it has already produced, and looking back 128 tokens gives it enough window to catch the pattern before it becomes a loop.
num_ctx 8192 allocates enough context for a full-page image representation plus the JSON output. The default context window on some models is too small for this, which causes silent truncation — you get partial output with no error message.
num_predict 4096 is the output length ceiling. Bank statements with many transactions produce long JSON. The default output limit on some configurations cuts off mid-object, which produces invalid JSON that fails silently. 4096 tokens is enough headroom for dense statements.
The SYSTEM prompt is where the behavioral contract gets set. By telling the model it is a precise OCR engine for financial documents, you are orienting every response toward extraction rather than conversation. “Never add commentary” and “never repeat fields” directly counter two behaviors you would otherwise need to filter out. “Include every transaction row without omission” addresses the tendency of language models to summarize rather than enumerate when lists get long.
Get Mhjorleifsson’s stories in your inbox
Join Medium for free to get updates from this writer.
To build and register the model on your Ollama server:
OLLAMA_HOST=http://your-ollama-host:11434 ollama create glm-ocr-optimized -f ModelfileThe Implementation: Walking Through the Script
The processing script is intentionally minimal. It handles two input types — PDF and image — converts them to base64-encoded JPEG, sends them to Ollama, and streams the JSON response to stdout. All dependencies are declared inline using uv's script metadata format, so there is no requirements file and no virtual environment to manage manually.
Dependencies and Configuration
# /// script
# dependencies = [
# 'ollama',
# 'pillow',
# 'pymupdf',
# ]
# ///MODEL_NAME = "glm-ocr-optimized"
MAX_DIMENSION = 1024
OLLAMA_HOST = "http://10.0.0.123:11434"
OLLAMA_PROMPT = (
"OCR this bank statement image. "
"Extract ALL text into a single valid JSON object with these fields: "
"account_number, account_type, statement_period, opening_balance, closing_balance, "
"and transactions (array of objects with: date, description, debit, credit, balance). "
"Include every transaction row. Output only the JSON object, nothing else."
)
SKIP_OUTPUTS = {"```json", "```", "json", "%"}Three libraries do all the work. ollama is the Python client for the Ollama API. Pillow handles image loading and resizing. PyMuPDF (imported as fitz) renders PDF pages to raster images before they are sent to the model.
MAX_DIMENSION = 1024 means any image edge larger than 1024 pixels gets scaled down proportionally before encoding. This keeps the base64 payload size manageable without materially affecting recognition quality for standard-resolution bank statement scans.
SKIP_OUTPUTS is a filter set for streamed tokens. GLM-OCR occasionally emits markdown code fence markers even when instructed not to. Rather than post-processing the assembled output, these tokens are dropped inline during streaming.
Image Processing
def encode_image(img: Image.Image) -> str:
if max(img.size) > MAX_DIMENSION:
img.thumbnail((MAX_DIMENSION, MAX_DIMENSION), Image.Resampling.LANCZOS)
buffered = BytesIO()
img.save(buffered, format="JPEG", quality=85)
return base64.b64encode(buffered.getvalue()).decode("utf-8")The image gets resized with LANCZOS resampling, which preserves sharpness better than bilinear when downscaling. It is saved as JPEG at quality 85 — good enough to preserve text legibility, small enough to keep network transfer fast for remote Ollama instances.
PDF Processing
def process_pdf(path: Path) -> None:
doc = fitz.open(str(path))
for i, page in enumerate(doc):
pix = page.get_pixmap(dpi=200)
img = Image.frombytes("RGB", (pix.width, pix.height), pix.samples)
run_ocr_on_image(encode_image(img))
doc.close()PDF pages get rendered at 200 DPI. This is a deliberate choice: 72 DPI (screen resolution) produces blurry rasters from PDFs that cause OCR errors on small text. 300 DPI is the traditional archival standard but produces images too large to benefit from at this scale. 200 DPI hits the quality-to-payload-size balance point for bank statement fonts, which are typically 8–10pt.
Sending to Ollama and Streaming
def run_ocr_on_image(image_b64: str) -> None:
stream = ollama.generate(
model=MODEL_NAME,
prompt=OLLAMA_PROMPT,
images=[image_b64],
stream=True,
)
for chunk in stream:
fragment = chunk["response"].strip()
if fragment not in SKIP_OUTPUTS:
print(chunk["response"], end="", flush=True)Note the use of ollama.generate rather than ollama.chat. This is not arbitrary. GLM-OCR on Ollama uses a custom renderer and parser that are only activated through the /api/generate endpoint. The OpenAI-compatible /api/chat endpoint does not invoke them, which means vision inputs are handled incorrectly. The Ollama Python client's generate method maps to the right endpoint.
Running It
# Install uv if you haven't already
curl -LsSf https://astral.sh/uv/install.sh | sh# Run against a PDF (dependencies install automatically)
uv run ocr_statements.py /path/to/statement.pdf# Or pass an image
uv run ocr_statements.py /path/to/statement.jpg
No pip install, no venv activation, no dependency conflicts. uv reads the inline metadata and handles everything.
What the Output Looks Like
For a standard business checking statement, the streamed output looks like this:
{
"account_number": "400005513901",
"account_type": "Premier Business Checking",
"statement_period": "September 1, 2023 Thru September 30, 2023",
"opening_balance": 12345.67,
"closing_balance": 9876.54,
"transactions": [
{
"date": "09/01/2023",
"description": "ACH DEPOSIT",
"debit": null,
"credit": 500.00,
"balance": 12845.67
},
{
"date": "09/03/2023",
"description": "ONLINE TRANSFER",
"debit": 1200.00,
"credit": null,
"balance": 11645.67
}
]
}This is ready to pipe into any downstream process — a pandas DataFrame, a PostgreSQL insert, a reconciliation script, a ledger import. Because the schema is fixed in the prompt, you can parse it with a single json.loads() call without needing a flexible extractor.
The Bigger Picture
What this repo demonstrates is less about GLM-OCR specifically and more about a pattern that is now practical at the local level: take a purpose-trained small model, configure it with tight sampling parameters and a behavioral system prompt, and you get extraction quality that was previously only available from cloud APIs — at zero marginal cost, with no data leaving your network.
The combination of uv for zero-friction dependency management, Ollama for local model serving, and GLM-OCR's document-native architecture means you can stand this up on a reasonably capable machine in under thirty minutes. The Modelfile is the configuration layer that makes the model reliable for this specific task rather than a general-purpose one.
If you are building financial automation tooling, document ingestion pipelines, or anything that touches scanned or PDF-format records, this is worth having in your toolkit. The code is at github.com/rapidarchitect/ollama-glm-ocr-bankstatements.
