Running Gemma-3 OCR Without Cloud Dependencies

If you’ve spent any time building Document AI pipelines, you know the standard playbook: a user uploads an image, you ship it off to a cloud API like AWS Textract or OpenAI’s vision endpoints and you wait for the structured JSON to return to.

Photo by Growtika on Unsplash

For a while, this was fine. But as soon as you start dealing with sensitive data think like healthcare records, legal contracts, or internal financial invoices the “just send it to the cloud” approach hits a massive compliance and privacy roadblock. Not to mention, the API costs at scale can quickly devour your project’s budget.

We needed a way to extract text and semantic meaning from images securely, reliably, and entirely on our own hardware. Enter Gemma-3.

1. The Power of Local Multimodal AI

Google’s Gemma-3 family shifted the landscape by treating visual reasoning as a first-class citizen. Unlike older OCR tools like Tesseract which are great for clean, typed text but stumble on complex layouts or noisy backgrounds Gemma-3 is a true multimodal language model.

It doesn’t just “read” the pixels; it understands the context.

For our pipeline, the Gemma-3 12B model became the sweet spot. It comfortably fits on a single consumer-grade GPU (or even a high-end Mac via unified memory), boasts a massive 128K context window, and handles zero-shot document OCR with the kind of accuracy we used to only see from heavy cloud models.

2. The Stack: Keeping it Hermetic

To build a robust, offline OCR application, you need to minimize friction. We didn’t want to manage a sprawling PyTorch cluster if we didn’t have to. The modern local AI stack is surprisingly lean:

• The Engine: Ollama. It abstracts away the complex model loading and handles quantization natively.
• The Model: gemma3:12b. Powerful enough for complex vision tasks, small enough for local inference.
• The Wrapper: A lightweight Python script using the Ollama client and Pillow for image handling.

By keeping the stack simple, we ensure that the entire system can be containerized, deployed in an air-gapped environment, and scaled laterally across local hardware without pinging a single external server.

3. The Engine Room: Writing the Extraction Logic

The beauty of using a multimodal LLM for OCR is that you aren’t just getting a raw string of text. You can use prompt engineering to dictate exactly how you want that text structured.

Get Dyna Joshy’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Here is the battle-tested snippet we use to pass a document to Gemma-3 and force it to return clean, markdown-formatted text:

import ollama
import os

def extract_text_from_image(image_path):
    """
    Passes an image to local Gemma-3 to extract and format text.
    """
    if not os.path.exists(image_path):
        raise FileNotFoundError(f"Cannot locate image at {image_path}")

    print(f"Processing {image_path} via local Gemma-3...")
    
    # The prompt dictates the structure of our OCR output
    system_prompt = (
        "You are an expert OCR system. Extract all text from the provided image. "
        "Preserve the structural layout, use markdown for headings and lists, "
        "and do not include any conversational filler in your response."
    )

    try:
        response = ollama.chat(
            model='gemma3:12b',
            messages=[{
                'role': 'user',
                'content': system_prompt,
                'images': [image_path]
            }]
        )
        return response['message']['content']
    
    except Exception as e:
        return f"OCR Pipeline Failed: {str(e)}"

# Example Usage:
extracted_markdown = extract_text_from_image('./scanned_invoice_001.jpg')

Note: To run this, you simply need Ollama installed on your machine and to pull the model via the terminal using ollama run gemma3:12b

4. The Results: Speed, Security, and Structure

Moving our OCR pipeline to a local Gemma-3 instance felt like taking the training wheels off.

1. Absolute Privacy: Because the images never leave the local environment, we completely bypassed the usual security audits required for cloud integrations.
2. Semantic Formatting: Older OCR tools would often smash columns of text together. Gemma-3 understands the visual hierarchy of a document, returning perfectly formatted markdown tables and bulleted lists.
3. Cost Efficiency: Our variable monthly API bill dropped to zero. We are now constrained only by our hardware compute limits, not a cloud provider’s pricing tiers.

Running advanced vision AI locally is no longer a theoretical research project; it is a practical, deployable solution. If you are still shipping your sensitive documents to a cloud endpoint just to read the text, it might be time to bring your pipeline home.