How Large Language Models Are Revolutionizing Receipt Scanning: Inside the ADACE3 Project
Imagine scanning a crumpled supermarket receipt and instantly seeing a neatly organized digital record of your purchases, prices, date, VAT, and store details. What sounds mundane on the surface is actually an incredibly complex task for machines. Traditional approaches to receipt data extraction struggle with the chaotic variety in formats, fonts, and noise levels. But now, a new class of models — large language models (LLMs) — is dramatically improving accuracy and flexibility in this field.
One groundbreaking effort showcasing this advancement is ADACE3, a project developed by the University of Malta in collaboration with local researchers and funded by Malta’s national R&I program. ADACE3 leverages the power of multi-modal LLMs combined with rule-based systems to extract structured data from receipts with nearly human-level accuracy. Let’s explore how this hybrid system works, and why it represents a leap forward for AI in document understanding.
The Challenge of Understanding Receipts
Receipts are notoriously tricky for machines to read. Unlike standardized documents such as forms or passports, receipts vary wildly in design. Some are handwritten or faded, others crammed with tightly packed items, odd abbreviations, or regional tax information.
Early systems used template matching or hardcoded rules — essentially “if this text appears here, label it X.” But these methods fail when confronted with unfamiliar formats. Next came convolutional and recurrent neural networks, which offered more flexibility but still struggled with noisy layouts and visual inconsistencies.
What’s needed is a model that understands both what the text says and where it appears — a model that interprets the document holistically, like a human would.
Enter Multi-Modal Large Language Models
LLMs like OpenAI’s GPT or Google’s BERT are already adept at processing language. But when it comes to documents, text alone isn’t enough. Multi-modal models, such as LayoutLMv3, integrate textual content, visual layout, and image data to form a comprehensive understanding of a document.
In the ADACE3 project, researchers fine-tuned a LayoutLMv3 model specifically for receipt data. They trained it to classify each word in a scanned receipt into one of 12 predefined categories: from the “Name of Shop” and “Date & Time” to “Line Items” and “VAT Info.” This task is known as token classification, where each word is tagged with its correct semantic role.
Combining Machine Learning with Rule-Based Logic
Despite their power, LLMs are not perfect. They can be confused by low-confidence text or ambiguous layouts. To mitigate this, ADACE3 uses a hybrid system: the LLM first makes predictions, and a rule-based model then applies logic-based corrections.
For instance, if a word classified as “Name of Shop” is nestled between words labeled “Address,” and has a low confidence score, the rule-based system might override it. Regex patterns help identify common formats, like dates, telephone numbers, or VAT codes. This two-tier approach closes the performance gap between humans and machines.
The results? The hybrid model achieved a remarkable 98% accuracy across real-world receipts — a benchmark that rivals human annotators.
Building the Dataset: Real and Synthetic
High-quality training data is critical. ADACE3 constructed a unique dataset of about 600 receipts — 450 real-world samples manually annotated and 150 synthetically generated. The synthetic receipts provided perfect ground-truth labels and helped simulate varied layouts and text patterns.
Get Gianluca Valentino’s stories in your inbox
Join Medium for free to get updates from this writer.
Still, real receipts are invaluable. They introduce noise, formatting inconsistencies, and real-world edge cases that synthetic data can’t fully replicate. The most effective training came from a mixture of both types, yielding the most robust model.
OCR: The First Gatekeeper
Before anything can be classified, the text must be read. Optical Character Recognition (OCR) converts images of receipts into machine-readable text. The ADACE3 team evaluated several OCR engines — including PyTesseract, EasyOCR, and the transformer-based DocTR.
DocTR stood out for its resilience to poor lighting and image warping, which are common in mobile-captured receipts. It was faster, more accurate, and better suited for integration with LLMs. The team also experimented with aggregate OCR, running multiple scans with slight variations and merging results using character-level clustering to improve accuracy.
Real-Time Learning and User Feedback
One of the most exciting aspects of ADACE3 is its human-in-the-loop feedback system. Users can upload a receipt, receive an annotated version, and manually correct any mistakes. These corrections are stored in a knowledge base and can periodically used to retrain the model — enabling continuous learning and adaptation to new formats or languages.
This makes the system future-proof. As it encounters more documents from different regions, it gets smarter without requiring developers to handcraft new rules.
Why It Matters
Extracting information from receipts might seem niche, but it has massive implications:
- Expense Tracking & Auditing: Automatically processing hundreds or thousands of receipts for tax, compliance, or reimbursement purposes.
- Retail Analytics: Aggregating item-level data from consumer receipts to gain insights into purchasing behavior.
- Financial Automation: Powering apps that manage budgets, generate reports, or detect fraudulent claims.
- Sustainability & Digitization: Reducing reliance on paper by enabling accurate, searchable digital records.
Challenges and Next Steps
The ADACE3 system excels in English and Maltese receipts. But when applied to documents in Polish, German, or French, performance dropped — highlighting the importance of language and regional adaptation. Fine-tuning the model for other languages is feasible, but adapting rule-based components is more labor-intensive.
Future efforts may involve building language-agnostic models, improving table recognition for line items, and integrating voice or chatbot interfaces to allow users to interact conversationally with their receipts.
Conclusion
ADACE3 demonstrates that combining state-of-the-art large language models with logical rule-based systems can dramatically improve how machines understand messy, real-world documents. It’s a prime example of how AI can transition from the lab to practical applications — bringing order to everyday chaos, one receipt at a time.
