Surya and EasyOCR: Vision Models and Data Extraction in Financial Documents
Motivation
Automation tasks such as audits, compliance checks, and financial analysis often require extracting information from documents. Typically, this involves transforming documents into images, followed by layout detection, and OCR recognition.
Financial documents pose unique challenges:
- High number of charts and tables.
- Symbols are highly relevant. Think of currencies, $, £; magnitudes >,<, ~; negative numbers, -, (, ).
- Irregular formats. Documents in the same category might share similar sections, but the format, positioning, details and font sizes can be highly diverse.
In this post I explore and test two models that can be used for OCR in financial documents: EasyOCR and Surya.
TL;DR
- Overview of financial document diversity.
- Descriptions of Surya and EasyOCR models.
- Evaluation of both models on text detection and recognition.
- Conclusions.
Financial Document Examples
Layout, colors, font sizes are usually quite diverse in financial documents.
Documents like 10-K reports from different companies (e.g., Starbucks, Nike, JPMorgan) can vary significantly in format, including differences in positioning, color schemes, and table structures, making extraction challenging.
Data Extraction Task
One of the most common tasks from financial documents is extracting information. As we have seen the non-standardized formatting makes the problem challenging. Usually, the pipeline for data extraction follows some of the following steps:
- Layout detection: to get the structure of the page/document.
- OCR: to go from pixels in the image to text.
- Correction: since OCR usually introduces errors, there can be a step to fix mistakes.
- Extraction: to obtain the requested information.
LVMs vs OCR models
Large Vision Models (LVMs) like GPT-4 and Qwen2 bypass the need for most of the steps in the above pipeline. This is, they take images as input and work directly from them.
However, one of the main limitations of LVMs is the computational resources needed, since these models range from a few billion parameters to hundreds of billions, making their usage quite expensive. In contrast, OCR models can be much smaller and can fit in smaller GPUs or might be run even on CPUs.
Deciding which to use depends on the task, available resources, and data extraction needs.
Models to Test: Surya and EasyOCR
EasyOCR
EasyOCR is a ready to use OCR model for more than 80 languages which we’ll test as baseline. According to their documentation, for detection EasyOCR uses the CRAFT model (Character Region Awareness For Text detection) which is based on convolutional neural networks. For recognition, EasyOCR uses Convolutional Recurrent Neural Network (CRNN), a combination of LSTMs and CNNs.
The total number of parameters is 27M, which is considerably smaller than LVMs.
Surya
Surya is a recent OCR toolkit that performs text detection, OCR, layout analysis and reading order detection in more than 90 languages. According to the documentation, the detection model is based on an EfficientViT model and the recognition model is based on Donut.
This model aims to improve Vision Transformer model performance on hardware. It proposes removing the softmax operation in transformers with a lightweight ReLU linear attention. This is, given the generalized form of softmax attention as:
we can recover the original attention using the following similarity
the authors, propose to modify with ReLU linear attention as follows:
This modification gives improvements on hardware costs. However, ReLU linear attention is not good at handling concentrated attention maps which will decrease the quality of the model.
To mitigate this issue, the architecture is modified by adding a convolution with each FFN layer.
Document understanding transformer (Donut) is a model for Visual Document Understanding, and was built with the idea of bypassing the OCR step, this means, going directly from an image to a structured output (JSON).
Get Pablo Hernandez Leal’s stories in your inbox
Join Medium for free to get updates from this writer.
The three main elements of Donut are:
- Transformer encoder. Takes as input an image and split into patches to generate embeddings. It uses Swin Transformer as encoder.
- Decoder. Takes as input the transformer encoder output and generates a token sequence. It uses BART as the decoder architecture.
- Output Conversion. The output token sequence is converted to JSON by adding two special tokens [START ∗] and [END ∗], where ∗ indicates each field to extract.
Testing on Financial Documents
To evaluate the models we will use a diverse set of financial documents:
- Alibaba 10-K
- Royal Bank of Canada Annual Report
- Bank statement from JPMorgan
- A scanned Correspondent Banking Due Diligence Questionnaire (CBDDQ)
We select some pages from these PDFs and converted to images using pdf2image.
Text Detection
Surya
For Surya we can generate predictions with the following code:
from PIL import Image
from surya.ocr import run_ocr
from surya.model.detection.model import load_model as load_det_model, load_processor as load_det_processor
from surya.model.recognition.model import load_model as load_rec_model
from surya.model.recognition.processor import load_processor as load_rec_processor
IMAGE_PATH = ''
langs = ["en"] # Replace with your languages - optional but recommended
det_processor, det_model = load_det_processor(), load_det_model()
rec_model, rec_processor = load_rec_model(), load_rec_processor()
preds=[]
i = 0
p = IMAGE_PATH + 'pg_'+str(i) + '.png' #file name of the images
image = Image.open(p)
predictions = run_ocr([image], [langs], det_model, det_processor, rec_model, rec_processor)
preds.append(predictions)For each text line detected Surya provides the bounding box, text, and confidence score. We can use the bounding box to visually highlight the detected text on top of the image.
from PIL import ImageDraw
p = IMAGE_PATH + 'pg_'+str(i) + '.png'
page = 0
objs = []
for t in preds[page][0].text_lines:
objs.append([t.polygon,t.confidence,t.text,t.bbox])
image = Image.open(p)
for obj in objs:
# draw bbox on image
draw = ImageDraw.Draw(image)
draw.line(obj[3], fill='red', width=3)
display(image)EasyOCR
For EasyOCR, the code changes slightly, for OCR predictions we do:
import easyocr
reader = easyocr.Reader(['en']) # this needs to run only once to load the model into memory
IMAGE_PATH = ''
preds = []
i = 0
p = IMAGE_PATH + 'pg_'+str(i) + '.png'
result = reader.readtext(p)
preds.append(result)Since EasyOCR does not provide the bounding box we need an additional function to transform the data, after that we can plot a red line to highlight the detected text.
#transform polygon coordinates objs[0] to bounding box
def polygon_to_bbox(polygon):
# Unpack the polygon coordinates
x1, y1 = polygon[0]
x2, y2 = polygon[1]
x3, y3 = polygon[2]
x4, y4 = polygon[3]
# Get the list of x and y coordinates
x_coords = [x1, x2, x3, x4]
y_coords = [y1, y2, y3, y4]
# Find the minimum and maximum coordinates
x1_bbox = min(x_coords)
y1_bbox = min(y_coords)
x2_bbox = max(x_coords)
y2_bbox = max(y_coords)
# Return the bounding box as (x1, y1, x2, y2)
return (x1_bbox, y1_bbox, x2_bbox, y2_bbox)
for page in range(0, 1):
objs = []
for p,t,c in preds[page]:
# print(t.bbox)
objs.append([p,t,c])
#show bbox of all elements in objs
p = IMAGE_PATH + 'pg_0.png'
image = Image.open(p)
for obj in objs:
# draw bbox on image
draw = ImageDraw.Draw(image)
draw.line(polygon_to_bbox(obj[0]), fill='red', width=3)
display(image)Examples
We compare side by side, the results from EasyOCR and Surya on some selected pages from the diverse documents.
First, on the RBC annual report. Both models do a very good work detecting text in the images, even with different color schemes and font sizes. It is hard to spot but EasyOCR missed a ~ symbol. Surya missed the “$22.1 billion” inside the circle (bottom right page).
When testing in the Alibaba 10-K we note that EasyOCR can handle diverse font sizes detecting all the text in the image. In contrast, Surya does not capture the text “102 years”. My assumption of why this happens it’s because of this comment on Surya repo: “ The text detection model has trained itself to ignore advertisements”.
When looking at the page with table Surya makes an excellent work even detecting the small “-” in the cells.
On the bank statement, EasyOCR misses 3 numbers in the column Instances. In contrast, Surya detects correctly all the text.
Lastly, in the scanned CBDDQ, EasyOCR misses most of the text in the first column of the table. Surya, is able to detect correctly most of the text. The logo in the upper left is ignored by Surya.
Text Recognition
To visually show the recognition results, we will “recreate” the document with the coordinates of the bounding box and the recognized text.
from PIL import Image, ImageDraw,ImageFont
p = IMAGE_PATH + 'pg_0.png'
font = ImageFont.truetype("/ubuntu-mono.ttf",20) # specify a font path to change size
image = Image.open(p)
# create a blank image with the same size
blank_image = Image.new('RGB', image.size, (255, 255, 255))
for obj in objs:
# draw OCR text
draw = ImageDraw.Draw(blank_image)
draw.text((polygon_to_bbox(obj[0])[0],polygon_to_bbox(obj[0])[1]), obj[1], fill='red', font=font)
display(blank_image)We start with one of the pages from the annual report from RBC. To make it easier I highlighted the differences from EasyOCR and Surya in black. Note that sometimes EasyOCR breaks the text into different bounding boxes even when it should remain on the same line.
In the table from Alibaba 10-K, EasyOCR did quite well correctly recognizing the most of the text. Surya was able to detect everything, however, in the recognition, there are some cells where the “-” was incorrectly recognized as “l”.
We have seen good results of the models in the digital documents, let’s compare the results with a low quality document like a scan.
In this test I was able to found more than 30 words with mistakes made by the EasyOCR model.
Surya did a very good job only missing the logo but correctly recognizing the text in the document.
Conclusion
| Feature | EasyOCR | Surya |
|--------------------|--------------------|---------------------------------|
| Detection model | CRAFT model | EfficientViT |
| Recognition model | CRNN | Donut |
| Language support | 80+ languages | 90+ languages |
| Layout analysis | Limited | Yes |
| Ideal use case | Simpler documents | Complex, multi-layout documents |EasyOCR was able to detect most of the text, however, it tends to break the text into different bounding boxes even when it should be on the same line. EasyOCR struggled with a scanned document showing many errors in the recognition step.
Surya showed consistent results in detection and recognition across different formats and even with scanned documents. Surya tends to ignore large texts that might look like logos, maybe because the model was trained to ignore advertisements.
This area of Vision models, OCR, and document understanding is an active research area where many models are being developed for specific needs, more examples below:
- DocOWL is a Multimodal Large Language Model that focuses on understanding text-rich images from 5 categories: documents, webpages, tables, charts, and natural images. It has around 8B parameters.
- ColPali is a Retrieval model that takes images without needing additional OCR or layout detection. Here’s a post testing ColPali in financial documents.
- OCRonos is a correction model of badly digitized texts.
- TFT-ID is a Table/Figure/Text detection model for academic papers.
