QUANTRIUM GUIDES
Extracting text from an image using docTR OCR
Converting coordinate systems for OCR models
Introduction
In one of my previous articles, I explained how to perform detection and recognition of text separately in an image/document using multiple OCR models. The article is found here:-
I used easyOCR for text detection and tested the recognition of text using easyOCR, PPOCR, pytesseract, and MMOCR Robust Scanner models. One of the major benefits that could be found in the above-mentioned OCR models is that they follow a universal coordinate system (which I would like to call the graphical coordinate system). But that’s not the case all the time.
Some OCR models follow a different coordinate system. In such cases, we need to convert it to a universal coordinate system to use the coordinates in a different recognition model. One such model is the docTR OCR model. It is based on Tensorflow 2 and Pytorch and is very useful for extracting data from document/image files. It has pre-trained detection and recognition models and also provides support to build custom models based on the application or dataset used.
In this article, I have explained how to perform text detection and recognition with docTR OCR using pre-trained models, following which I have explained the coordinate system followed by docTR OCR to detect text blocks. Then, I describe the procedure to convert the coordinate system from the docTR OCR geometry system to the graphical coordinate system, which could then be used to perform recognition on a different OCR recognition model if needed. So, let’s get started!
Performing text detection and recognition using docTR OCR
To demonstrate the OCR process using docTR OCR, I have used the following image document which I have generated manually for this purpose (in fact, I had used a similar image document in my previous article too):-
First, let’s install the required packages:
!pip install python-doctr
!pip install "python-doctr[tf]"
!pip install "python-doctr[torch]"Next, import the packages and define the model:
from doctr.io import DocumentFile
from doctr.models import ocr_predictor
model = ocr_predictor(det_arch = 'db_resnet50',
reco_arch = 'crnn_vgg16_bn',
pretrained = True
)However, there are different detection and recognition architectures and you can find the detailed information in the docTR OCR documentation.
Here is the link to the docTR OCR repository.
Detection and recognition using docTR OCR
The following section guides you through the steps to perform detection and recognition using docTR OCR.
img_path = "path/image.jpg" #Specify your image path here
img = DocumentFile.from_images(img_path)
result = model(img)
output = result.export()The variable output contains the output in JSON format. However, the JSON output obtained after execution is too big. Hence I have stored it in a JSON file using the below code:
import json
with open("path/docTR_OCR_output.json", "w") as f:
f.write(json.dumps(output, indent=1))
f.close()The JSON output can be found here. As you can see, the output format is completely different when compared to the other OCR models. In the upcoming parts, I have shown the coordinates of text in the docTR OCR system along with the corresponding text recognized.
Get Bhargav Sridhar’s stories in your inbox
Join Medium for free to get updates from this writer.
Now let’s view the output.
result.show(img)
In this case, we have perfect text detection with all text blocks detected, and we can notice that each block is detected separately. Now, let us check the text coordinates and the corresponding text recognition.
for obj1 in output['pages'][0]["blocks"]:
for obj2 in obj1["lines"]:
for obj3 in obj2["words"]:
print("{}: {}".format(obj3["geometry"],obj3["value"]))Here, we have performed some nested dictionary comprehensions to obtain the coordinates and texts that give the following output.
((0.3359375, 0.013671875), (0.458984375, 0.0419921875)): Sample ((0.466796875, 0.0126953125), (0.5390625, 0.0380859375)): PDF ((0.5478515625, 0.0146484375), (0.716796875, 0.0361328125)): Document
((0.0205078125, 0.123046875), (0.07421875, 0.14453125)): This ((0.076171875, 0.123046875), (0.1015625, 0.1455078125)): is ((0.1025390625, 0.1259765625), (0.1220703125, 0.14453125)): a ((0.123046875, 0.123046875), (0.2109375, 0.1474609375)): sample ((0.21484375, 0.123046875), (0.26953125, 0.14453125)): PDF ((0.2744140625, 0.125), (0.3896484375, 0.142578125)): document ((0.39453125, 0.125), (0.4814453125, 0.1435546875)): created ((0.484375, 0.1220703125), (0.5224609375, 0.1455078125)): for ((0.5234375, 0.1240234375), (0.6025390625, 0.1484375)): testing ((0.6083984375, 0.125), (0.703125, 0.1435546875)): ditterent ((0.7080078125, 0.123046875), (0.7685546875, 0.14453125)): OCR ((0.7724609375, 0.125), (0.86328125, 0.1435546875)): models. ((0.0205078125, 0.265625), (0.07421875, 0.287109375)): This ((0.076171875, 0.2685546875), (0.126953125, 0.287109375)): was ((0.130859375, 0.267578125), (0.2177734375, 0.2861328125)): created ((0.2216796875, 0.265625), (0.255859375, 0.2919921875)): by ((0.0224609375, 0.353515625), (0.1171875, 0.3720703125)): NAME:- ((0.4873046875, 0.353515625), (0.6142578125, 0.37109375)): BHARGAV ((0.61328125, 0.3515625), (0.638671875, 0.3740234375)): S ((0.021484375, 0.408203125), (0.0966796875, 0.4296875)): OCRs ((0.1005859375, 0.41015625), (0.1806640625, 0.4287109375)): used:- ((0.4853515625, 0.41015625), (0.5634765625, 0.4287109375)): docIR ((0.5673828125, 0.408203125), (0.6298828125, 0.4296875)): OCR ((0.630859375, 0.4072265625), (0.6689453125, 0.431640625)): for ((0.6708984375, 0.4091796875), (0.919921875, 0.4326171875)): detectionarecognition
((0.0234375, 0.4677734375), (0.189453125, 0.48828125)): Programming ((0.1953125, 0.46484375), (0.2587890625, 0.490234375)): lang. ((0.2646484375, 0.466796875), (0.34375, 0.4853515625)): used:- ((0.4853515625, 0.46484375), (0.568359375, 0.4892578125)): Python ((0.021484375, 0.978515625), (0.1044921875, 0.99609375)): Created ((0.10546875, 0.9794921875), (0.14453125, 0.998046875)): on: ((0.220703125, 0.9794921875), (0.3564453125, 0.994140625)): DD/MM/YYYY
((0.79296875, 0.9775390625), (0.849609375, 1.0)): Page
((0.8515625, 0.978515625), (0.9423828125, 0.99609375)): Number: ((0.9638671875, 0.978515625), (0.978515625, 0.99609375)): 1You can observe that the overall recognition quality is very good (except in a few places, like “different” is extracted as ditterent, “&” as a, docTR as docIR).
Regarding the text coordinates, we can observe that the coordinates are by default normalized/scaled between 0 and 1, which is completely different from the graphical coordinate system.
The format of a graphical coordinate system is as follows:
[x_min, x_max, y_min, y_max]It can be visualized as follows:
In the following section, let us discuss how to convert these normalized geometries to a graphical coordinate system.
Converting geometry to the graphical coordinate system
Let us create a few methods to perform this operation.
import math
def convert_coordinates(geometry, page_dim):
len_x = page_dim[1]
len_y = page_dim[0]
(x_min, y_min) = geometry[0]
(x_max, y_max) = geometry[1]
x_min = math.floor(x_min * len_x)
x_max = math.ceil(x_max * len_x)
y_min = math.floor(y_min * len_y)
y_max = math.ceil(y_max * len_y)
return [x_min, x_max, y_min, y_max]
def get_coordinates(output):
page_dim = output['pages'][0]["dimensions"]
text_coordinates = []
for obj1 in output['pages'][0]["blocks"]:
for obj2 in obj1["lines"]:
for obj3 in obj2["words"]:
converted_coordinates = convert_coordinates(
obj3["geometry"],page_dim
)
print("{}: {}".format(converted_coordinates,
obj3["value"]
)
)
text_coordinates.append(converted_coordinates)
return text_coordinatesWe can obtain the page dimensions from the docTR OCR output using dictionary comprehensions similar to the ones performed previously. Then we used the length and width of the page to convert the coordinates.
By applying floor for min coordinates and ceil for max coordinates, we get the largest possible bounding box from the geometrical coordinates.
Now, let us use the output obtained in the previous section to convert coordinates.
graphical_coordinates = get_coordinates(output)Since we have included print() statements in the method, the following will be printed.
[208, 286, 10, 32]: Sample
[289, 335, 9, 29]: PDF
[340, 446, 10, 27]: Document
[12, 47, 91, 108]: This
[47, 64, 91, 109]: is
[63, 76, 93, 108]: a
[76, 131, 91, 110]: sample
[133, 168, 91, 108]: PDF
[170, 242, 92, 106]: document
[245, 299, 92, 107]: created
[300, 325, 90, 109]: for
[325, 375, 92, 111]: testing
[377, 437, 92, 107]: ditterent
[439, 478, 91, 108]: OCR
[479, 537, 92, 107]: models.
[12, 47, 197, 214]: This
[47, 79, 199, 214]: was
[81, 136, 198, 213]: created
[137, 159, 197, 217]: by
[13, 73, 262, 277]: NAME:-
[302, 382, 262, 276]: BHARGAV
[380, 397, 261, 278]: S
[13, 61, 303, 320]: OCRs
[62, 113, 304, 319]: used:-
[301, 350, 304, 319]: docIR
[352, 392, 303, 320]: OCR
[391, 416, 302, 321]: for
[416, 572, 304, 322]: detectionarecognition
[14, 118, 347, 363]: Programming
[121, 161, 345, 365]: lang.
[164, 214, 346, 361]: used:-
[301, 353, 345, 364]: Python
[13, 65, 727, 741]: Created
[65, 90, 727, 742]: on:
[137, 222, 727, 739]: DD/MM/YYYY
[492, 528, 726, 743]: Page
[528, 586, 727, 741]: Number:
[598, 608, 727, 741]: 1Now, let us plot the graphical_coordinates .
import PIL
from PIL import ImageDraw
import matplotlib.pyplot as plt
def draw_bounds(image, bound):
draw = ImageDraw.Draw(image)
for b in bound:
p0, p1, p2, p3 = [b[0],b[2]], [b[1],b[2]], \
[b[1],b[3]], [b[0],b[3]]
draw.line([*p0,*p1,*p2,*p3,*p0], fill='blue', width=2)
return image
image = PIL.Image.open(img_path)
result_image = draw_bounds(image, graphical_coordinates)
plt.figure(figsize=(15,15))
plt.imshow(result_image)The following output image was obtained after drawing bounding boxes.
Now this list of coordinates can be used to perform text recognition using a different OCR model.
Conclusion
To sum up, you have seen the procedure to perform text detection and recognition using docTR OCR. For the test image used, we were able to obtain perfect detection results with all the text blocks in the image. This is a best-case scenario and may not be the situation all the time. If text detection results are moderate or less accurate, we can perform image pre-processing techniques using inbuilt Python modules to improve image quality before performing text detection and recognition.
Now you have an idea about the coordinate systems followed by different OCR models and how to convert one system to another using Python. This similar idea can be followed for other coordinate systems too. This would ensure uniformity in the OCR process if multiple models are deployed.
I hope I have given you a good knowledge of these. I would be more than willing to acknowledge questions that can be posted through comments. Thanks!
