Automatic vehicle number plate detection using YOLOv5 and extraction of text using OCR
Problem Formulation:
The given problem is to design a model which would detect a vehicle number plate automatically and in addition to that extract the text and number from the detected number plate image using Optical character recognition(OCR). So it's a combination of both object detection(detecting number plate) and OCR(extract text from images).
Number plate detection can be done by using deep learning-based state-of-the-art convolution neural network such as YOLO, SSD with resnet, Faster-RCNN, and many more. Each model has its own trade-off between speed and mAP. In this project, YOLOv5 was used which is based on the PyTorch library.
Performance metric:
Before we deep dive into the object detection model, we need to know how to measure the performance of the model.
Intersection over Union(IoU):
In the above image, the green-colored rectangle box is a ground truth and the red-colored rectangle box is predicted by the model. IoU measures the area of overlap between two rectangles. Calculating IoU is clearly given below
The above formula is similar to the Jaccard similarity. The ideal case of IoU is 1 because in this case, the overlap of ground truth bounding box and predicted bounding box is a perfect match. The worst case is 0. Normally, in the object detection model, the common threshold value is 0.5 to consider the prediction is correct. Otherwise, the prediction is incorrect. So this is simply a binary classification model if our dataset has only one class label.
In this project, our dataset has only one class label (i.e) detecting number plate. So consider this as a simple binary classification model. If IoU is greater than or equal to 0.5, then the prediction is correct otherwise it's incorrect. In addition to this calculate precision and recall.
Average Precision:
For all possible threshold values calculate the corresponding precision and recall values and plot the precision-recall graph which is similar to the ROC curve. This area under the curve of the precision-recall graph is called average precision(AP).
mean average precision(mAP):
Remember we had only one label in this project, so we calculated all this for one class. But in most object detection cases, we need to detect multiple objects in an image. So we need to calculate average precision for all the classes. Average precision for first-class and then for second class and so on. Finally, we calculate the mean for all average precision. This is called mean average precision. This is the metric that is often used in object detection.
Yolov5 Architecture:
Its architecture mainly consisted of three parts, namely-
- Backbone: Model Backbone acts as a feature extractor from an input image. Feature extractor is nothing but using convolutional layers such as kernel, stride, batch normalization applies on input images to extract important features such as edges, shapes, and so on. CSP(Cross Stage Partial Networks) are used as a backbone in YOLO v5 to extract useful characteristics from an input image.
- Neck: In the Neck layer, the network was designed to perform Multi-scale prediction in addition to the Feature pyramid network. Multi-scale prediction helps to detect objects of different sizes by sending images into three different grid values. So small grid images detect large objects and large grid images detect small objects.
- Head: The model Head is mostly responsible for the final detection step. It uses anchor boxes to construct final output vectors with class probabilities, objectness scores, and bounding boxes.
This full network was trained to perform regression task which is nothing but coordinates of a bounding box and the probability of a class
Training yolov5:
The dataset was downloaded from Roboflow websites and the labeled data should be in a yolov5 PyTorch format.
#follow the link below to get your download code from from Roboflow!pip install -q roboflow
from roboflow import Roboflow
rf = Roboflow(model_format="yolov5", notebook="roboflow-yolov5")!curl -L "https://public.roboflow.com/ds/K9IjAYLk5y?key=9th6uXdcDR" > roboflow.zip; unzip roboflow.zip; rm roboflow.zip
In this project, yolov5m(medium) was used. Below are the steps to train the yolov5 for our custom dataset.
# clone YOLOv5 repository
!git clone https://github.com/ultralytics/yolov5 #change directory
%cd yolov5# install dependencies as necessary
!pip install -qr requirements.txt # install dependencies (ignore errors)#start training
!python train.py --img 416 --batch 32 --epochs 5000 --data '../data.yaml' --cfg ./models/custom_yolov5m.yaml --weights '' --name yolov5s_results --cache
Evaluation of custom yolov5:
After training, we need to evaluate the training losses and performance metrics for yolov5. For this evaluation, Tensorboard will be useful. Tensorboard uses a logfile to plot the graph while training itself.
# Start tensorboard
# Launch after you have started training
# logs save in the folder "runs"%load_ext tensorboard
%tensorboard --logdir runs
The mAP at 0.5 and the mAP at 0.5:0.95 are plotted for 5000 epochs. mAP@[0.5:0.95] means average mAP over different IoU thresholds, from 0.5 to 0.95, step 0.05 (0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95).
The Precision and recall for 5000 epochs are given below plot. Precision was good and stable over the last few epochs but recall values were lower over the last few epochs.
Training & Validation loss:
In object detection, there were a total of 3 losses to calculate
- Box loss — which is a squared loss because detecting coordinates of the box is a simple regression task
- Class loss — which is a log loss because it predicts the probability of an object belonging to a class.
- Object loss
Validation loss
Confusion matrix:
The dataset which I used for training has two class labels i.e detecting vehicle and license plate. So to know how well my model detects each class label, a confusion matrix will be useful.
My trained model detects license plate 75% correctly as license plate and it detects vehicle correctly 63% as vehicle.
Get ashok .c’s stories in your inbox
Join Medium for free to get updates from this writer.
The below graph shows all metric plot in one figure.
Run inference with trained model:
To run the inference for test image or unseen image, the below code will be used
# use the best weights!%cd /content/yolov5/
!python detect.py --weights /content/yolov5/runs/train/yolov5s_results/weights/best.pt --img 416 --conf 0.4 --source ../test/images
Region of Interest:
We need only the bounding box which contains the license plate so that we can input only that license plate to OCR for extracting text from images. This is called the region of interest(ROI). We need to extract ROI from the image inside the bounding box of the license plate automatically.
We need to modify the code in the detect.py file of the yolov5 directory to automatically crop the license plate whenever it detects.
## add the below code after line number 114 in detect.py file inside ## yolov5 directory..save_obj = True
if save_obj: if int(cls) ==0:
for k in range(len(det)):
x,y,w,h=int(xyxy[0]), int(xyxy[1]), int(xyxy[2] - xyxy[0]), int(xyxy[3] - xyxy[1])
img_ = im0.astype(np.uint8)
crop_img=img_[y:y+ h, x:x + w ###!!rescale image !!! filename=p.name
filepath=os.path.join(r'test_images/results/', filename)
print(filepath)
cv2.imwrite(filepath, crop_img)
else:
print("There is no detected object")
continue
Now, run inference on test images after modified code…
!python detect.py --weights /content/yolov5/runs/train/yolov5s_results/weights/best.pt --img 416 --conf 0.4 --source ../test/images
Whenever a test image contains a vehicle with a number plate, it automatically crop the number plate by using bounding box coordinates and gives input to OCR.
Optical Character Recognition(OCR):
Extracting text from an image is a challenging task because character recognition from an image is always difficult for a model due to noise, color, variance, skewness in an image.
So basic image processing needs to be done before inputting an image to OCR. we need to denoise the image, change color images to grayscale images, reduce contrast in the background image, deskew the image, and so on. This can be done easily by the OpenCV library.
So we need to define a function for the basic image processing
def my_function(image):
# import the necessary packages
import numpy as np
#import argparse
import cv2
# convert the image to grayscale and flip the foreground
# and background to ensure foreground is now "white" and
# the background is "black"
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
gray = cv2.bitwise_not(gray)
# threshold the image, setting all foreground pixels to
# 255 and all background pixels to 0
thresh = cv2.threshold(gray, 0, 255,
cv2.THRESH_BINARY | cv2.THRESH_OTSU)[1]
# grab the (x, y) coordinates of all pixel values that
# are greater than zero, then use these coordinates to
# compute a rotated bounding box that contains all
# coordinates
coords = np.column_stack(np.where(thresh > 0))
angle = cv2.minAreaRect(coords)[-1]
# the `cv2.minAreaRect` function returns values in the
# range [-90, 0); as the rectangle rotates clockwise the
# returned angle trends to 0 -- in this special case we
# need to add 90 degrees to the angle
if angle < -45:
angle = -(90 + angle)
# otherwise, just take the inverse of the angle to make
# it positive
else:
angle = -angle
# rotate the image to deskew it
(h, w) = image.shape[:2]
center = (w // 2, h // 2)
M = cv2.getRotationMatrix2D(center, angle, 1.0)
rotated = cv2.warpAffine(image, M, (w, h),
flags=cv2.INTER_CUBIC, borderMode=cv2.BORDER_REPLICATE)
# draw the correction angle on the image so we can validate it
#cv2.putText(rotated, "Angle: {:.2f} degrees".format(angle),
# (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0, 0, 255), 2)
# show the output image
print("[INFO] angle: {:.3f}".format(angle))
cv2.imshow("Input", image)
cv2.imshow("Rotated", rotated)
return rotated
cv2.waitKey(0)For the OCR we will be using Pytessaract. I have tried EasyOCR but its performance is lower than tesseract.
!pip install pytesseract
!sudo apt install tesseract-ocr
import pytesseract
text = pytesseract.image_to_string("/content/city-call-taxi.jpg")
print(text)
In the above image, tesseract recognized the number wrongly as 4 for 1. Yes, Character recognition using OCR is not accurate.
The original text and number in the license plate are given below.
There is still a lot that needs to improve on image processing in the future.
Thanks for reading…
GITHUB REPO: link
