SpeculAR : Implementation
Real time face tracking and head pose estimation for Augmented Reality in mobile platforms
Face detection
The first step of the application process is to identify whether a human face is present in an image frame. If any human faces are not found not further processing is done.
For this purpose, the cascade detector provided in OpenCV was used along with a pre trained cascade classifier trained to identify frontal human faces. This cascade detector is built based on the famous Viola Jones face detection algorithm. The input image is swept through using a window of a specified size to check for any possible human faces. If found a bounding rectangle of that face is returned.
The OpenCV API allows specifying certain parameters in running this detector and to suit the requirements of the application a window of size 90x90 pixels was used and scale factor of the image of 2 was specified.
Facial feature detection and tracking
Once a face is detected, the next step is to extract the feature points on the face. Those detected feature points are then used to track the face and its pose variations.
The initial attempt was to use cascade detectors trained to detect specific parts of the face namely right eye, left eye, nose and then use those feature points to track the face using the KLT point tracker. This approach could give satisfactory results in general but the tracking of the two eye points lost its accuracy significantly when the person blinked eyes. The reason behind was that KLT point tracker makes use of the textural variations in the images and tracking the black pupil of the eye was not a problem, but as the pupil cannot be seen when the person blinks the tracked texture variation suddenly vanishes causing the tracking to lose accuracy. Accordingly it became evident that this approach is not robust enough to meet the requirements of our application.
The next attempt was to extract some good features to track from the facial region, ideally some sharp corner points, and use them to track the face. From the geometrical movement of those detected points, the positions of the four facial features two eyes, nose tip and center of mouth were estimated. This was achieved by computing the geometric transform between the matching corner point pairs in two consecutive image frames. This approach could solve the problem with the person blinking the eyes but showed a trend of reduced accuracy in the position estimates of the nose tip. Accordingly it was decided to use the normal feature tracking with the nose tip point and use the position estimates from geometrical movement of the corner points for the points of two eyes and center of mouth.
For extracting the good features to track (corner points) on face, the Shi-Tomasi feature extractor was used .
In determining the transformation matrix between two sets of matching points, two points from one set are chosen at random along with their corresponding points in the other set. Then those points are normalized so that their center of mass is at (0,0) and the mean distance from the center is
. The normalized points are next used in generating a matrix which is then subjected to single value decomposition. One of the resultant three matrices are used in the final computation of the transform matrix. This obtained matrix is used again to evaluate how close the transformed points using that matrix are to the actual points and if the results are not good enough the same process is followed a certain number of times until satisfactory results are achieved.
Finally to improve the initial position extraction of the four facial features, a Constrained local model was used as described in next section.
Facial feature extraction using Constrained Local Model
A joint model of a shape model and a patch model was developed to find landmarks of the main four facial feature points (left eye, right eye, tip of the nose and mouth) and some additional feature points (corner point of the left eye, corner point of the right eye and bottom most point of the mouth). The implementation of this procedure can be summarized as below.
This approach required a database of faces to set the landmark positions along with the faces. MUCT face database that consists of 3755 faces with 76 manual landmarks was selected as the database of interest. Once data were prepared, shapes made of landmarks were aligned using Procrustes Distance based Alignment. This refers to the process of removing global rigid motion from a collection of points.
After aligning the shapes, standard Principal Component Analysis (PCA) was computed to find main shape distortions. Following diagram shows an example of PCA mean at the center and main rigid basis deformations of it.
The shape model analytical expression for shape ‘X’ can be expressed in a linear function as below.
Here ‘X’ refers to the mean shape which is represented by a 2n x 1 vector.
X = X’ + Pb
X = ( x1,…,xn,y1,…,yn)
‘P’ is a matrix constructed using the few first eigenvectors obtained from the Principal Component Analysis on shapes. ‘b’ is the corresponding mode of variation. By varying values of b, usually between ±3σ, different face deformations can be modeled. This formulation is the shape model which is also known as the Point Distribution Model (PDM). That represents the mean geometry of a shape and some statistical modes of geometric variation inferred from the training set of shapes. Each mode changes the shape by moving landmarks along straight lines through mean positions. Thus, new shapes could be created by modifying mean shape with weighted sums of modes.
OpenCV implements a class for computing PCA, however, it requires the number of preserved subspace dimensions to be prespecified. As this is often difficult to determine a priori, we used a common heuristic to choose it based on the fraction of the total amount of variation it accounts for.
Then we built the facial feature extractor using a representation of a linear image patch. This representation gave reasonable estimates of facial feature locations for use in the KLT face tracking algorithm while enabling an extremely rapid evaluation that makes real time face tracking possible. Due to their representation as an image patch, the facial feature detectors were referred to as patch models. This image patch, when cross-correlated with an image region containing the facial feature, yields a strong response at the feature location and weak responses everywhere else. A visualization of this setup is shown in the following figure for the left outer eye corner.
In this way, the landmarks of the shape model were extracted out and the main feature points were sent as inputs for the tracking process.
Head Pose Estimation
As suggested by A.H. Gee and R. Cipolla, head pose can be estimated geometrically by finding the facial normal of a given facial image considering the geometric distribution of the facial features. A modified version of their method is used in the application.
Get Nanduni Indeewaree Nimalsiri’s stories in your inbox
Join Medium for free to get updates from this writer.
The roll angle of the head is computed by taking into account the angle between the line connecting the pair of eyes and the horizontal axis. The computation of the pitch angle relies on the assumption that in an average human being, the ratio between the lengths from the eye level to the base of the nose and to the center of the mouth stays roughly the same. Several authors have proposed this ratio to be 0.6. And when the subject changes the pitch angle of the head, this ratio varies between 0 and 1. Hence the pitch angle can be calculated proportionately to the change in this ratio. As this ratio can vary from person to person, the exact ratio is calculated at the instance the face is detected, assuming that the person is looking straight at the camera.
Similarly the yaw angle can be calculated proportionately to the change in the ratio between the length of the perpendicular drawn from the nose tip to the facial symmetrical axis and the length from the mid point between the eyes to the center of the mouth.
Optimizations
Histogram Equalization
Initially, without a proper way of handling low light conditions and irregularly illuminated faces, the application often could not detect faces and its features even in normally lit indoor environments. The first attempt at addressing this issue was to use the classical histogram equalization technique and do all the further processing of the image on the histogram equalized image. This could not show significant improvements since the classical histogram equalization follows a global approach and does not suite in instances where there is a foreground and a background with different levels of illumination. In a general usage scenario of our application, the face of the user which is close to the camera would be more lit than the background which is further away from the camera.
Classical histogram equalization applied on such a scenario would improve the contrast of the background but cause the foreground to lose a majority of its features. The Adaptive Histogram equalization technique addresses this issue by dividing the image into smaller regions of fixed size and performing the histogram equalization locally to those regions. The downside of this approach is that any existing noise in the image also getting amplified. The Contrast Limited Adaptive Histogram Equalization (CLAHE) imposes a limit for the contrast so that it will not exceed a certain threshold and solves that issue.
So finally, it was decided to apply CLAHE to each input image frame before proceeding with further calculations. This significantly improved the sensitivity of the application and both face detection and feature tracking with decent accuracy became possible in even dimly lit environments.
Divergence Check
At each frame, a method is implemented to check whether the locations of the four facial feature points (the pair of eyes, tip of the nose and the center of the mouth) estimated move out of a defined area bounding the face. A face bounding rectangle is obtained at the first instance when a face is detected by the Viola-Jones face detector. Then the corner points of this rectangle is transformed at each frame using the transformation matrix calculated by taking into account the movement of salient feature points of the detected face in the video stream. This gives corner points of a convex polygon which bounds the actual face region. Then the following procedure is implemented to check whether a given facial feature point lies inside this convex polygon or not.
First vectors are drawn from the feature point to each of the vertices of the bounding polygon. Then the angle between consecutive vectors is calculated. The addition of the angles is equal to 360 if the point lies inside the polygon, otherwise the addition is equal to 0. These two situations are depicted in following figures.
When the point is inside the polygon, a+b+c+d = 360.
When the point is outside the polygon, a+b+c+d != 360.
In case if a particular feature point going beyond the bounding polygon which leads to a divergence of the feature points, the entire algorithm is reinitialized preventing the virtual object getting rendered in the incorrect location, scale and orientation.
Graphical user interface of the mobile application
The graphical user interface of the main mobile application was designed using Unity. Unity provides WebCamTexture class that starts the camera to get the live video feed. The set of 3D models of spectacles are displayed on the main interface of the application. User can select a preferred model and try it in real time.
The available 3D models of the accessories are stored in the ‘Resources’ folder of the project. That contains the 3D models as well as the images of the spectacles. These images can come in different file formats like png, jpeg and jpg. They are used to display a 2D view of the spectacle to the user through the main user interface. The 3D models, which can be developed using different 3D model building software like 3DSMax can come in different file types like fbx, obj, 3ds etc. Each model comprises ‘Materials’ and ‘info’ file as shown in Figure 6.11. Materials are useful to quickly change the appearance of the objects to look much more realistic. Info file is a JSON file which contains information about the 3D model, the distance between the pair of glasses in pixels, it’s original rotation, the position it should be originally placed when dynamically adding them onto the scene.
Each model is displayed to the user as a button with its corresponding image. Once the user selects a model by clicking the button, a 3D object of that model is dynamically added as a child object under a dummy parent object which is originally pivoted at position (0,0,0) and rendered to the canvas. First of all, Unity’s Resources.Load function is used to load the asset stored at path in the Resources folder. Then the relevant parameters of the model are read from the info.json file and the positioning of the model is adjusted accordingly. Finally, the model is successfully added to the parent game object.
Addition of the 3D models for an empty parent object was developed as a workaround for the pivot position of the downloaded .fbx or .obj models being inconsistent. The model’s pivot position inside the parent object is set to a little below the bridge of the glasses and included in the info.json file. This helps in correctly placing the glasses at mid of the line connecting the two eyes.
Unity’s OnGUI function is used for handling the GUI events. A scroll view is added to the GUI in case the screen width is not sufficient to add buttons for all available 3D models of accessories. Unity’s GUI functions are used to generate the buttons. A custom UnityGUI ScrollView was developed to facilitate dragging the list itself up and down with the finger. OnGUI code draws the scrolling list but no longer responds to clicks. The code for detecting and responding to touches is added to FixedUpdate method.
Another button control called ‘Re-init’ is also included in the main interface to restart the whole process of face detection, tracking and pose estimation.
References
- The MUCT Face Database. (2016). Milbo.org. Retrieved 9 January 2017, from http://www.milbo.org/muct/
- Lelis Baggio, D., Emami, S., Escrivá, D., Ievgen, K., Mahmood, N., Saragih, J., & Shilkrot, R. (2012). Mastering OpenCV with practical computer vision projects (1st ed.). Birmingham, UK: Packt Pub.
