SpeculAR : Tools and libraries
Real time face tracking and head pose estimation for Augmented Reality in mobile platforms
All implementation of the application was done using the Unity Game Engine. Unity 3D is one of the world’s most famous game development platforms that provide a multitude of different tools to build both 2D and 3D gaming, AR or VR applications with ease. It abstracts away a lot of low level implementation details like platform specific optimizations and deployments and lets the developers better focus on application logic without having to worry about platform specific considerations. Unity also has a very active community and a dedicated asset store that proves to be very useful in getting development issues solved quickly and easily.
To avoid reinventing the wheel, some computer vision libraries were used to facilitate the various image processing and computer vision requirements of the application. Namely, the libraries Accord.NET, EmguCV and OpenCV were tried out to choose the best candidate for the application development.
Accord.NET
Accord.NET is a machine learning framework written completely in C#. It has audio and image processing libraries built into it. By including pre-built dlls of Accord.NET inside Unity’s assets folder, the functions provided by Accord.NET can be accessed from Unity scripts. Using Accord we implemented a Viola-Jones face detector that runs on the PC platform. In addition, the pair of eyes, nose and the mouth were detected by loading HaarCascades for eye pair, left and right eyes, nose and the mouth from OpenCV-compatible XML files provided at http://alereimondo.no-ip.org/OpenCV/34.
Get Nanduni Indeewaree Nimalsiri’s stories in your inbox
Join Medium for free to get updates from this writer.
Imaging functions in Accord.NET have dependencies on the C# System.Drawing library. The issue of trying to get the face detector to run on Android was that some of the functions in System.Drawing library, which are being used by Accord did not have support on Android. Due to this lack of support, this solution had to be abandoned.
OpenCVSharp
OpenCV is a famous open source computer vision library available to be used in several languages. Since development in Unity is done using C#, the freely available OpenCVSharp library which is a .NET wrapper for the OpenCV library was used. The library supported all the functionality we required on the PC but failed to perform certain required operations when built and run on a mobile device. The reason behind was that mobile devices could not run the OpenCV native scripts built for PC.
Emgu CV
Emgu CV is also a cross platform .Net wrapper to the OpenCV image processing library. This allows OpenCV functions to be called from .NET. It is written entirely in C#. The benefit with EmguCV was that it could be compiled in Mono and therefore it was able to run on any platform Mono supports, including iOS, Android, Windows Phone, Mac OS X and Linux. We implemented face detection with EmguCV to find well that performs.
OpenCV
After researching for the better option between OpenCV and Emgu CV, we decided to proceed with OpenCV. This is because we required ultra fast and support with complex algorithms that we used, which can be readily provided by the OpenCV image processing library. On the other hand, OpenCV was cheaper than Emgucv.
Not being able to proceed with the free C# version of OpenCV, we purchased the proprietary OpenCV plugin built for Unity, OpenCVForUnity, which could run on any platform supported by Unity. This is the library that was finally used in the development of the final product.
