<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vincent Dedun on Medium]]></title>
        <description><![CDATA[Stories by Vincent Dedun on Medium]]></description>
        <link>https://medium.com/@vincent_17917?source=rss-72f2021ca475------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*uAZHnEZQpKkVYUX8h5ijCQ.jpeg</url>
            <title>Stories by Vincent Dedun on Medium</title>
            <link>https://medium.com/@vincent_17917?source=rss-72f2021ca475------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 20:38:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vincent_17917/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Real Time 3D Face Tracking with Deep Learning]]></title>
            <link>https://medium.com/data-science/real-time-3d-face-tracking-with-deep-learning-963b91bb5ad4?source=rss-72f2021ca475------2</link>
            <guid isPermaLink="false">https://medium.com/p/963b91bb5ad4</guid>
            <category><![CDATA[face-tracking]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[augmented-reality]]></category>
            <dc:creator><![CDATA[Vincent Dedun]]></dc:creator>
            <pubDate>Wed, 16 Sep 2020 20:55:44 GMT</pubDate>
            <atom:updated>2020-09-17T16:10:24.764Z</atom:updated>
            <content:encoded><![CDATA[<h4>What I Learned Trying to Replicate the AI Tech Behind Snapchat Lenses And TikTok Effects</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wan5HKmMesdGM5ZT2xdOCQ.jpeg" /><figcaption>Me as seen by a computer</figcaption></figure><p>Snapchat was made popular by putting funny dog ears on people’s head, swapping faces and other tricks, that beyond funny, look impossible, even <em>magical</em>. I am in the digital visual effects industry so I am familiar with that magic.. and the desire to understand how it works behind the scene.</p><h3>Behind the magic</h3><p>Modifying people faces is routine work in Hollywood visual effects, it’s a well understood craft nowadays, but it typically requires tens of digital artists to achieve a photorealistic face transformation. How can we automate that?</p><p>Here’s a simplified breakdown of the steps these artists follow:</p><ol><li><strong>Tracking</strong> the position, shape and movement of the face relative to the camera in 3D</li><li><strong>Animation</strong> of the 3D models to snap on the tracked face (e.g. a dog nose)</li><li><strong>Lighting and rendering</strong> of the 3D models into 2D images</li><li><strong>Compositing</strong> of the rendered CGI images with the live action footage</li></ol><p>Automation of steps 2 and 3 is not very different from what happens in video games, it’s relatively straightforward. Compositing can be simplified to 3D foreground over live background, easy. The challenge is the tracking, how can a program ‘see’ the complex motion of a human head?</p><h3>Tracking faces with Artificial Intelligence</h3><p>The Computer Science community has been trying to track faces automatically for a long time and it’s <em>hard</em>. In the recent years, Machine Learning came to the rescue and many Deep Learning papers are published every year on the topic. I’ve spent a while looking for the “state of the art” and realised doing this in real-time is<em> VERY HARD</em>! A good reason to try and tackle the challenge (and that would work nicely with <a href="https://medium.com/swlh/how-i-implemented-my-own-augmented-reality-beauty-mode-3bf3b74e5507">the AR beauty mode I have implemented</a>).</p><blockquote>“trying to track faces.. it’s hard.. doing this in real-time is VERY HARD!”</blockquote><p>Here’s how I did it.</p><h4>Designing the network</h4><p>Convolutional Neural Networks are popular for visual analysis of images and commonly used for applications such as object detection and image recognition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/1*ytcUSl3DQnidyzpAhgmf6g.png" /><figcaption>Basic Convolutional Neural Network Architecture (from <a href="https://www.researchgate.net/publication/336805909_A_High-Accuracy_Model_Average_Ensemble_of_Convolutional_Neural_Networks_for_Classification_of_Cloud_Image_Patches_on_Small_Datasets">this publication</a>⁹)</figcaption></figure><p>For a deep neural network to be evaluated in real-time (at least 30 times per second), a compact network is desired¹. With the popularity of Machine Learning and smart phones, new models are discovered every year that push the limit of efficiency — offering a trade-off between computational <em>precision</em> and <em>overhead</em>. Among such models, MobileNet, SqueezeNet and ShuffleNet are popular for applications on mobile devices, thanks to their compactness.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w6cpeXyqXR2mZCvATpD9Mw.png" /><figcaption>Architecture of ShuffleNet V2 for different levels of complexity (from the authors¹)</figcaption></figure><p><a href="https://arxiv.org/abs/1807.11164">ShuttleNet V2</a>¹ was recently introduced and offers state of the art performances, coming in various sizes to balance between speed and accuracy. It ships with PyTorch, one more reason to pick that model.</p><h4>Choosing the features to learn</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/368/1*r9fMs5gELVGkOsBkypag2Q.jpeg" /><figcaption>Facial landmarks on a face (taken from <a href="https://arxiv.org/abs/1803.04108">this paper</a>²)</figcaption></figure><p>Now I need to find what features the CNN should learn. A common approach is defining a list of anchor points for different key parts of the face, also called ‘facial landmarks’.</p><p>The points are numbered and associated strategically around the eyes, eyebrows, nose, mouth and jawline. I want to train the network to identify the coordinate of each point, so I can later reconstruct masks or geometric meshes based on them.</p><h4>Building a training dataset</h4><p>Because I want to augment videos with 3D effects, I looked for a dataset with 3D landmark coordinates. 300W-LP is one of the few dataset that comes with 3D positions, it’s pretty large and as a bonus offers a good diversity of face angles. I want to benchmark my solution against the state of the art, recent publications test their models on AFLW2000–3D so I go for 300W-LP for training and test on AFLW2000–3D for comparison.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*Qx-9lHqVT--FpMhqgx-BOw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*N-AnqDTCHrDaGYbE72Y4uA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*8eazk1HAk1KejPaYbDsxlw.jpeg" /><figcaption>Images from dataset <a href="http://www.cbsr.ia.ac.cn/users/xiangyuzhu/projects/3DDFA/main.htm">300W-LP</a>³, profile views are generated mathematically</figcaption></figure><p>A note on these datasets, they are meant for the research community and generally not free for commercial use.</p><h4>Augmenting the dataset</h4><p>Dataset augmentation improves the accuracy of the training by adding even more variations to the set that it already has. I apply the following transformations to each image and landmark, to create new ones, by a random amount: rotation up to -/+ 40° around the centre, up to 10% translation and scale, and horizontal flip. I apply a different random transformation in memory on each image and for each learning pass (epoch) for additional augmentation.</p><p>It’s also necessary to crop the input image close to the bounding box of the landmarks for the CNN to recognise the landmarks at their relative locations. That’s done as a preprocess to save on load time from disk during training.</p><h4>Designing the loss function</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*QIplS2NYpXG1qhTxJz-sUA.png" /><figcaption>Plot of Wing loss function with different parameter settings (image from <a href="https://arxiv.org/abs/1711.06753">the publication</a>⁴)</figcaption></figure><p>Typically an L2 loss function is used to measure the prediction error for landmark positions. A recent <a href="https://arxiv.org/abs/1711.06753">publication</a>⁴ describes a so-called Wing loss function, that performs better for this application, which I could verify. I parametrise it with w=10 and ε = 2 as suggested by the author and sum the result over all landmark coordinates.</p><h4>Training the network</h4><p>Training a deep neural network is a very expensive operation that requires powerful computers. Using my laptop would have taken weeks, literally, for one training phase and building a decent setup costs thousands of dollars. I decided to leverage the cloud so I can pay just for the compute power I need.</p><p>I chose <a href="http://www.genesiscloud.com/">Genesis Cloud</a>, that offers very competitive prices and $50 free credit to get started. I build a Linux VM with a GeForce GTX 1080 Ti, prepare an OS and storage image where I setup PyTorch and upload my code and the datasets, all through ssh. Once the system is setup, it can be started and shut down on demand, creating a snapshot allows to resume the work where I left it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b4li6xffPQWhzUh4DsS6Bw.png" /><figcaption>Plot of mean error for each epoch</figcaption></figure><p>The inner training loop processes mini-batches of 32 images to maximise the parallel computation on GPU. A learning pass (epoch) process the entire set of about 60,000 images and takes about 4 minutes. The training converges around 70 epochs so I let it run overnight for 100 epochs to be safe.</p><p>I use the popular Adam optimiser that automatically adapts the learning rate, starting with a rate of 0.001. I found that setting the initial learning rate right is critical, if it’s too small the training converges too early in a sub-optimal solution. If it’s too large it has difficulties converging at all. I found the value through trial and error, which is time consuming.. and actually costly when paying the cloud per use!</p><h4>Evaluation</h4><p>All these efforts paid off, with the bigger network ShuffleNet V2 2x, I obtain a Normalised Mean Error (NME) of 2.796 on AFLW2000–3D. That’s better than the state of the art <a href="https://arxiv.org/abs/1812.01936">model</a>⁵ on that dataset and its NME of 3.07, by a good margin, despite that model being much heavier! 💪</p><p>A comparison of the predicated landmarks with the ground truth confirms the theoretical result, the landmarks find their way with precision even for large angles (though AFLW2000–3D contain challenging cases with extreme angles where my model fails).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IAEKGNQerIiCeL9TnQ3ZRQ.png" /><figcaption>Evaluation of my model on AFLW2000–3D, red markers are from ground truth, green are predicted</figcaption></figure><p>I don’t have a big CUDA GPU for inference (evaluation of the model) on my MacBook Pro. To maximise the utilisation of hardware resources I convert the model to the portable format ONNX and use the <a href="https://github.com/microsoft/onnxruntime">library</a> ONNX Runtime from Microsoft that can infer the model much more efficiently on my machine (yes it runs on OSX!). The inference time is under 100ms on the <strong>CPU</strong>, which is pretty good, despite not real-time. But I keep in mind I use the biggest version of ShuffleNet V2 (2x) here for maximal precision and I can opt for a smaller one for speed (e.g. the 1x version is 4 times faster). I could also get better numbers running on GPU.</p><p>All was set for a robust facial tracking system.. or so I thought.</p><h3>Limitations with videos</h3><p>I finally get to plug the trained network to a video stream. An additional step is required in using a separate model to detect the face bounding box, so we can crop the image close to where the landmark will be, like in the training dataset. After a good amount of research, I opt to use <a href="https://medium.com/syncedreview/smaller-is-better-lightweight-face-detection-for-smartphones-bcd27a1a1a82">this lightweight face detector</a> that is very fast to evaluate yet precise (I did try others).</p><p>Despite all my efforts, with great sadness, I find that the result on videos is not robust at all: the markers shake a lot, drift over time and don’t detect eye blinks. 😱</p><blockquote>What did I do wrong?</blockquote><h4>Shaky markers</h4><p>After digging further, I realised what it is. The training dataset are still photographs, not videos frames! It does seem to work well on stills, but videos are different. They represent an additional challenge: temporal consistency (and motion blur).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*R42U_DWhLN3eJrVwq78kRg.png" /><figcaption>Annotations are imprecise (from <a href="https://arxiv.org/abs/1807.00966">this publication</a>⁶)</figcaption></figure><p>As noted in <a href="https://arxiv.org/abs/1807.00966">this work</a>⁶ from University of Technology Sydney and Facebook Reality Labs, the ground truth markers from the training dataset are annotated manually on each photo, which is not very precise. Different annotators position the landmarks at slightly different locations, as illustrated above.</p><p><strong><em>Our network actually learns imprecise landmark locations </em></strong>— which result into random jittering in a small zone during prediction rather than exact location.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IKwaRNkRsf5LWBqTLmtYzg.png" /><figcaption>Illustration of the <em>1€</em> filter from the <a href="https://cristal.univ-lille.fr/~casiez/1euro/InteractiveDemo/">interactive demo</a></figcaption></figure><p>A common approach to address this issue is to stabilise the predicted markers after the fact. To that effect, as suggested by this <a href="https://arxiv.org/abs/1907.06724">other publication</a>⁷ from Google Research, I use the <a href="http://cristal.univ-lille.fr/~casiez/1euro/"><em>1€ filter</em></a><em>⁸ is </em>to smooth the motion noise. I found it gives better result than the traditional <em>Kalman</em> <em>filter</em> and is simpler to implement. I also use it to stabilise the box returned by the face detector, because it shakes a lot as well and that is not helping.</p><p>The motion filter makes a big difference. But it’s not perfect, specially with a low quality webcam like my laptop’s that produce noisy videos, which the model doesn’t seem to like very much.</p><p>The paper from UTS and Facebook RL mentioned above suggests to fine-tune the trained model essentially as follow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*Uh4cTxVr1DPuzCFyvEX6PA.png" /><figcaption>Forward-backward communication scheme (from <a href="https://arxiv.org/abs/1807.00966">this publication</a>⁶)</figcaption></figure><ul><li>Predict the landmark positions by computing the optical flow forward across frames (OpenCV can do that)</li><li>Compute the optical flow backward from that result</li><li>Compute a loss function based on the distance between the prediction from optical flow and the prediction from pre-trained model</li></ul><p>I haven’t tried it myself but the <a href="https://github.com/facebookresearch/supervision-by-registration/blob/master/cache_data/cache/demo.gif">demo video</a> looks promising.</p><h4>Stiff eyes and eyebrows</h4><p>The trained network suffers with eye motion, it doesn’t see eye blinks and can’t fully ‘close’ the eye landmarks. The eyebrows landmarks also seem stiff in their motion compared to what they are tracking.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/224/1*os6T5vRQ3aGPjVlb70DaEA.jpeg" /><figcaption>My model on a wide face angle</figcaption></figure><p>My model became pretty good at predicting profiles, because there is a ton of them in the dataset (it’s actually perfectly balanced in terms of face angles), but did not learned eyes that well. A CNN can perform better if the features it learns tend to live around a consistent location, that’s why we have to detect the face bounding box before inference and crop around the landmarks during training: that puts the face at the centre every time.</p><p>As the face angle varies, the features inside the face — eyes, mouth and nose — change shape and position in the frame and can even disappear at wide angle. This makes its more difficult for the model to recognise them well every time. I tried augmenting the dataset for extreme eye poses (closed, wide open), but that had little effect.</p><h3>Next steps</h3><p>This project was fun but costly in time and money ($15 from my pocket on top of the initial free credit of $50). It was a great opportunity to learn but I don’t intend to reinvent the wheel. Since I started this journey, Google has <a href="https://github.com/google/mediapipe">open-sourced MediaPipe</a>, that offers a cross-platform implementation of their model. Though initial demos of MediaPipe show some drifting, stiffness and shaking, which validates my outcomes.</p><p>Still, I have some ideas to improve my approach (starting with the optical flow based fine-tuning mentioned above).</p><h4>3D Morphable Face Models</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/295/1*e690O1DNHrpjR55yaqNouA.png" /><figcaption>The <a href="https://www.4dface.io/4dfm/">4DFM</a> morphable model</figcaption></figure><p>To be able to build an app like Snapchat, we actually need more than a couple of 3D points. A common approach is to use a 3D Morphable Face Model (3DMM) and fit it on the annotated 3D points. The dataset 300W-LP actually comes with such data. The network could learn all the vertices of the 3D model instead of a small amount of landmarks, such that a 3D face will be predicted directly by the model and ready to use for some AR fun!</p><h4>Multiple Networks</h4><p>In order to improve overall precision, the aforementioned Wing loss paper, as well as the work from Google Research, suggest to predict some landmarks with a first lightweight model. Thanks to the initial information, we can crop and align the face vertically to help the second model deal with less variation of poses.</p><p>I believe we can push that concept of specialised models further to involve multiple networks. We could train different CNNs for specific face angles, say one network for large angles (60°+), one for intermediate angles (30° to 60°) and one for frontal angles (0° to 30°), and somehow combine them.</p><p>Alternatively, different networks could learn different face parts, one for the eyes, one for the mouth, etc.. with the hope that simplifying the work of a model will increase precision. A first pass could detect the rough position and orientation of the head as above then crop and align the area of interest for the relevant expert networks.</p><h3>Conclusion</h3><p>Application of deep learning for face tracking is an active area of research where a lot of progress was made in the last couple of years. There is still room for improvement, in particular for tracking videos. The temporal coherence of predicted facial features and the precision for key areas like eyes and mouth remain a challenge.</p><p>I am excited to see what the future holds on that topic!</p><p>[1]: N. Ma et al., <a href="https://arxiv.org/abs/1807.11164">ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design</a> (2018), ECCV</p><p>[2]: X. Dong et al., <a href="https://arxiv.org/abs/1803.04108">Style Aggregated Network for Facial Landmark Detection</a> (2018), CVPR</p><p>[3]: X. Zhu et al., <a href="http://www.cbsr.ia.ac.cn/users/xiangyuzhu/projects/3DDFA/main.htm">Face Alignment Across Large Poses: A 3D Solution</a> (2016), CVPR</p><p>[4]: Z. Feng et al., <a href="https://arxiv.org/abs/1711.06753">Wing Loss for Robust Facial Landmark Localisation with Convolutional Neural Networks</a> (2018), CVPR</p><p>[5]: J. Guo et al., <a href="https://arxiv.org/abs/1812.01936">Stacked Dense U-Nets with Dual Transformers for Robust Face Alignment</a> (2018), BMVC</p><p>[6]: X. Dong et al., <a href="https://arxiv.org/abs/1807.00966">Supervision-by-Registration: An Unsupervised Approach to Improve the Precision of Facial Landmark Detectors</a> (2018), CVPR</p><p>[7]: Y. Kartynnik et al., <a href="https://arxiv.org/abs/1907.06724">Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs </a>(2019), CVPR</p><p>[8]: G. Casiez et al., <a href="http://cristal.univ-lille.fr/~casiez/1euro/">1 € filter: a simple speed-based low-pass filter for noisy input in interactive systems</a> (2012), CHI</p><p>[9]: V.H. Phung et al., <a href="https://www.researchgate.net/publication/336805909_A_High-Accuracy_Model_Average_Ensemble_of_Convolutional_Neural_Networks_for_Classification_of_Cloud_Image_Patches_on_Small_Datasets">A High-Accuracy Model Average Ensemble of Convolutional Neural Networks for Classification of Cloud Image Patches on Small Datasets</a> (2019), Applied Sciences</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=963b91bb5ad4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/real-time-3d-face-tracking-with-deep-learning-963b91bb5ad4">Real Time 3D Face Tracking with Deep Learning</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Implemented My Own Augmented Reality “Beauty Mode”]]></title>
            <link>https://medium.com/swlh/how-i-implemented-my-own-augmented-reality-beauty-mode-3bf3b74e5507?source=rss-72f2021ca475------2</link>
            <guid isPermaLink="false">https://medium.com/p/3bf3b74e5507</guid>
            <category><![CDATA[filters]]></category>
            <category><![CDATA[image-processing]]></category>
            <category><![CDATA[skin]]></category>
            <category><![CDATA[beautification]]></category>
            <category><![CDATA[augmented-reality]]></category>
            <dc:creator><![CDATA[Vincent Dedun]]></dc:creator>
            <pubDate>Sat, 12 Sep 2020 02:59:15 GMT</pubDate>
            <atom:updated>2020-12-01T21:11:46.826Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Comparison of original and smoothed skin" src="https://cdn-images-1.medium.com/max/1024/1*PlDxiCQ9LoS_71YRXrxGwg.png" /></figure><p>I was looking for a personal project to keep me busy during lockdown and decided to have a crack at implementing a Beauty Mode similar to Snapchat and TikTok, that is, a skin smoothing filter that would run in real-time on videos. It was more challenging than I anticipated, here’s a breakdown of how I did it.</p><h3>How do professionals do it?</h3><p>‘Improving’ skin in photoshop is common knowledge nowadays and it’s quite easy to find resources on the topic. There are surprisingly many different techniques to make skin look better, most having common grounds: <em>frequency separation.</em></p><figure><img alt="Original image" src="https://cdn-images-1.medium.com/max/1024/1*ppzB0rEoDuRIMKkdf3vEqw.jpeg" /><figcaption>Original image</figcaption></figure><p>What we are trying to achieve is to remove the various blemishes from the base picture above, the small variations in shape or colour on the skin, while keeping the overall look similar. Separating the small variations from the large variations can be achieved by separating the <em>high frequency</em> signal, which contains the blemishes, from the <em>low frequency</em> signal in the picture, the smooth skin.</p><p>To obtain the so-called <em>high pass</em>, we first apply a Gaussian blur with a radius just large enough to make the undesired small imperfections disappear (e.g. 10 pixels).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EowUl2fke7RSxjdwn0XmWg.jpeg" /><figcaption>Original image with Gaussian blur</figcaption></figure><p>Then, we subtract the blurred image from the original one and add a 0.5 grey (128 for 8bits colour depth), that’s our <em>high pass</em>. In Gimp, the <em>grain extract </em>layer mode does exactly that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*86-rm9K2pmnMbmkAM_l7cw.jpeg" /><figcaption>High pass</figcaption></figure><p>For the next steps, we want to somehow attenuate the details from the <em>high pass </em>on the original image, through some layering mode. Let’s overlay the <em>high pass</em> on the normal image, we get a sharper skin with even more visible imperfections, the opposite of what we want.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LcJGDxODFN9LJyDHUkZHTg.jpeg" /><figcaption>High pass overlaid on original image</figcaption></figure><p>To get the desired result, we simply invert the <em>high pass</em> colours,<em> </em>now we get a nice smooth skin!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XrL4EDqRPtQb4jF-wmVwBw.jpeg" /><figcaption>Inverted high pass overlaid on original image</figcaption></figure><p>Although.. we also smoothed parts of the image without skin.</p><p>Here’s a GLSL snippet to implement the overlay layering mode:</p><pre>vec4 overlay(vec4 <strong>a</strong>, vec4 <strong>b</strong>)<br>{<br>  vec4 <strong>x</strong> = vec4(2.0) * <strong>a</strong> * <strong>b</strong>;<br>  vec4 <strong>y </strong>= vec4(1.0) - vec4(2.0) * (vec4(1.0)-<strong>a</strong>) * (vec4(1.0)-<strong>b</strong>);<br>  vec4 <strong>result</strong>;<br>  <strong>result</strong>.<strong>r</strong> = mix(<strong>x</strong>.<strong>r</strong>, <strong>y</strong>.<strong>r</strong>, float(<strong>a</strong>.<strong>r</strong> &gt; 0.5));<br>  <strong>result</strong>.<strong>g</strong> = mix(<strong>x</strong>.<strong>g</strong>, <strong>y</strong>.<strong>g</strong>, float(<strong>a</strong>.<strong>g</strong> &gt; 0.5));<br>  <strong>result</strong>.<strong>b</strong> = mix(<strong>x</strong>.<strong>b</strong>, <strong>y</strong>.<strong>b</strong>, float(<strong>a</strong>.<strong>b</strong> &gt; 0.5));<br>  <strong>result</strong>.<strong>a</strong> = mix(<strong>x</strong>.<strong>a</strong>, <strong>y</strong>.<strong>a</strong>, float(<strong>a</strong>.<strong>a</strong> &gt; 0.5));<br>  return <strong>result</strong>;<br>}</pre><h3>How do we only smooth the skin?</h3><p>For Photoshop work, an artist would then use brush tools to create a mask and apply the smooth skin only where desired and keep sharp details elsewhere: eyes, hairs, mouth, etc. In our filter, we can’t do that easily.</p><p>Luckily, there is a type of blur that can preserve sharp edges such as eyes and hairs: it’s called S<em>urface Blur</em> in Photoshop, S<em>elective Blur</em> in Gimp and <em>Bilateral Filter</em> in the literature. Applying it to our original image by itself gives a result close to what we want out of the box:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AqTvIAjobPDffRjlRdqEcA.jpeg" /><figcaption>Bilateral filter on original image</figcaption></figure><p>Let’s use the same<em> frequency separation</em> technique as above and simply swap the Gaussian blur with the Bilateral Filter:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U0aHiqUGarhZc-tG6TK1vA.jpeg" /><figcaption>Frequency separation with bilateral filter</figcaption></figure><p>Pretty good! We can mix the raw bilateral filtered image with the result of the <em>frequency separation</em> process to balance between skin smoothness and detail preservation. In the image above, I bring 10% of the former back.</p><h3>Meh.. can we do better?</h3><p>Of course we can! Despite all ours efforts, while the skin looks pretty good, we’ve lost quite a few details in other areas that we’d like to keep. Remember, in Photoshop an artist would create a mask to apply the effect selectively.</p><p>Let’s try to generate a mask of the skin using some heuristics, based on the colour of the skin.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*TlO39ZjYM3VvA-gVIYviMQ.jpeg" /><figcaption>CIELAB colour space</figcaption></figure><p>For that we’re going to use the CIELAB colour space. It’s designed to take into account how humans perceive colours and has convenient properties for what we’re trying to do.</p><p>Consider the 3 axes of the colour space with a range [0, 1]. The <strong>L</strong> axis defines luminance, darker towards 0, lighter towards 1. The <strong>a</strong> axis represents the amount of red (+1) or green (0) and the <strong>b</strong> axis the amount of yellow (+1) or blue (0).</p><p>Our goal is to mask out pixels with colours that are unlikely to represent skin. We don’t want to be too strict, on a video or photo of the real world, as a number of factors will affect the final colour of the pixels: lighting conditions, lens, exposure and post-processing from the camera. In particular, indoor lighting is generally not neutral and will shift the hue of perceived colour quite a lot.</p><p>So rather that aiming at isolating skin colour exactly, we can use some heuristics to reduce the range of colour we’re going to pick based on the probability of that colour representing some skin.</p><p>The possible range of human skin colour is well understood, any possible skin tone has a mix of red and yellow tint. That is, these tones are reflected by the skin, while the tones opposite on the colour wheel are essentially absorbed.</p><blockquote>“we can use some heuristics to reduce the range of colour we’re going to pick based on the probability of that colour representing some skin.”</blockquote><p>If we consider the plane defined by axis <strong>a</strong> and <strong>b </strong>in<strong> </strong>Lab, clearly there is little chance skin will have a significant blue or green tint. We can discard colours with small <strong>a</strong> values and small <strong>b</strong> values. To get a smooth mask, we can use a <em>smoothstep</em> operator in a small range around 0.5, the middle of the colour axes.</p><p>Let’s now consider the other boundary on the <strong>a </strong>and <strong>b </strong>axes, values close to 1 are the most saturated yellow / red. Skin doesn’t reflect that much light (moderate albedo), so we can discard the highest values and reverse <em>smoothstep</em> in the range [0.9, 1.0]. For the same reason, we can attenuate the highest luminance values on the <strong>L</strong> axis with a reversed <em>smoothstep</em> in the range [0.98, 1.02].</p><p>Putting this altogether, we get the following GLSL snippet:</p><pre>float skin_mask(in vec4 <strong>color</strong>)<br>{<br>  vec3 <strong>lab</strong> = rgb2lab(<strong>color</strong>.<strong>rgb</strong>); # return values in the range [0, 1]<br>  float <strong>a</strong> = smoothstep(0.45, 0.55, <strong>lab</strong>.<strong>g</strong>);<br>  float <strong>b</strong> = smoothstep(0.46, 0.54, <strong>lab</strong>.<strong>b</strong>);<br>  float <strong>c</strong> = 1.0 — smoothstep(0.9, 1.0, length(<strong>lab</strong>.<strong>gb</strong>));<br>  float <strong>d</strong> = 1.0 — smoothstep(0.98, 1.02, <strong>lab</strong>.<strong>r</strong>);<br>  return min(min(min(<strong>a</strong>, <strong>b</strong>), <strong>c</strong>), <strong>d</strong>);<br>}</pre><p>The colour space conversion needs to be handled carefully. Photos and videos are typically encoded in <em>sRGB</em> colour space, properly converting to <em>CIELAB</em> looks likes this: <em>sRGB</em> to <em>linear</em> <em>RGB</em> to <em>CIEXYZ</em> to <em>CIELAB.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PlDxiCQ9LoS_71YRXrxGwg.png" /><figcaption>Final result</figcaption></figure><p>Now we can use the skin mask to mix the smoothed skin, that looks much better! Note how the hair, eyes and mouth look more natural.</p><p>One last thing about colour spaces, generally it’s a good idea to work in <em>linear</em> <em>RGB</em> space when manipulating colours, so we should convert the input image from <em>sRGB</em> to <em>linear</em> <em>RGB</em>, compute all our treatments in <em>linear</em> space, including the blur/filter, and convert the final result back to <em>sRGB</em>.</p><h3>Ok that looks great, but bilateral filter is slow!</h3><p>Indeed, if you ever tried the <em>Selective Blur</em> in Gimp, you’ll notice it takes its time to compute. A brute-force implementation of the bilateral filter looks like this pseudo code:</p><pre>For each pixel <strong>m</strong> in <strong>image</strong>:<br>  <strong>result</strong> = 0<br>  For each neighbour pixel <strong>n</strong>:<br>    <strong>weight</strong> = spatial_filter(spacial_distance(<strong>m</strong>,<strong> n</strong>))<br>    <strong>weight</strong> *= range_filter(range_distance(<strong>image</strong>(<strong>m</strong>), <strong>image</strong>(<strong>n</strong>)))<br>    <strong>total_weight</strong> += <strong>weight</strong><br>    <strong>result</strong> += <strong>weight </strong>*<strong> image</strong>(<strong>n</strong>)<br>  <strong>filtered_image</strong>(<strong>m</strong>) = <strong>result / total_weight</strong></pre><p>We can use a Gaussian filter kernel for both the <em>spatial_filter</em>() and <em>range_filter</em>() kernel operators, the former would return a weight based on pixel distance in space (coordinates), and the later based on distance in pixel luminance (we can use the <strong>L</strong> axis of the CIELAB colour space). Both kernels are defined with their respective, fixed, standard variation <strong>σ</strong>.</p><p>In the worst case (large neighbourhood), for each pixel, we would read every other pixel of the image, for a quadratic complexity O(N²)! Even with a moderate neighbourhood of 10 pixels, that’ll take too much time.</p><p>Fortunately, researchers found a way to compute a Bilateral Filter.. in real-time! The paper <a href="https://ieeexplore.ieee.org/document/5206542">“<em>Real-Time O(1) Bilateral Filtering</em>”</a> by Q. Yang et al. describes a simple technique where Bilateral Filtering can be computed as a linear combination of a constant number of spatially filtered images. Note that O(1) here refers to the complexity of the filter in terms of kernel size, applying the filter to an image yields a linear complexity O(N) with respect to the number of pixels.</p><p>Basically, we divide the range of luminance into a constant amount of <strong>K </strong>image<strong> </strong>‘slabs’ (called ‘PBFIC’ in the paper) and apply a spatial kernel filter to them (a.k.a. blur), for which we know an O(1) implementation. The result is obtained by linear interpolation of the filtered slabs based on the luminance of the original image. The pseudo code for this approach looks like this:</p><pre>For each slab <strong>l </strong>in range(0, <strong>K</strong>)<strong>:<br>  slab_range</strong><em> = </em><strong>l</strong> / (<strong>K</strong> - 1)<br>  For each pixel <strong>m </strong>in<strong> image</strong>:<br>    <strong>weights(m)</strong> = range_filter(range_distance(<strong>image</strong>(<strong>m</strong>), <strong>slab_range</strong>))<br>  <strong>blurred_slabs</strong>[<strong>l</strong>] = <strong>gaussian_blur</strong>(<strong>image</strong> * <strong>weights) / weights</strong><br>combine_slabs(<strong>blurred_slabs, image</strong>)</pre><p>The <em>combine_slabs</em>() operation linearly interpolates the 2 nearest (in range) images from the array <strong>blurred_slabs </strong>based on the luminance of the pixel in the original <strong>image </strong>and the <strong>slab_range</strong> value computed for each slab.</p><p>For our application, we can get good results with 5 layers, a Gaussian kernel of <strong>σ=3</strong> as spatial filter (in pixel size) and <strong>σ=0.1</strong> as range filter (on CIELAB <strong>L</strong> axis).</p><p>Gaussian kernel filters have known O(1) implementation on both CPU and GPU, and since the number of slabs <strong>K</strong> is constant, applying such solution on an image consequently offers a linear O(N) complexity. The <em>gaussian_blur</em>() operation can be implemented efficiently on the GPU with the optimised separable Gaussian filter described by Filip Strugar on his Intel blog: <a href="https://software.intel.com/content/www/us/en/develop/blogs/an-investigation-of-fast-real-time-gpu-based-image-blur-algorithms.html">https://software.intel.com/content/www/us/en/develop/blogs/an-investigation-of-fast-real-time-gpu-based-image-blur-algorithms.html</a></p><h3>An OpenGL implementation</h3><p>We need to use 11 textures, one for the original image, and two for each slab of the bilateral filter. Multiple GLSL render passes are required, so we should use Framebuffer Objects to hold intermediate results.</p><p>Here’s the order of operations, a separate pixel shader is used per pass and rendered on the entire image domain:</p><ol><li>Prepare 5 slab textures from the original image converted to <em>linear RGB</em></li><li>Run separable Gaussian blur horizontally on each slab texture</li><li>Run separable Gaussian blur vertically on the output of previous step</li><li>Combine slabs, apply frequency separation and convert back to <em>sRGB</em></li></ol><p>The Gaussian blur pass can render all slabs at the same time by writing to an array of output FBOs, to which we attach 5 of the allocated textures. That’s why we need two textures per slab, one for the input, one for the output. The slabs can be rendered half resolution without visible degradation for a massive performance boost.</p><p>This is still a lot of texture reads, it runs on a 720p video at 60fps and 1080p at over 30fps on an average GPU (Radeon Pro 555). Note that this is with a Python/OpenGL implementation running on OSX, an optimised implementation in C++/Vulkan or Swift/Metal could probably perform much better.</p><h3>Possible improvements</h3><p>I tested this video filter on various images, videos and live webcam on different people. The result looks quite robust, the effect noticeably smooth the skin while not producing strange results elsewhere.</p><p>If we’re picky, we could consider the skin is smoothed a bit too much, looking like baby skin. To alleviate that, a popular Photoshop technique is blurring the <em>high pass</em> a bit before layering it down with overlay, by just a pixel or two. This has the effect to smooth out the medium sized features but keep the very thin ones. I tried that and got interesting results on sharp photographies, but consumer grade videos from webcams and phones tend to be too blurry to be worth it. We can also simply mix the original image with the result a little to recover some sharpness or apply a subtle sharpness filter to the end result.</p><p>Also, the effect can sometimes pick other things than skin where the colour is similar. Wooden hard floor for example will loose fine patterns:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LaR0GbSIhitHA-v7ZxlTTQ.jpeg" /><figcaption>Smooth skin filter applied on wood</figcaption></figure><p>The wood still looks like wood, so it’s not really a problem. If that’s annoying we could restrict the skin smoothing filter to faces, by tracking them with a facial landmark detection technique.. that will be a topic for a <a href="https://link.medium.com/SktBFo94P9">future article</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3bf3b74e5507" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/how-i-implemented-my-own-augmented-reality-beauty-mode-3bf3b74e5507">How I Implemented My Own Augmented Reality “Beauty Mode”</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>