<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[MIRA BEAUTY™️ - Medium]]></title>
        <description><![CDATA[Data is Queen - Medium]]></description>
        <link>https://medium.com/mathematical-beauty?source=rss----167604512b2b---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>MIRA BEAUTY™️ - Medium</title>
            <link>https://medium.com/mathematical-beauty?source=rss----167604512b2b---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 13:39:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/mathematical-beauty" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Adaptive Personalization for Beauty Retail]]></title>
            <link>https://medium.com/mathematical-beauty/adaptive-personalization-for-beauty-retail-1dd05d34ca8d?source=rss----167604512b2b---4</link>
            <guid isPermaLink="false">https://medium.com/p/1dd05d34ca8d</guid>
            <category><![CDATA[retail]]></category>
            <category><![CDATA[personalization]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[beauty]]></category>
            <dc:creator><![CDATA[Jay Hack]]></dc:creator>
            <pubDate>Tue, 13 Apr 2021 13:21:01 GMT</pubDate>
            <atom:updated>2021-04-13T13:22:00.470Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>“Software is eating the world”<em> — Marc Andreessen</em></blockquote><blockquote>“AI is eating software” — Unknown</blockquote><p>Something magical is happening in consumer tech: digital experiences are coming to life as they learn to adapt to their users.</p><p>As anyone with a TikTok account can attest, its <a href="https://newsroom.tiktok.com/en-us/how-tiktok-recommends-videos-for-you">AI-powered feed algorithm</a> learns your preferences and fluidly guides you through troves of compelling content, producing record user engagement in the process. Likewise, Facebook <a href="https://research.fb.com/wp-content/uploads/2018/10/Horizon-Facebooks-Open-Source-Applied-Reinforcement-Learning-Platform.pdf">leverages deep reinforcement learning</a> to trade spammy notifications for just the right amount of outreach, avoiding user burnout and cultivating impressive user retention. This emerging breed of <em>adaptive personalization</em> looks beyond single, short-sighted recommendations, instead optimizing long-term business objectives such as retention and LTV.</p><p>At <a href="https://mirabeauty.com/">Mira AI, Inc.</a>, we’re disrupting beauty commerce with truly intelligent personalization. In this article, we’ll describe how we’re leveraging the machine learning methods powering TikTok, Facebook and others to optimize the beauty shopping journey.</p><p>At a glance, we’ll cover:</p><ul><li>Why personalization is so important for beauty shoppers</li><li>How we can extract insights from beauty content with machine learning</li><li>A technical description of how we’re leveraging these insights to guide customers towards purchase</li><li>An overview of our preliminary tests and results</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-n0RoL2r2ywwNH4r5GXHEw.png" /><figcaption><a href="https://mirabeauty.com/">https://mirabeauty.com</a> — the next generation of beauty retail</figcaption></figure><h3>Beauty’s Customer Journey Towards Purchase</h3><p>Beauty is a complex domain with a long and involved journey towards purchase. Everyone has received a raving recommendation from a friend for a beauty product that couldn’t have been more incompatible with their unique complexion or skin type — in a category where you literally apply a (pigmented) chemical to your face, understanding how a product works with your unique needs and preferences is crucial.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W5ZAhNdzDlnkIIXR9vGihQ.png" /><figcaption>Every beauty buyer is unique and the products are complex, leading to a long and involved journey towards purchase</figcaption></figure><p>Despite an ever-increasing number of physical and digital touch points between customers and retailers, the industry has largely adhered to a one-size-fits-all model: e-commerce, marketing, communications and even promotional samples fail to take into account the <em>unique traits of each customer</em> and properly educate them about what products will work best.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EYi0-tjIZ7zVkaQRZJerOw.png" /><figcaption>In an era of online commerce, the customer lifecycle has dozens of touch points. Each is an opportunity to educate customers in a way that takes into account their unique needs and preferences.</figcaption></figure><p>As a result, consumers are largely left to their own devices to discover and research which products will work for them. The modern customer journey often devolves into “attack of the tabs,” in which a customer spends hours pouring over fragmented resources — videos, images, reviews, blogs, etc. — to build conviction in a beauty purchase.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zJeQDpPmzPviNtyvnxzvmQ.png" /><figcaption>“Attack of the Tabs” — the process of researching beauty products across a fragmented online ecosystem</figcaption></figure><p>What should otherwise be a fun opportunity to shop is a high-friction and confusing trudge across the web. However, as we’ll demonstrate, there’s a better way: we can leverage a nuanced understanding of content, preferences and customer behaviors to personalize product discovery across digital and physical touch points. In other words, we help customers find products they will love.</p><h3>Extracting Insights from Beauty Content</h3><p>As a first, primitive approach to a more personalized shopping journey, we can leverage machine learning to serve contextually-relevant, personalized content to customers within individual touch points — e.g. recommended articles in emails and more.</p><p>Recent advancements in AI enable us to extract semantic content from <a href="https://www.forbes.com/sites/bernardmarr/2019/10/18/whats-the-difference-between-structured-semi-structured-and-unstructured-data/">“unstructured” data</a> — or, more tangibly, to <em>figure out what’s going on </em>in text, images, video and beyond — then prioritize the things that most closely match our customers’ needs and preferences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B3THxQ3G5_aZUmXexmVb4A.png" /><figcaption>Example of personalizing a customer’s view into a product detail page by simply matching their traits — skin tone, skin type, and content preferences — with semantically enhanced beauty data.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vsNqf1KNb5pQmisg" /><figcaption>Recent advancements in computer vision, namely convolutional neural networks (CNNs), enable the extraction of insights from beauty videos and images</figcaption></figure><p>At Mira AI, Inc., we leverage computer vision (<a href="https://medium.com/mathematical-beauty/deep-learning-for-cosmetics-2d1b427bbfa2">as described in a previous blog post</a>) to analyze videos and images to extract domain-specific information including the creator’s eye shape, skin tone and much much more, then sort videos on a given customer’s view of a product page accordingly.</p><p>Similarly, recently developed machine learning algorithms such as <a href="https://en.wikipedia.org/wiki/GPT-3">Open AI’s GPT-3</a> (and more generally <a href="https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html">the deep learning “transformer” architecture</a>) have enabled us to be <em>shockingly good</em> at automatically extracting semantics from text. At Mira AI, Inc., we’ve leveraged this technology to analyze millions of reviews, extracting meaningful insights that actually help inform purchasing decisions, such as user skin type and sentiment.</p><p>In practice, these <em>personalized content filtering</em> techniques remove noise and clutter while creating a more delightful customer experience. See, for example, our intelligent selection of product imagery based on user-declared skin tone:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1AJaijL9mr1P3S_0-pHcJA.gif" /><figcaption>Here, we combine users’ self-declared traits with semantically enhanced content to show custom product images based on the customer’s skin tone.</figcaption></figure><p>This type of personalization can be found in limited quantities in many other online retailers today. Ultimately, however, personalized content filtering is limited. (Similar, perhaps, to Facebook optimizing individual notification clickthrough rates instead of their holistic user experience.) It only directly optimizes for the customer experience within a <em>single touchpoint</em>. The true power of adaptive personalization, as we will demonstrate, lies in <em>aligning all touch points towards greater customer outcomes such as customer satisfaction, retention and LTV</em>.</p><h3>Adaptive Personalization for Beauty Retail</h3><p>While Intelligent content filtering is limited, it will inevitably become the de facto standard of online retail. <em>Adaptive personalization, </em>however, reaches beyond optimizing isolated touch points to supercharge the holistic beauty shopping experience. Here, we describe how we’re working to optimize customer LTV through adaptive personalization.</p><h4>The Customer Journey as an Interactive Feed</h4><p>Consider a customer’s beauty shopping journey as a “feed”: whereas TikTok provides a feed of videos, beauty e-commerce is essentially a feed, or sequence, of interactions between customers and content. For example, a customer interacts with a sequence of content “blocks,” or page elements, as they navigate an e-commerce website, as illustrated below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mrxhl7qrp0_ylTd796jmqA.png" /><figcaption>A simplistic model of e-commerce pages distills them to sequences of content and customer interactions. As we will demonstrate, we can intelligently select this content to drive higher customer satisfaction and LTV.</figcaption></figure><p>All cross-channel interactions between retailers and customers, including email/CRM, service interactions, promotional offers, and even the unboxing experience fall into this paradigm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1omfXyy-KhZa-aE6Eqqfzg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oh4VNfbbcpZC4fVAphNM-Q.png" /><figcaption>We can model every customer touchpoint, including email, on-page e-commerce and beyond, as a sequence, or “feed,” of customer interactions with content.</figcaption></figure><p>Each touchpoint in this feed, or customer journey, presents the opportunity to <em>nudge our customers towards products they will love</em>. Showing a customer a promotional offer in a category they frequently purchase, for instance, is likely to drive more purchases. Similarly, recommending personalized gifts with purchase to a customer can help drive short-term conversion <em>and</em> get them to return to buy the full-sized versions. The same can be said for showing editorial content customized to a user’s interests when they visit your home page, sending them relevant new product releases in their inbox, and much much more.</p><p>With adaptive personalization, we can customize these touch points with the <em>explicit objective of maximizing customer LTV.</em> When properly executed, the value of such a system is clear: more satisfied customers and significantly greater revenue.</p><h3>The Adaptive Personalization Algorithm</h3><p><em>This section contains an in-depth explanation of the math behind the methods described above and assumes familiarity with the math of sequence-to-sequence deep learning. Skip to “In the Wild” to see our personalization in action.</em></p><p>We’ve formalized the customer LTV optimization problem as follows: given a state space of customer touch points (products recommendations, emails, promotional offers, purchases, etc.), traverse the graph to optimize for (time-discounted) customer LTV.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RTNpiXjiq3Lapr9Su_yrsg.png" /><figcaption>The state space for customer touch points: at any point in time, a customer can engage with many different pieces of content</figcaption></figure><h4>Mathematically Modeling Customer Touch Points</h4><p>As a customer navigates the state space of possible touch points, their journey consists of a sequence (or “feed”) of encounters with pieces of content. We form semantic embeddings of said content — e.g. products, images, videos, reviews, blog content, and more — using a combination of hand-written feature extractors and the computer vision/NLP methods described earlier. Thus, a customer journey is represented as a sequence of N-dimensional vectors, consisting of the content they encounter, along with a set of engagement variables representing their expressed level of interest.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_e9ovqndbqHAlOli8rnR3A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0NtR-G1932NkFbBb7SQMIw.png" /><figcaption>As input to our model, we create N-dimensional vector embeddings from products, images, editorial content. These embeddings can be laid out, in sequence, to represent the sequence of touch points a customer progresses through.</figcaption></figure><h4>Learning to Predict Customer LTV</h4><p>In order to produce a useful output from our model — the ability to intelligently select content to maximize LTV for any given customer — we train our model to jointly predict the following:</p><ul><li>At each time step, a continuous value representing customer engagement</li><li>At the end of a finite sequence of time steps, the customer’s time-discounted LTV</li></ul><p>That is,<em> our model learns to simultaneously predict, for a given input at a customer touchpoint, to what extent they will (1) engage with said content and (2) eventually buy large amounts of products.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PN15yuW4uFfIOuVInFDZxg.png" /><figcaption>ADAPT Model architecture: predicting customer LTV as a function of sequential encounters with beauty content. We add an additional term to the loss function, a prediction of customer engagement with content, to “teach” the model what content users find most compelling.</figcaption></figure><h4>Inference: Selecting Optimal Content for Each User</h4><p>Once trained, the power of this model is in <em>content selection via hypothetical content evaluation</em>: for any given customer and an associated touch point, we can query this model to determine <em>which piece of content, among a set of alternatives, is most likely to drive the highest time-discounted LTV.</em></p><p>Mathematically, this means that if we are e.g. selecting an article to send in an email, a product to feature on the home page, or many, many other things, we can select the best option as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nsZ9-dT7yISr51gsGeOFgA.png" /><figcaption>Algorithm for picking optimal content to drive time-discounted customer LTV</figcaption></figure><p>For performance reasons (avoiding an <em>O(N²) </em>or worse slowdown)<em> </em>and implementation simplicity, we limit ourselves to <a href="https://en.wikipedia.org/wiki/Greedy_algorithm"><em>greedily</em> <em>selecting</em></a><em> </em>content at any given time step. Furthermore, we find that a low-dimensional content representation (<em>s</em><strong> </strong>∈ R³²) and lightweight transformer-based architecture nearly attains the loss of much larger models, enabling fast inference times and easy model deployment on a range of hardware.</p><h3>In the Wild: Application and Results</h3><p>Our early results on mirabeauty.com demonstrate the power of this approach to personalization. We’ve experienced a 2.25x increase in conversion rates, and produced the world’s <a href="https://mirabeauty.com/intelligent-sampling">most efficient and transparent product sampling campaign</a> in the process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DSZOAMyYkSBbkzdW-LlK_g.png" /><figcaption>Adaptive personalization in practice: a few examples of adaptive personalization in the Mira AI, Inc. ecosystem, leading to an overall 2.25x increase in conversion</figcaption></figure><p>We’ve found that this personalization system has grown scalably with our greater technical ecosystem. Incrementally adding touch points comes at a minimal cost due to the flexible nature of the approach, making making this a sound technical investment.</p><h3>Beauty Shopping and Beyond</h3><p>The approach to adaptive personalization we’ve presented here is particularly powerful in a domain as multi-faceted as beauty shopping. We’ve received overwhelmingly positive feedback from our customer base and are excited to continue development, empowering beauty consumers to cut through the noise and find products they will love.</p><p>It’s clear that, just as AI is “eating software” more generally, adaptive personalization systems like ours will soon permeate a much broader domain of consumer digital products. We hope other organizations can benefit from and build upon our research.</p><p>All code and results produced in this post used Python, Segment, Apache Airflow, Pandas, PyTorch, and was developed with proprietary Mira AI, Inc. datasets. All images displayed above are solely for non-commercial illustrative purposes. Please reach out if you’re curious to learn more about our approach and implementation!</p><h3>Disrupt Big Beauty: Come work with us!</h3><p>If problems like this interest you, you’ll love working at Mira AI, Inc. We’re an agile and hard-working community of experienced hackers, data scientists and hardcore beauty enthusiasts disrupting beauty retail with a state-of-the-art commerce stack — <a href="https://angel.co/company/mirabeauty">and we’re hiring!</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AJPisiojXiVFT8ccPSVDoQ.png" /><figcaption>Disrupt beauty commerce with Mira AI, Inc’s passionate team and state-of-the-art tech stack</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1dd05d34ca8d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mathematical-beauty/adaptive-personalization-for-beauty-retail-1dd05d34ca8d">Adaptive Personalization for Beauty Retail</a> was originally published in <a href="https://medium.com/mathematical-beauty">MIRA BEAUTY™️</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deep Learning for Cosmetics]]></title>
            <link>https://medium.com/mathematical-beauty/deep-learning-for-cosmetics-2d1b427bbfa2?source=rss----167604512b2b---4</link>
            <guid isPermaLink="false">https://medium.com/p/2d1b427bbfa2</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[beauty]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[MIRA BEAUTY™️]]></dc:creator>
            <pubDate>Wed, 03 Jan 2018 19:15:34 GMT</pubDate>
            <atom:updated>2020-08-21T04:05:54.195Z</atom:updated>
            <content:encoded><![CDATA[<p>At <a href="http://mirabeauty.com/">MIRA BEAUTY</a>, we build tools that empower beauty enthusiasts to learn, gather inspiration and make informed buying decisions. In conversations with hundreds beauty consumers, we’ve learned that one of the foremost challenges that a consumer faces in finding the right products and techniques is identifying authentic and authoritative voices who can speak to their individual concerns.</p><p>In this blog post, we’ll demonstrate how we can use computer vision to solve a particularly poignant instance of this problem: finding influencers, images and videos that address a specific eye shape and complexion. Along the way, we’ll illustrate how three simple yet powerful ideas — <a href="https://en.wikipedia.org/wiki/Geometric_transformation">geometric transformations</a>, the <a href="https://www.coursera.org/learn/convolutional-neural-networks/lecture/HuUtN/triplet-loss">triplet loss function</a> and transfer learning — allow us to solve a variety of difficult inference problems with minimal human input.</p><h3>Background: Eye Shape and Complexion</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y8otjiRO_M9LTRK-WIEpTA.png" /><figcaption>A sample of useful eye classifications, from <a href="https://mirabeauty.com/smashbox">Smashbox</a></figcaption></figure><p>Finding the right products and techniques for your eyes is notoriously tricky — every individual has a unique shape and complexion. The same type of look (for instance, a <a href="https://www.makeup.com/smoky-eye-for-your-eye-shape">smoky eye</a>) can require wildly different techniques, depending on eye shape. While Birchbox and others have published <a href="https://www.birchbox.com/magazine/slideshow/shape-up-find-your-eye-shape-and-maximize-your-makeup">helpful visual guides</a>, one of the things we’ve learned from our community of beauty enthusiasts is that people typically seek advice from authentic, independent voices in their community and that finding quality advice from others with similar eye concerns challenging even for experts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/374/1*_4GwqhY8joTgL2zAY9dVnw.png" /><figcaption>Techniques with the same product can vary wildly across eye shapes. Adapted from <a href="https://www.makeup.com/smoky-eye-for-your-eye-shape">Makeup.com</a></figcaption></figure><p>But what if the characteristics of your eyes, along with the countless other facets that make you unique, seamlessly informed your beauty browsing and buying decisions?</p><h3>The Problem</h3><p>Let’s formalize the problem: given a set of images of faces, along with a small number of human-labeled images, (eye color, lid shape, etc.) find an <em>intuitive</em> visual similarity metric between eyes (“this beauty guru has eyes similar to yours!”) and a classifier that captures the human-labeled properties. In this blog post, we will focus on eye similarity; a follow-up will address classification tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I3__vravtPX12llD2pVp5Q.png" /><figcaption><a href="https://www.instagram.com/jackieaina/">Jackie Aina</a>, aka LaBronze James, #slaying with a smokey eye</figcaption></figure><p>Raw images are not suited well to either computing visual similarity or performing classifications. They can contain many superficial similarities (e.g. similar makeup applied, different skin tones washed out by strong lighting, etc.) that are unrelated to eye structure/complexion. Furthermore, raw images live in a high dimensional space, requiring a large amount of labeled training data for classification tasks. (See the <a href="https://en.wikipedia.org/wiki/Curse_of_dimensionality">curse of dimensionality</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H7iWtOoH8en5S8Yk_wHZ7g.png" /><figcaption>Similar eyes when pixels are compared directly; note that eyeshadow, lighting conditions and gaze direction are consistent, but eye color/complexion vary.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xyZMrVHwQ-uXjl6STfOzgQ.png" /><figcaption>The challenges of working with raw images: while clearly quite different to the human eye, these two images are relatively close when their raw data is compared. (Uses euclidean distance between raw pixels)</figcaption></figure><p>Our primary challenge lies in deriving low-dimensional and dense mathematical representations of eye images — known as <em>embeddings</em> — that capture the qualities that we care about and nothing more. That is, these embeddings should intentionally ignore:</p><ul><li>Eye pose/gaze direction</li><li>Specific lighting conditions (and insta filters, of course)</li><li>Whatever makeup is already applied</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9JSJaS7oFxVlFGEtOKXX-Q.png" /><figcaption>When eye embeddings are trained with the triplet loss function, we learn the ability to ignore superficial/irrelevant features (e.g. applied eyeshadow/eye pose in the images above) and focus on what matters.</figcaption></figure><h3>Image Normalization via Projective Transform</h3><p>We can eliminate an entire class of superficial similarities with a simple preprocessing step: the <a href="https://ags.cs.uni-kl.de/fileadmin/inf_ags/3dcv-ws11-12/3DCV_WS11-12_lec04.pdf">projective transform</a>.</p><p>While cropped images of eyes will exhibit many obvious structural differences (e.g. the eye isn’t at the center, or is rotated due to head tilt, etc.), the projective transformation allows us to “warp” images such that the same eye landmarks are guaranteed to occupy the same coordinates.</p><p>This is explained well in the <a href="http://scikit-image.org/docs/dev/auto_examples/xx_applications/plot_geometric.html">scikit-image documentation</a>. With a little bit of linear algebra mathemagic, we can warp an image such that a set of points map to a new, desired shape, rotating and stretching the image in the process:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*FZP0tuTRHax5kBwDJ9rgOg.png" /><figcaption>Using a projective transformation, we can warp the top image such that the four red points become a rectangle, “straightening” the text. We apply a similar method to normalize images of eyes. (from the scikit-image documentation)</figcaption></figure><p>We can apply the same technique to normalizing eye images, rotating/stretching them to a more consistent form. We detect facial landmarks using <a href="http://blog.dlib.net/2014/08/real-time-face-pose-estimation.html">dlib</a>, crop the eyes, and warp them to ensure alignment and consistency. This preprocessing step significantly improves our ability to converge on embeddings that are invariant to head tilt/pose. (A detailed overview of this method, when applied to general face alignment, is <a href="http://blcv.pl/static//2017/12/28/demystifying-face-recognition-iii-face-preprocessing/">available here</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yUCn72N1-119dQg7oMBZlQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WkZuo7uaMMErx9WCfCdu3Q.png" /><figcaption>Image normalization: detect facial landmarks, crop the eyes, then apply a projective transformation to “warp” the eyes into a standard location. Note that this also allows us to align both eyes from a single face.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s_cWdqU3IairBRzsqF8_UQ.png" /><figcaption>Samples from the image preprocessing pipeline</figcaption></figure><h3>Representation Learning with Triplet Loss</h3><p>The warped images, when directly compared, still exhibit many superficial similarities, including gaze direction and similar applied makeup. Deep learning to the rescue!</p><p>Simply put, we will train a <a href="http://cs231n.github.io/convolutional-networks/">convolutional neural network</a> (CNN) to consume eye images, then output vectors that are <em>more similar for the same individual than they are between different people</em>. The network will learn to output stable/consistent representations of an individual’s eyes across a variety of contexts.</p><p>The hero, of course, is the aforementioned <a href="https://www.coursera.org/learn/convolutional-neural-networks/lecture/HuUtN/triplet-loss">triplet loss function</a>. In the <a href="https://docs.chainer.org/en/stable/reference/generated/chainer.functions.triplet.html">Chainer implementation</a>, the exact formula is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VDOSM4zQMQxMkUBezWAkiw.png" /></figure><p>This specifies that our model’s loss, and our optimization objective, will decrease when it places two embeddings of a specific individual (the <em>anchor </em>and the <em>positive sample) </em>closer than the anchor and an unrelated individual. (The <em>negative sample</em>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OkHo9m1mW-hITHC0iqd6MA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Zl2gpZg-L5rpjh4bRLbng.png" /><figcaption>Network architecture: we pass images through a convolutional neural network, outputting dense vectors, then reward the network for embedding images of the same individual (the anchor and positive embeddings) in closer proximity than the positive and negative sample</figcaption></figure><p>When applied to images of eyes, we find that the resultant embeddings are excellent indicators of similar eye structure and complexion between single images, while robust to superficial differences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/1*zrd0uhKMKlFwlyMHTsE4-Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*6k4fsiqouAKZgOp6G2vQ5A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*Oz75WwZeZj2BELYoyp8NaA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*Q2mjdC2WNuy_xa0WGSUE6w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*kwLBziBbnzZVfxvqZs7rwg.png" /><figcaption>Results visualization: groups of samples with embeddings in near proximity to each other.</figcaption></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F3jIPpcK89dU%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D3jIPpcK89dU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F3jIPpcK89dU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/50beb4493e7290f3c2564e08615e4c6c/href">https://medium.com/media/50beb4493e7290f3c2564e08615e4c6c/href</a></iframe><p>Our approach here is similar to (and inspired by) Google’s <a href="https://arxiv.org/abs/1503.03832">FaceNet</a>, which produces face-level image embeddings via image warping/alignment and the triplet loss function.</p><h3>Combining Embeddings for Person-level Similarity</h3><p>With a simple tweak, we can adapt our embeddings to support person-level eye representations as well — abstracting away any noise present in individual frames.</p><p>Using our pre-trained weights from the network above, we now adopt a new loss function that rewards the network for placing <em>averages of sets of embeddings from a given individual</em> in close proximity, relative to an unrelated individual, as illustrated below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TwGu1HcbavHobzh5_2Nlfw.png" /><figcaption>Using pre-trained weights from our earlier network, we can adapt the network to a modified task — making eye embeddings that can be combined through averaging — and see it converge quickly. This is known as transfer learning.</figcaption></figure><p>This forms embeddings that can be combined into a more holistic representation of an individual’s eye. While this is a complicated network architecture, it converges quickly due to our usage of <a href="http://cs231n.github.io/transfer-learning/">transfer learning</a>: appropriating our earlier, related embeddings, trained for single-image similarity. When applied across our dataset of beauty gurus, we find that it produces embeddings that capture fine-grained similarity between individuals, demonstrated below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pxaW8x7XOzfH9-sSM4Zr9w.png" /><figcaption>Person-level embeddings: each row contains a set of beauty gurus with aggregate eye embeddings tightly clustered in space.</figcaption></figure><h3>Conclusion and Future Work</h3><p>Having arrived at high-quality mathematical representations of eyes, in single images and aggregated across multiple views of an individual, we can perform image similarity and retrieval tasks. (“Check out these tutorials by people who speak to your concerns”) This is made possible by diligent image preprocessing with the projective transformation, a clever loss function and transfer learning.</p><p>As we will see in a future blog post, these embeddings make several *supervised* machine learning tasks (classifying eye shape, regressing onto eye color, etc.) straightforward and trivial. In addition, when combined with a more naive analysis, they allow us to perform more sophisticated image retrieval tasks, such as searching for similar makeup compositions.</p><p>All code and results produced in this post used NumPy, SciPy, Matplotlib, Chainer, <a href="http://dlib.net/">dlib</a>, and the <a href="https://github.com/jayhack/squeezenet-chainer">SqueezeNet</a> architecture. All images displayed above are solely for non-commercial illustrative purposes. Feel free to reach out to learn more about our approach and implementation!</p><h3>Come work with us!</h3><p>If problems like this interest you, you’ll love working at <a href="http://mirabeauty.com/">MIRA BEAUTY</a>. We’re an agile and hard-working community of experienced hackers, data scientists and hardcore beauty enthusiasts — and we’re hiring :). Come help us organize the knowledge of the hive mind and revolutionize e-commerce!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d1b427bbfa2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mathematical-beauty/deep-learning-for-cosmetics-2d1b427bbfa2">Deep Learning for Cosmetics</a> was originally published in <a href="https://medium.com/mathematical-beauty">MIRA BEAUTY™️</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>