<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Matthew Moellman on Medium]]></title>
        <description><![CDATA[Stories by Matthew Moellman on Medium]]></description>
        <link>https://medium.com/@moellman?source=rss-3ee419b4b108------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*mwnGz4XJLWAPEvHm8yfWLw.png</url>
            <title>Stories by Matthew Moellman on Medium</title>
            <link>https://medium.com/@moellman?source=rss-3ee419b4b108------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 21:04:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@moellman/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Training a Tiny Pix2Pix GAN for Snapchat]]></title>
            <link>https://heartbeat.comet.ml/training-a-tiny-pix2pix-gan-for-snapchat-c6062e86204c?source=rss-3ee419b4b108------2</link>
            <guid isPermaLink="false">https://medium.com/p/c6062e86204c</guid>
            <category><![CDATA[lens-studio]]></category>
            <category><![CDATA[pix2pix]]></category>
            <category><![CDATA[snapml]]></category>
            <category><![CDATA[heartbeat]]></category>
            <category><![CDATA[mobile-ml]]></category>
            <dc:creator><![CDATA[Matthew Moellman]]></dc:creator>
            <pubDate>Thu, 15 Jul 2021 14:02:00 GMT</pubDate>
            <atom:updated>2021-10-18T16:10:25.104Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*Lxsriko_sDc563pBAEkA7w.png" /></figure><h4>Performing image-to-image translation to transform outlines of cat faces into realistic cats with a model 95% smaller than Pix2Pix</h4><p><a href="https://www.snapchat.com/unlock/?type=SNAPCODE&amp;uuid=17b8cff4e3e048bb80a8d930ecabf3f5&amp;metadata=01">Cat Face</a> is the latest SnapML-powered Lens built by the team at Fritz AI. The Lens lets users draw photo-realistic cats and wear the result like a mask. If you haven’t already, try it out for yourself here:</p><p><a href="https://www.snapchat.com/unlock/?type=SNAPCODE&amp;uuid=17b8cff4e3e048bb80a8d930ecabf3f5&amp;metadata=01">Try out Cat Face by Fritz AI</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*ByVJ0bGe0ZHS-os2dAwbxw.png" /><figcaption><a href="https://www.snapchat.com/unlock/?type=SNAPCODE&amp;uuid=17b8cff4e3e048bb80a8d930ecabf3f5&amp;metadata=01">Snapcode Link</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*HXtmKnqdoHnNsmAg5QOsRg.gif" /></figure><h3>Introduction: Model and Deployment Target</h3><p>In this post, I’ll cover the process I went through to train a modified Pix2Pix GAN and publish it in a Snapchat Lens. I modified the model architecture to satisfy a set of constraints unique to Snapchat, cleaned a dataset of cat face images that I used to train the model, and ran the model in an ML Component in a Snapchat Lens.</p><h4>The Pix2Pix Model</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XXxqNWrKYJ36-04aaqUR_w.jpeg" /><figcaption>Example input/output for <a href="https://affinelayer.com/pixsrv/">Christopher Hesse’s edges2cats model</a>.</figcaption></figure><p>I was inspired to reproduce this <a href="https://affinelayer.com/pixsrv/">interactive cat generator</a> made with the Pix2Pix model described in the paper titled <a href="https://phillipi.github.io/pix2pix/">Image-to-Image Translation with Conditional Adversarial Nets</a>. The model takes a sketch of a cat and produces a photo-realistic version as output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GePb4xSQ2qY9VIo2TrAuTg.jpeg" /></figure><p>The Pix2Pix generator architecture is similar to the <a href="https://heartbeat.fritz.ai/deep-learning-for-image-segmentation-u-net-architecture-ff17f6e4c1cf">U-Net architecture</a>. It features an encoder/decoder network with skip connections. I implemented Pix2Pix in Keras because ultimately we need to convert it to TensorFlow Lite to import it into Lens Studio and put it in a Snapchat Lens. I found <a href="https://www.tensorflow.org/tutorials/generative/pix2pix">this official TensorFlow tutorial</a> to be a useful reference while implementing the model, although I ended up making quite a few modifications, which I’ll get into later.</p><h4>SnapML Constraints</h4><p>SnapML places a few constraints on the Pix2Pix GAN architecture. Only certain layer types and operations are supported and the final model needs to be small enough and fast enough for use in a Snap Lens.</p><p><strong>Model File Format</strong></p><p>There are a few supported file formats for SnapML: Open Neural Network Exchange (.onnx), TensorFlow Frozen Graph (.pb), and TensorFlow Lite (.tflite). Of those, TensorFlow Lite is the easiest to work with in my opinion, but if you are developing a model in pytorch, ONNX may be the way to go.</p><p><strong>Model File Size</strong></p><p>The model file (architecture and weights) must be less than 10 MB. Other Lens assets (e.g. textures, 3D models) do not contribute to this limit.</p><p><strong>Frame Rate</strong></p><p>According to the official performance requirements, iOS Lenses should run at 27 FPS at a minimum on iPhone 6 and higher, and Android Lenses should run at 15 FPS at a minimum on Samsung Galaxy S6 and higher.</p><p><strong>Neural Network Layer Restrictions</strong></p><p>Not all neural network layers are supported in SnapML, and some layers are only supported for certain hardware like the CPU or GPU. Sometimes there are issues with importing models into Lens Studio even though all the layers are supposed to be supported — you might have to hack around some unexpected compatibility issues (more on that later).</p><p><strong>Compatibility Drift</strong></p><p>Unfortunately, Lens Studio updates are not consistently backwards compatible with respect to SnapML — the same model that imports into the current version of LS might not import after updating, so be careful if your project timeline spans Lens Studio releases.</p><h3>Model Architecture</h3><p>In this section, I’ll go over the modifications I made to the Pix2Pix generator architecture to satisfy the model size and neural network layer constraints outlined in the previous section.</p><p><strong>Alpha Hyperparameter</strong></p><p>The Pix2Pix model as described in the paper ends up being about 200 MB, which is 20x too big for Snapchat. As a first step towards reducing the model size, I borrowed the concept of a model shrinking factor (called alpha) from MobileNet. This parameter controls the number of filters used in a convolution layer. For example, if the default number of filters for the first three layers were 32, 64, and 128, an alpha value of 0.5 would mean that those layers would instead use 16, 32, and 64 filters, respectively. By reducing the number of filters in the convolution layers, the number of parameters in the model decreases, and that will help us get the model size closer to the required 10 MB from the original 200 MB. In my early experiments, I found that the alpha parameter alone was not enough to produce a flexible model under the 10 MB limit, so I looked for other opportunities to squeeze performance out of the model while shrinking it. It turns out that separable convolutions help a lot in that regard.</p><p><strong>Separable Convolutions</strong></p><p>There are two types of convolutions used in Pix2Pix. The encoder portion of the network uses regular convolutions with a stride of 2 for downsampling the input image. The decoder uses transposed convolutions to upsample the encoder output. Separable convolutions — a factorized type of convolution layer that cuts down on the number of parameters needed — can be used in place of both regular and transposed convolutions. For regular convolutions, separable convolutions are a drop-in replacement, and in the Pix2Pix case, we just need to make sure the stride is set to 2 to perform the downsampling properly. For transposed convolutions, which are essentially an upsampling layer with learned weights, we can replace them with a nearest-neighbor upsampling layer followed by a separable convolutional layer. The end result is a network with a much smaller number of parameters that still retains enough flexibility to learn what a cat looks like. By using separable convolutions in place of regular convolutions and transposed convolutions, I was able to shrink the model down to 20 MB.</p><p>One other benefit of using separable convolutions in this model is that I was able to sidestep an unexpected layer compatibility issue. According to <a href="https://lensstudio.snapchat.com/guides/machine-learning/compatibility/">this list of compatible layers</a>, I should be able to use transposed convolutions in the model, but this was broken for me. Sometimes the low-level representation of neural network operations in the model file (.tflite in this case) are different from what Lens Studio is expecting. These representations are difficult to modify and debug. Lens Studio could be more flexible in handling issues like these to ensure a robust and complete compatibility with the layers they claim to support, but alas, separable convolutions are a good workaround for this model.</p><p><strong>Weight Quantization</strong></p><p>Finally, another technique that is useful for shrinking models is weight quantization. Typically, each weight and bias in a neural network is stored as a 32-bit floating point number, which is often way more precise than necessary. Converting the weights to 16-bit floats cuts the size of the network in half, while usually preserving the predictive performance very well. That takes us from 20 MB down to 10 MB, which is the largest the model can be in a Snapchat Lens. In the case of SnapML, we actually get the weight quantization for free when the model is imported into Lens Studio.</p><h3>Training Data</h3><p>Pix2Pix requires paired training data where each example consists of one image from domain A and a corresponding image from domain B. The model’s task is to translate an input image from domain A into domain B. In general, there is a wide range of choices for these two domains, but the output domain often consists of real images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J8NnXm-H1lW2SsEtIbMNUA.jpeg" /><figcaption>edges2shoes examples from the Pix2Pix paper</figcaption></figure><p>When the output domain is made up of real images, you may be able to generate the corresponding image in the input domain without human labeling. For example, if you were trying to train a model to translate a black-and-white image to a color image, you could start with a dataset of full color images and convert them to grayscale to generate the image pairs. In our case, we want to generate realistic cat faces from a simple outline. I’ll describe how I built that dataset in the following sections.</p><p><strong>Images of Cat Faces (Output Domain)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*YyuS9J974nqzc_1clI8YbQ.png" /><figcaption>Normalized images of cat faces.</figcaption></figure><p>For training data, I used <a href="https://www.kaggle.com/crawford/cat-dataset">this Kaggle dataset of annotated cat images</a> with nine keypoint annotations per cat: left eye, right eye, mouth, and three keypoints for each ear.</p><p>Here are the automated preprocessing steps I followed:</p><ol><li>Rotate the image so that they eyes lie on a horizontal line</li><li>Crop the image to the cat’s face (discard if the face is too small)</li><li>Pass the image through a <a href="https://github.com/xuebinqin/U-2-Net">pre-trained salient object detection model</a> to remove the background (discard if two few pixels are segmented)</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*-UtahXeVQYRhItT2aJYnHQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/1*V_FjPYrIatWD495CTyTz5g.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/1*N-URocii5d1hjAb9-_VLuQ.png" /><figcaption>Rotated, cropped, and background-removed images.</figcaption></figure><p>The third step is necessary because we are training a tiny model and we want it to focus its limited resources on generating cat faces, not cat faces and backgrounds. This is different from models like the one used to make <a href="http://thispersondoesnotexist.com/">thispersondoesnotexist.com</a> because those models have a much higher capacity and are trained with much larger datasets so they have no problem generating a background behind the face.</p><p>After going through these automated data processing steps, I manually inspected the remaining images and removed any undesirable images — ones that are blurry, have bad segmentation masks, closed eyes, objects in front of the cat, etc.</p><p><strong>Edge Images (Input Domain)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tIr0WHVNfrKrxRjWtOEtxg.jpeg" /><figcaption>edges2handbags examples from the Pix2Pix paper</figcaption></figure><p>The edge images from the applications in the Pix2Pix paper are quite intricate. The authors use <a href="https://arxiv.org/abs/1504.06375">Holistically-Nested Edge Detection (HED)</a> with some additional post-processing steps to reduce RGB images to black-and-white edge images. For the cat-generating Lens, the inference-time edge images are going to be hand-drawn by Snapchat users, so we want the training edge images to mimic something that can be easily drawn with a finger.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/352/1*KxaweA8m5Zg8f6qsEW4gQg.png" /><figcaption>A paired training example of a cat face with a transparent background and the corresponding edge image.</figcaption></figure><p>To generate edge images with the outline of the cat, I first performed Holistically-Nested Edge Detection followed by Canny edge detection on the HED output. There is probably a less computationally expensive way to get similar output, but my dataset was small enough that I could run this on my laptop in a reasonable amount of time. Once I had outlines of the cats, I used the eyes and mouth keypoint annotations to draw circles for the eyes and a triangle for the nose.</p><p><strong>Data Augmentation</strong></p><p>There are a few important considerations regarding data augmentation for Pix2Pix. The first is that you should make sure that geometric augmentations (e.g. cropping, flipping, rotating, etc.) are applied to pairs of images so that real images and the corresponding edge images are augmented in the same way. The second is that you shouldn’t perform augmentations that produce examples outside of the distribution that you want to generate — augmentations such as aggressive color shifts, brightness adjustments, or blurring may be beneficial for other types of computer vision models, but we don’t want the generator to learn to produce images with those qualities.</p><h3>Model Training</h3><p>In this section, I’ll cover an assortment of hyperparameter adjustments, “GAN hacks”, and other model training considerations that were interesting to me.</p><p><strong>Generator Encoder/Decoder and Discriminator Alpha Values</strong></p><p>Earlier, I described the alpha hyperparameter that we can use to control the number of filters used in the convolutional layers of the network. After switching to separable convolutions, I found that I could use an alpha value as high as 1.2 while remaining under the 10 MB limit. The separable convolutions aren’t as flexible as the regular convolutions and transposed convolutions they are replacing, so I still needed to squeeze out more performance. I ended up configuring the model in a way where I could specify a one alpha value for the encoder portion of the generator and another for the decoder portion. I settled on an alpha of 0.4 for the encoder (based on the assumption that the model would need fewer parameters to encode the simpler edge images) which left me with an alpha of 2.0 for the decoder. The architecture doesn’t break with these asymmetric alpha values because the resolution of the feature maps connected by the skip connections does not change. It’s also worth noting that since I restricted the generator by using separable convolutions and making it smaller, the discriminator also needs to be restricted to help keep the GAN balanced. I settled on an alpha of 1.0 for the discriminator.</p><p><strong>Loss Functions</strong></p><p>I experimented with a couple different loss functions: binary cross-entropy (BCE), and Wasserstein loss (W-loss). The Pix2Pix paper uses BCE loss, but GANs trained with BCE loss are susceptible to vanishing gradients which occur when the discriminator is too confident (i.e. when it predicts values close to 0 or 1). Using W-loss can help with the vanishing gradient problem because, unlike BCE loss, it does not require the discriminator to have a sigmoid output activation function (which is flat at the extremes, hence the vanishing gradients). I tried training the Pix2Pix model with W-loss, but it was still unstable so I decided to stick with BCE and implement some other gan hacks instead.</p><p><strong>GAN Hacks</strong></p><p>GANs are made up of two models — a generator and a discriminator — that compete with one another. Initially, the generator produces random noise, and the discriminator makes a random guess as to whether an image is real or fake. In theory, both models should eventually improve to the point where the generator can produce realistic looking examples. In practice however, training GANs is difficult in part because the generator and discriminator can easily become unbalanced with respect to one another. If the discriminator becomes too strong relative to the generator, it can no longer pass useful feedback to the generator (in the form of gradients), so the training plateaus.</p><p>My training runs weren’t producing good output, so I tried to slow down the discriminator by adding noise to the images and the labels as recommended in <a href="https://github.com/soumith/ganhacks">this collection of GAN hacks</a>.</p><p><strong>Add Noise to Real and Fake Images</strong></p><p>If the discriminator learns too quickly, that means it can easily separate the distribution of fake images produced by the generator from the distribution of real training images. The intuition behind adding noise to the images is that it makes those two distributions more similar, which makes the discriminator’s job harder and buys time for the generator to catch up. I experimented with adding random normal noise (that linearly annealed to no noise after 100 epochs or so) to the images before feeding them into the loss functions, and while that did stop the discriminator from dominating early on, the resulting cats didn’t look great and the model suffered from mode collapse. However, I had better luck with label smoothing.</p><p><strong>Label Smoothing</strong></p><p>The idea behind label smoothing is to discourage the discriminator from being too confident with its predictions. If the discriminator is too confident, you will run into vanishing gradients, the generator will stop learning, and the discriminator loss will drop close to 0. The discriminator is typically trained on batches of real images with a label of 1.0 and fake images with a label of 0.0. When applying label smoothing, we soften the binary labels by sampling a random float from a range instead (e.g. 0.0–0.3 for fake images, 0.7–1.0 for real images). I found that performing one-sided label smoothing worked best to stabilize training for the task at hand. I kept the fake cat labels at 0.0, and sampled from 0.8–1.0 for the real cats. This did stop the discriminator loss from going to 0, but that doesn’t guarantee that the generator is learning (although, it turns out it was). Ultimately, I found visualizing model output during training to be more helpful than looking at loss curves to monitor training progress.</p><p><strong>Visualization in TensorBoard</strong></p><p>Watching the values of the loss functions of a GAN during training can offer some insight, but the actual values depend on the changing state of the generator and discriminator so there isn’t really a concept of a “good” loss value. Visualizing the model output is a good idea in general, but especially for GANs whose loss values are not highly interpretable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hvtFMdnG0LY4GVhFuGPwnQ.jpeg" /><figcaption>Input edge image, ground truth, model output after 10 epochs. The output resembles a cat, but it’s low-fidelity.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XJHxLy6ZW03mar3HXA-P0A.jpeg" /><figcaption>Input edge image, ground truth, model output after 200 epochs. The ears look nice, but there is a third eye.</figcaption></figure><h3>Building a Lens</h3><p>Once I had a model that would load and run in an ML Component in Lens Studio, the next task was to build a Lens that would allow Snapchat users to interact with the model. At Fritz AI, we already built a <a href="https://twitter.com/fritzlabs/status/1370029209574281225">Clone to Face Lens</a> (that extended our <a href="https://heartbeat.fritz.ai/snapchat-lens-recipe-clone-76de4a66a8fe">Clone Lens</a>) that creates a cutout of an object and puts it on your face. I decided to go with a similar UX, so that meant I just needed to implement the user input part of the Lens (i.e. enable users to draw edge images that the model will turn into realistic cats). A user starts out by drawing the outline of a cat head, and once they are happy with the model output, they can turn it into a sticker and put it on a human face. I won’t get into too much detail on how I built the Lens, but I’ll go over a few design choices I made.</p><h4>Design Choices</h4><p><strong>Inference Frequency</strong></p><p>Initially, I planned to have the Lens run the model only once when the user was done editing the cat outline, but when I first tested the model, I forgot to configure that and the model ran on every frame by default. This ended up being a happy accident — it was interesting to see the model run in real time while editing the cat head outline, eyes, and nose. The Lens did not pass the FPS requirements on lower-end phones, so I lowered the inference frequency to 8 Hz which was close enough to real-time without being too choppy.</p><p><strong>RGB vs. RGBA</strong></p><p>Pix2Pix outputs 3-channel RGB images by default. I was able to train a model that produced RGB cat faces with white backgrounds, but I didn’t want the white background to show up in the Lens. To get rid of the background, I used a <a href="https://support.lensstudio.snapchat.com/hc/en-us/community/posts/360000014683-Chroma-Key-Shader-Material">chroma key shader</a>, but that either left too many white pixels near the edge of the cat or removed details within the cat (like reflections in the eyes and white fur). To get around this, I modified the model architecture to produce 4-channel RGBA images and trained a new model using images of cats with transparent backgrounds.</p><p><strong>Trace vs. Freehand</strong></p><p>Once the model was in a working version of the Lens, I personally was able to draw edges that resulted in realistic-looking cats, but other people that tested it for me had difficulty. I realized that I wasn’t really a typical user of the Lens because I had been staring at the training data so I knew what types of edges to draw to get the model to work well. Perhaps a larger model would have been able to generate high quality cats on a wider range of inputs, but that was not an option because of the strict 10 MB limit, so I opted for a UX change to make the model more user-friendly. I gave users the option to trace some example edges to point them in the right direction. This ended up helping a lot because the edges the model was trained on are quite different from what a typical person would draw if you asked them to sketch a cat (as you can see in <a href="https://quickdraw.withgoogle.com/data/cat">Google’s <em>Quick, Draw!</em> dataset</a>).</p><h4>Future Iterations</h4><p><strong>CycleGAN</strong></p><p>One could potentially leverage the <em>Quick, Draw!</em> dataset to build a similar model using CycleGAN. Like Pix2Pix, CycleGAN performs image-to-image translation, but unlike Pix2Pix, CycleGAN does not require paired images for training examples. Instead, CycleGAN makes use of two “piles” of images, so we could reuse the cat images I already have for one domain, and sketches of cats from <em>Quick, Draw! </em>for the other domain.</p><p><strong>User Input Mechanism</strong></p><p>The method I went with to capture user input (i.e. the edges they drew) involves moving a sphere around based on where they touch the screen. The sphere is invisible to the user but visible to a camera within the Lens, which allows the path that the user traced to be rendered to a texture that is ultimately used as input for the model. The drawback to this method is that if the user moves their finger too quickly across the screen, there will be gaps in the curve they make. Another method that I explored briefly was to create a <a href="https://support.lensstudio.snapchat.com/hc/en-us/community/posts/115020420083-Creating-a-Procedural-Mesh-Using-Mesh-Builder">procedural mesh</a>. Using a procedural mesh would solve the gap problem associated with the other method I described, but the procedural mesh method is much more complicated to implement and comes with problems of its own. I wish Snap would make the marker/pen drawing UI accessible to Lens Creators — that would make the process of capturing a user’s drawing input a whole lot less painful.</p><h3>Conclusion</h3><p>I hope you enjoyed this post! I ended up in a different place than I thought I would at the beginning, but I’m happy with how things turned out. The problems on the ML engineering side of things were the most interesting and fun to work on for me. If you are a data scientist or ML engineer looking to do something similar, I recommend pairing up with someone who has experience building Lenses, or at least checking out <a href="https://heartbeat.comet.ml/snapml-and-machine-learning-in-lens-studio-cca5677b2945">this list of Lens Studio resources</a>. I was lucky enough to have some teammates that helped me a lot in that part of the process. Happy coding, and be patient with your GANs!</p><p><em>Editor’s Note: </em><a href="https://heartbeat.comet.ml/"><em>Heartbeat</em></a><em> is a contributor-driven online publication and community dedicated to providing premier educational resources for data science, machine learning, and deep learning practitioners. We’re committed to supporting and inspiring developers and engineers from all walks of life.</em></p><p><em>Editorially independent, Heartbeat is sponsored and published by </em><a href="http://comet.ml/?utm_campaign=heartbeat-statement&amp;utm_source=blog&amp;utm_medium=medium"><em>Comet</em></a><em>, an MLOps platform that enables data scientists &amp; ML teams to track, compare, explain, &amp; optimize their experiments. We pay our contributors, and we don’t sell ads.</em></p><p><em>If you’d like to contribute, head on over to our</em><a href="https://heartbeat.fritz.ai/call-for-contributors-october-2018-update-fee7f5b80f3e"><em> call for contributors</em></a><em>. You can also sign up to receive our weekly newsletters (</em><a href="https://www.deeplearningweekly.com/"><em>Deep Learning Weekly</em></a><em> and the </em><a href="https://info.comet.ml/newsletter-signup/"><em>Comet Newsletter</em></a><em>), join us on</em><a href="https://join.slack.com/t/fritz-ai-community/shared_invite/enQtNTY5NDM2MTQwMTgwLWU4ZDEwNTAxYWE2YjIxZDllMTcxMWE4MGFhNDk5Y2QwNTcxYzEyNWZmZWEwMzE4NTFkOWY2NTM0OGQwYjM5Y2U"><em> </em></a><a href="https://join.slack.com/t/cometml/shared_invite/zt-49v4zxxz-qHcTeyrMEzqZc5lQb9hgvw"><em>Slack</em></a><em>, and follow Comet on </em><a href="https://twitter.com/Cometml"><em>Twitter</em></a><em> and </em><a href="https://www.linkedin.com/company/comet-ml/"><em>LinkedIn</em></a><em> for resources, events, and much more that will help you build better ML models, faster.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c6062e86204c" width="1" height="1" alt=""><hr><p><a href="https://heartbeat.comet.ml/training-a-tiny-pix2pix-gan-for-snapchat-c6062e86204c">Training a Tiny Pix2Pix GAN for Snapchat</a> was originally published in <a href="https://heartbeat.comet.ml/">Heartbeat</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating an Animated 3D Phoenix Lens in Lens Studio]]></title>
            <link>https://heartbeat.comet.ml/creating-and-animated-3d-phoenix-lens-in-lens-studio-b63889a53e3f?source=rss-3ee419b4b108------2</link>
            <guid isPermaLink="false">https://medium.com/p/b63889a53e3f</guid>
            <category><![CDATA[heartbeat]]></category>
            <category><![CDATA[augmented-reality]]></category>
            <category><![CDATA[lens-studio]]></category>
            <category><![CDATA[3d-modeling]]></category>
            <category><![CDATA[snapml]]></category>
            <dc:creator><![CDATA[Matthew Moellman]]></dc:creator>
            <pubDate>Tue, 09 Feb 2021 14:47:41 GMT</pubDate>
            <atom:updated>2021-10-19T15:12:02.900Z</atom:updated>
            <content:encoded><![CDATA[<h4>Working with 3D animation, the Refinement Pack, particle emitters, and more</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k7aRXuPOTH0tQW7fYZ3zxg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@prodigeon?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Laith Abushaar</a> on <a href="https://unsplash.com/s/photos/phoenix-dragon?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>In this post, I’ll explore how I made a Lens that features an animated phoenix flying through an AR scene anchored to the ground. This is the finished Lens:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*WZzQ74067Lpc9LSD" /></figure><p>To get there, we will leverage the<a href="https://support.lensstudio.snapchat.com/hc/en-us/articles/360052179692-Refinement-Pack"> Refinement Pack</a> and<a href="https://lensstudio.snapchat.com/templates/world/particles/"> Particles</a> template from Lens Studio, as well as a 3D phoenix asset from Sketchfab.</p><p>Specifically, this post will cover:</p><ul><li>Moving an object along a 3D path through the scene using a Prefab from the Refinement Pack</li><li>Applying a texture to a 3D model of a phoenix downloaded from Sketchfab</li><li>Working with particle emitters to add a stationary column of smoke, embers throughout the scene, and a bokeh effect that follows the phoenix</li></ul><p>Let’s dive in!</p><h3>Project Setup</h3><p>Open up a blank project in Lens Studio, and add a Device Tracking component to the default camera, with Tracking Mode set to Surface. This will allow us to anchor the scene on a flat surface, like the ground or the floor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/0*DG561bXvyWiWkqqE" /></figure><p>More details on tracking modes can be found<a href="https://lensstudio.snapchat.com/guides/general/tracking/tracking-modes/"> here</a>.</p><h3>Move an object along a path</h3><p>We’ll start by loading a Prefab that moves an object along a specified path. Eventually we’ll edit the path and hook it up to the phoenix so that it flies along the path, but for now we are just setting up the scaffolding.</p><p>Lens Studio’s<a href="https://support.lensstudio.snapchat.com/hc/en-us/articles/360052179692-Refinement-Pack"> Refinement Pack</a> provides a handful of helper scripts to let you hit the ground running when working on a custom Lens. We’ll be making use of the Move Along Path helper.</p><p>Once we’ve downloaded and unzipped the Refinement Pack, drag and drop the Refinement Pack.lso into the Resources panel. You’ll find a prefab named Move Along Path [EDIT_ME] nested under Refinement Pack / Prefabs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/0*ozTQy56BSQOonKop" /></figure><p>Go ahead and drag that to the Objects panel, and you should see a rotating cube moving along a path outlined by four spheres.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/808/0*jgp0VP3vZ86VbSR5" /></figure><h3>Animate the phoenix</h3><p>Now we’re ready to add the phoenix to the Lens. Go ahead and download this<a href="https://sketchfab.com/3d-models/phoenix-bird-844ba0cf144a413ea92c779f18912042"> animated phoenix</a> 3D asset from Sketchfab (in the FBX format), unzip it, and drop it into the folder into the Resources tab in Lens Studio. In the phoenix-bird folder, there is a prefab called “fly”. Drag that to the objects panel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/0*ETrGMk9VgPDTgxRN" /></figure><h3>Scale the phoenix</h3><p>The phoenix is quite large, so we need to scale it down so it will fit in the scene. Select the “fly” prefab in the Objects panel to bring it up in the Inspector panel. In the Transform component at the top, set the scale to 0.05 for x, y, and z.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/0*QkC6bxoiuPA1B7nw" /></figure><p>Now you should be able to see the phoenix flapping its wings in the Preview panel.</p><h3>Apply a texture to the phoenix</h3><p>The phoenix asset we downloaded came with some textures in the form of PNG files that we can apply to the 3D meshes to add some color to the object. The asset came with four PNG textures, but I ended up only using two of them. From my experience, it often takes a little guesswork to match up textures to the appropriate meshes. The phoenix asset came with two materials — one for the body, another for the wings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/0*7KdKHlX7EQ5ATIAO" /></figure><p>Based on these filenames, I tried matching up the Tex_Ride_FengHuang_01a_D_A texture with the MatI_Ride_FengHuang_01a material, and similarly for Tex_Ride_FengHuang_01<strong>b</strong>_D_A and MatI_Ride_FengHuang_01<strong>b</strong>.</p><p>To apply the textures to the materials, select one of the materials in the Resources panel to open it in the Inspector panel. Then, set the Blend Mode to Normal, check the Base Texture box, and drag the corresponding PNG to the Texture input.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/0*digGpLvKR28RM_7w" /></figure><p>I ended up using only two of the four textures that came with the phoenix asset. As my coworker put it, it’s like I assembled a piece of IKEA furniture but had some parts left over at the end, and I’m not sure where they go.</p><h3>Move the phoenix along a path</h3><p>In this section, we’ll focus on moving the phoenix along a path through the scene to make it look like it’s flying. It will help to visualize the Lens if you switch to the back camera (the button to do that is on the top right of the Preview panel).</p><p>The first thing to do is attach the phoenix to the Move Along Path refinement example. Select Move Along Path [EDIT_ME] in the Objects panel to bring it up in the Inspector, then drag the “fly” prefab to the input Value 0 for Movable Objects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/0*pyE85lcewkBr-9vH" /></figure><p>At this point, the phoenix should be moving (and rotating wildly — we will fix that soon) along the path. Next, we will adjust the path so that it flies along a square path. The Move Along Path refinement has a Transform on it that we need to update. We will set the Position and Rotation transforms for the x, y, and z directions all to 0, because any transformations we do on objects nested under this refinement will be relative to this base transformation.</p><p>To arrange the points (Path Example 1, 2, 3, 4) that make up the flight path for the phoenix, select them one-by-one and edit their position values in the Transform component. Set the x, y, z positions to (40, 50, 0), (0, 50, 40), (-40, 50, 0), and (0, 50, -40) for Path Examples 1, 2, 3, and 4, respectively. Feel free to adjust the y-values to make the path move vertically.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/368/0*eOW9r7YxGlDjfeRH" /></figure><p>Next, in those same Transform components, adjust the Rotation so that the phoenix is facing the center of its path by setting the x- and z-components of the rotation to 0 and the y-component to 180, 90, 0, and -90 for Path Example 1, 2, 3, and 4, respectively.</p><p>Now, back in the Move Along Path [EDIT_ME], we need to set the Loop Type to Loop, and add a fifth value to the list of objects that make up the path to complete the loop. After making those changes, the prefab should look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/0*JBTbCZRrPMfIfl2R" /></figure><p>Now that the phoenix is moving correctly, we can hide the Path Objects (don’t delete them!) and delete the Movable Example Object.</p><h3>Add a bokeh particle emitter</h3><p>In this next step, we’ll add a particle emitter that follows the phoenix and produces a bokeh effect (the fading circles around the bird).</p><h4>Add particle emitter</h4><p>Open up the Particles template in Lens Studio and export the Partice_Preset_Bokeh object in the Objects panel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*MY0vI8R-J30vAonV" /></figure><p>Next, add the Partice_Preset_Bokeh.lso file (the one you just exported) in the Resources panel. You should now see the bokeh effect in the scene. The current settings create a global effect, but we want to make a local effect so we will edit the bokeh material.</p><p>Navigate to the material in the Resources panel (ParticleMaterials / World / Bokeh / Bokeh [EDIT_ME]) and select it to open it in the Inspector panel. Uncheck the three boxes nested under Spawn Location / Shape, set Gravity — cm/s to -2 (nested under Force / Gravity), and set Amount to 2 under Spawn / Time / Particle Count.</p><p>That last step will cause the effect to virtually disappear, but it will be visible again once we add the code to move the particle emitter along the phoenix’s flight path.</p><h4>Move particle emitter along a path</h4><p>Particle emitters in Lens Studio can only emit particles from a stationary point. To achieve the effect of emitting particles along a path, we will need to create many particle emitters. I found an example project with a script that does most of the work in<a href="https://support.lensstudio.snapchat.com/hc/en-us/community/posts/360012401863-Particle-question?page=1#community_comment_360000612463"> this forum post</a>. I edited the script to allow the particle emitter to track the moving phoenix that we have set up. Here is the full script:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d71d0ae7371dd0e6d82d399f582b15bf/href">https://medium.com/media/d71d0ae7371dd0e6d82d399f582b15bf/href</a></iframe><p>To make use of this code, create a new Script in the Resources panel, name it WorldParticleController, and paste the code above into it. Next, bring up the Particle_Preset_Bokeh object in the Inspector panel, add a script component, and point it to the WorldParticleController script.</p><p>Drag the Particle_Preset_Bokeh object to the Particle input box on the script component, and drag the “fly” prefab to the Tracker input box. The finished component should look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/0*H_-eDkeLDB_rG-BX" /></figure><p>You should now see bokeh particles falling gently from the phoenix.</p><h3>Add a column of smoke to drive home the 3D feel</h3><p>Finally, we’ll add some smoke to the middle of the scene to make the 3D nature of the phoenix movement and particles more apparent. Feel free to put whatever you want in the scene — I originally added a 3D plant asset that I got from Sketchfab, but it ended up being too large of a file to keep the Lens under the 4 mb limit, so I scrapped it.</p><p>To add the smoke, open up the Particles template again and export the Particle_Preset_Smoke_Lighting and Particle_Preset_Embers objects. Then, in the phoenix project, add those objects in the Resources panel to see them in the scene—and that’s it! We have a Lens featuring a phoenix flying around a column of smoke.</p><p>Here again is the finished product:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*4LMRHAL4MeOR7zTb" /></figure><h3>Conclusion</h3><p>To sum up our work…</p><p>We downloaded a textured/animated phoenix from Sketchfab, caused it to move along a path in the scene with help from a Prefab in the Refinement Pack, and added some particle emitters to round out the Lens.</p><p>Hopefully you were able to learn how to use a few different building blocks along the way — instead of a phoenix, you could use similar components to build a Lens that has a butterfly or bumblebee flying around to different flowering plants, fish swimming around a shipwreck, or anything else you can imagine.</p><p><em>Editor’s Note: </em><a href="https://heartbeat.comet.ml/"><em>Heartbeat</em></a><em> is a contributor-driven online publication and community dedicated to providing premier educational resources for data science, machine learning, and deep learning practitioners. We’re committed to supporting and inspiring developers and engineers from all walks of life.</em></p><p><em>Editorially independent, Heartbeat is sponsored and published by </em><a href="http://comet.ml/?utm_campaign=heartbeat-statement&amp;utm_source=blog&amp;utm_medium=medium"><em>Comet</em></a><em>, an MLOps platform that enables data scientists &amp; ML teams to track, compare, explain, &amp; optimize their experiments. We pay our contributors, and we don’t sell ads.</em></p><p><em>If you’d like to contribute, head on over to our</em><a href="https://heartbeat.fritz.ai/call-for-contributors-october-2018-update-fee7f5b80f3e"><em> call for contributors</em></a><em>. You can also sign up to receive our weekly newsletters (</em><a href="https://www.deeplearningweekly.com/"><em>Deep Learning Weekly</em></a><em> and the </em><a href="https://info.comet.ml/newsletter-signup/"><em>Comet Newsletter</em></a><em>), join us on</em><a href="https://join.slack.com/t/fritz-ai-community/shared_invite/enQtNTY5NDM2MTQwMTgwLWU4ZDEwNTAxYWE2YjIxZDllMTcxMWE4MGFhNDk5Y2QwNTcxYzEyNWZmZWEwMzE4NTFkOWY2NTM0OGQwYjM5Y2U"><em> </em></a><a href="https://join.slack.com/t/cometml/shared_invite/zt-49v4zxxz-qHcTeyrMEzqZc5lQb9hgvw"><em>Slack</em></a><em>, and follow Comet on </em><a href="https://twitter.com/Cometml"><em>Twitter</em></a><em> and </em><a href="https://www.linkedin.com/company/comet-ml/"><em>LinkedIn</em></a><em> for resources, events, and much more that will help you build better ML models, faster.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b63889a53e3f" width="1" height="1" alt=""><hr><p><a href="https://heartbeat.comet.ml/creating-and-animated-3d-phoenix-lens-in-lens-studio-b63889a53e3f">Creating an Animated 3D Phoenix Lens in Lens Studio</a> was originally published in <a href="https://heartbeat.comet.ml/">Heartbeat</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Custom Face Mask Snapchat Lens with Fritz AI and Lens Studio]]></title>
            <link>https://medium.com/@moellman/building-a-custom-face-mask-snapchat-lens-with-fritz-ai-and-lens-studio-ba541b8ca5ae?source=rss-3ee419b4b108------2</link>
            <guid isPermaLink="false">https://medium.com/p/ba541b8ca5ae</guid>
            <category><![CDATA[heartbeat]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[mobile-machine-learning]]></category>
            <category><![CDATA[snapchat]]></category>
            <category><![CDATA[lens-studio]]></category>
            <dc:creator><![CDATA[Matthew Moellman]]></dc:creator>
            <pubDate>Mon, 24 Aug 2020 13:31:01 GMT</pubDate>
            <atom:updated>2020-12-10T02:19:15.642Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*n7FMF0Y_kD_6WGhbBLrT2Q.jpeg" /></figure><h3>Introduction</h3><p><a href="https://heartbeat.fritz.ai/exploring-snapml-working-with-custom-neural-networks-in-lens-studio-57459a51cb3d">In my last post</a>, I covered some of the difficulties of working with custom neural networks in Lens Studio. This time, I’ll show you what it looks like, with a step-by-step guide for building and implementing your own custom neural network in a Snap Lens.</p><p>By the end of this post, you’ll know how to:</p><ol><li>Train your own custom segmentation model using <a href="https://www.fritz.ai/product/studio.html/utm_campaign=face-mask-lens&amp;utm_medium=blog_source=heartbeat">Fritz AI Studio</a>. I used face masks for my example, but you can pick whatever target object(s) you want.</li><li>Incorporate your model into a Lens Studio project via an ML Component.</li><li>Visualize real-time predictions from the model.</li><li>Make a color overlay and an interactive color slider to change the appearance of the segmented object.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*7UCO8DmZgg3z49s5x0MeRA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*T6lHM3ream-gvOcWHm5rew.gif" /></figure><h3>Part 1: Building a model with Fritz AI Studio</h3><blockquote>For a more in-depth look at working with Fritz AI Studio, you can check out our <a href="https://docs.fritz.ai/overview/studio">Quickstart Guide</a>; or, for an applied use case, read through our <a href="https://heartbeat.fritz.ai/mobile-machine-learning-with-fritz-ai-studio-a-non-developers-journey-bb644bbede17">end-to-end cat detector tutorial</a>.</blockquote><p><a href="https://www.fritz.ai/image-segmentation/">Image segmentation</a> models (like other computer vision models) require a lot of labeled data for training. We could always collect and manually annotate data, but that can be incredibly time consuming, since we’d need to do this for thousands of images.</p><p>Fortunately, we can get started with a much smaller number of images, thanks to the <a href="https://heartbeat.fritz.ai/synthetic-data-a-bridge-over-the-data-moat-29f392a52f27">synthetic data</a> generation tool in Fritz AI Studio. This tool allows users to:</p><ul><li>Easily upload and manually label a set of <a href="https://docs.fritz.ai/dataset/seed-images">seed images</a>.</li><li>Automatically apply image augmentation techniques that are specifically targeted for mobile devices.</li><li>From those seed images, programmatically generate ready-to-train datasets of thousands of images with accurately-labeled keypoints, bounding boxes, class labels, or segmentation masks — in a matter of minutes.</li></ul><p>For my face mask model, I ended up labeling around 50 seed images for the final version seen in this post—but I was able to get a passable prototype with only 10 labeled images!</p><h4>Seed Images</h4><p>To generate a dataset in Fritz AI Studio, we start out by uploading and manually labeling a collection of seed images. A seed image is a .PNG of the object of interest (in my case, a face mask) with a transparent background. I used the free tool <a href="https://www.remove.bg/">remove.bg</a> to remove the backgrounds for my images, but you can also do so manually with image editing software like Photoshop.</p><p>For a face mask detector, you could make seed images of just the face mask itself, but I ended up including the subject’s head and shoulders, as well. As a consequence, the model learns what a face mask looks like in the context of a human face/head, so a mask sitting on a table, for example, would not be detected. The seed images I used were all close to 500x500 pixels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*b0tLY5eW0KucNWxdC-HgMw.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MdKm11pJSuzzxC9VwWjYSw.png" /><figcaption>Annotating a seed image with a segmentation mask over a face mask.</figcaption></figure><p>A group of seed images is called an <strong>Image Collection</strong> in Fritz AI Studio. Once I finished annotating these seed images with segmentation masks, I used the dataset generation tool to turn my 50 manually-labeled images into 4,000 automatically-labeled images in only a few minutes.</p><p>The dataset generation tool works by taking seed images and pasting them onto random backgrounds after applying some image augmentation. The generated images look a little funky, but as far as model training is concerned, they work just fine! This generated group of labeled, synthetic images is called a <strong>Snapshot</strong> in Fritz AI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3HPBaGUEct0QIupOadwBAQ.png" /><figcaption>Four synthetic images generated from the same seed image.</figcaption></figure><p>Fritz AI Studio also lets you control some of the aspects of dataset generation by editing the “Advanced Options” configuration when creating a Snapshot.</p><p>The Foreground Scale option lets you tweak the relative size of the seed images when they’re superimposed onto the background images to suit your needs. You can also adjust the Blank Ratio, which is the fraction of images that are left blank (no seed image is pasted on the background), and the Aspect Ratio, which controls the ratio of the width to height of the generated images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*tFsyHbwm57dDUzCYctiedg.png" /><figcaption>Advanced synthetic dataset generation options in Fritz AI Studio.</figcaption></figure><h4>Training a Model</h4><p>Now that I have a large-enough labeled dataset, I’m ready to train my face mask segmentation model. I can do that in Fritz AI studio, too — I just need to configure a new training job (no code involved!) by selecting the Snapshot I generated and the model type (i.e. Image Segmentation), as well as my training budget (I went with 5 hours of training time — recommended as a default). Note that if model training finishes early, users will not be charged for the unused training time.</p><blockquote>For more specific instructions for training models with Fritz AI Studio, check out <a href="https://docs.fritz.ai/training/overview">our core concepts and step-by-step walkthrough</a>.</blockquote><p>After I finished labeling my 50 seed images and configured and kicked off my training job, I had a trained model ready to go in about 3 hours — including both the dataset generation and model training!</p><p>Before you read further, note that we recently implemented functionality for Lens Studio/SnapML projects that lets you skip the following steps, which would otherwise be needed when working with your own custom neural networks:</p><ul><li>Converting a Keras model to TensorFlow protobuf format (.pb)</li><li>Setting up a Lens Studio Project with an ML Component (Part 2)</li><li>Building the Mask Lens (Part 3)</li></ul><p>During model training, Fritz AI Studio automatically converts the model to .pb format, which makes it compatible with Lens Studio. You’ll be able to download this model file and a template Lens Studio project to help you start building your Lens quickly.</p><p>If you’re bringing your own model to the party, make sure you refer to <a href="https://lensstudio.snapchat.com/guides/machine-learning/ml-frameworks/">Snap’s documentation</a> on exporting your model to a Lens Studio-friendly format. This can be a tricky process, since not all neural network layer types are supported by Lens Studio.</p><p>The rest of this tutorial will assume that you have a trained model saved in the .pb format. Feel free to read on if you’re interested in the nitty-gritty of building Lenses with custom machine learning!</p><blockquote>Adding custom ML models to Lenses opens a whole new world of immersion and interactivity. That’s why we’re building an integration between our no-code model-building platform and SnapML. <a href="https://bit.ly/3ihzYVw">Learn more and request early access</a>.</blockquote><h3>Part 2: Setting up a Lens Studio Project with an ML Component</h3><blockquote>Note: Lens Studio is a rich toolkit with its own set of terms and naming conventions. Before you start building, I’d suggest <a href="https://lensstudio.snapchat.com/guides/getting-started/">reviewing a bit of their documentation</a> to help familiarize yourself with the platform.</blockquote><p>To get started, go ahead and open up Lens Studio and click on the New Project button. There are lots of useful template projects, but we’ll start with an empty project for now to keep things simple.</p><p>Let’s first delete the default Camera and Lighting Objects under the Objects panel (which is on the upper left side by default), and the Echopark folder under the Resources panel (which is on the lower left side by default), since we won’t be needing those. Keep in mind that you can go to Window &gt; Panels at any point if you can’t find the panel you are looking for.</p><p>To make use of our model, we need to add an ML Component to the project. To do that, click the + sign at the top of the Objects panel, search for “ML”, and select the ML Component.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/1*2sAS15tc3BMNR7VSRjJ_Yg.png" /></figure><p>Next, in the system dialogue that pops up, we need to select our trained model .pb file to load it into the ML Component. Lens Studio provides RGB images with values ranging from 0–255 as input, and expects outputs in the range of 0–255 as well (for segmentation models), so you may need to adjust the scale and bias parameters to match your model’s expected inputs and outputs.</p><p>If you downloaded the .pb file from Fritz AI, the normalization is already baked into the model, so you can leave the scale and bias parameters in Lens Studio at their default values. If you’re doing this manually (possibly with a model you trained yourself), check out my <a href="https://heartbeat.fritz.ai/exploring-snapml-working-with-custom-neural-networks-in-lens-studio-57459a51cb3d">last post</a> for some tips on integrating the normalization into your model.</p><p>If the ML Component appears under a Perspective Camera, move it to the root of the object hierarchy and delete the Perspective Camera.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/1*W7Q_w_C-QJgDOKQJpC69tg.gif" /></figure><p>Once you import the model, it will show up under the Resources panel. Before we go any further with the model, though, let’s add some preview images (or videos) so we can visualize the model’s predictions once we connect the various pieces of the lens.</p><p>To do that, click on the top middle part of the Preview panel (on the right side in the default layout) where it says “Person 1”, click on the “+ From Files” button, and select your image file. Once the file is added, it should appear in the dropdown list where you can select it to use as your preview image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*KN4GH982T9ZMnpICeiqvAQ.png" /></figure><p>Now that we have our trained model and a test image loaded into Lens Studio, let’s hook everything up so we can pass our image through the model, generate predictions, and visualize the results.</p><h3>Part 3: Building the Mask Lens</h3><h4>Part 3.1: Visualizing mask predictions</h4><p>In this section, I’ll show you how to make a basic, non-interactive version of the face mask coloring lens.</p><p><strong>1. Configure ML Component input and output textures.</strong> Select the ML Component in the Objects panel, and it should appear in the Inspector panel (located to the left of the Preview panel). Near the bottom of the Inspector panel, set the input texture to Device Camera Texture. At the bottom of the panel, click the Create Output Texture button. This texture will now be available in the Resources panel (bottom left). Feel free to rename it — I’m calling mine Face Mask Model Output Texture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*3dcion_DECe6hvQNwSKS-g.png" /></figure><p><strong>2. Add a Screen Image object</strong>. Next, we need to add a Screen Image object (the same way we added the ML Component — click the + sign in the Objects panel). That should appear nested under an Orthographic Camera object, which we’ll rename to Mask Orthographic Camera, because we’ll need another Orthographic Camera object later, for the UI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*C_U72_LoIjeanXVa4tJmQA.png" /></figure><p><strong>3. Create a material in the Material Editor</strong>. To process the model output, we need to make a material in the Material Editor. The material will let us control where a solid color overlay is displayed on the screen. We’ll use the model’s prediction at each pixel to determine the transparency of the color overlay: it will be more opaque where the model is more confident, and more transparent where the model is less confident. To get started, add a Graph Empty material in the Resources panel:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/566/1*RJISm51tjiTJRI6e17qOHA.png" /></figure><p>Double click on your new Graph Empty material to open it in the Material Editor. Feel free to check out the <a href="https://lensstudio.snapchat.com/guides/material-editor/introduction-and-concepts/">Lens Studio Material Editor documentation</a> for an in-depth explanation of the underlying concepts, but the basic idea is that we’re making a graph (you can think of it as a function) whose inputs are:</p><ol><li>The segmentation model output</li><li>The color of the color overlay</li><li>The maximum alpha (opacity) value</li></ol><p>The output of the graph is a shader node, which in our case controls the color and transparency of every pixel in our color overlay.</p><p>To add nodes to the graph, click the + button and search for the node type:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*OkAdCqvxnnlCf9Hdd0Vqsg.png" /></figure><p>Here is the complete structure of the graph:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eOZhIcdf78SvOXzD01ByIg.png" /></figure><p>There’s quite a bit going on here—probably enough for a separate blog post— but for now I’ll go over the essential parts and give you a link to <a href="https://bit.ly/2FFfhVq">download the material</a> (<em>link initiates a download</em>) and drop it into your project so you don’t have to make it from scratch.</p><p>The Float Parameter node controls the maximum opacity of the color overlay. Setting this value to 0.6–0.7 worked well for my use case because it allows for some of the details of the face masks to come through.</p><p>The Texture 2D Parameter node takes in the model output. My model has two output channels: one for a background class, and one for face masks. Having two channels is redundant for segmenting a single class of objects, but Fritz AI Studio produces models with a background class because it offers the flexibility of training models that can segment multiple classes of objects separately.</p><p>So, for our material, we just need to grab the second channel of the model output using a Swizzle node. Finally, the Color Parameter node takes in a color as input. For now, we’ll have to select a color manually, but eventually we’ll hook this up to the color slider UI element.</p><p>Both the Texture 2D Parameter and Color Parameter nodes are exposed to custom scripts — more on that in Step 3 of the next section.</p><p><strong>4. Connect the model output texture</strong>. Now that we have the material graph set up, we need to connect the model output texture. Select the material in the Resources panel, and you should see Graph Parameters in the Inspector panel. Under Graph Parameters, set the Model Output parameter as the model output texture created in Step 1 above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*7-1QMWkTev7aeD_1fD2N2w.png" /></figure><p><strong>5. Configure the Screen Image component.</strong> Set the Material as the segmentation material from Step 3, set the Texture to the model output texture from Step 1, and set the Stretch Mode to Stretch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*5gKgbsB1DJfsvAJlBGDjfQ.png" /></figure><p>At this point, you should be able to see your model’s predictions! Try selecting the mask material in the Resources panel and playing with the Mask Color and Segmentation Mask Alpha parameters (which control the transparency of the color overlay) in the Inspector panel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*ILdNjJ5ciZtY8uji4yyHSw.gif" /></figure><h4>Part 3.2: Adding a Color Slider</h4><p>Let’s spice things up a bit and add an interactive UI element that will let users pick the color of the face mask overlay.</p><p><strong>1. Create another Orthographic Camera.</strong> The easiest way to do this is to duplicate the Mask Orthographic Camera (right click &gt; Duplicate) and then delete the Full Frame Region 0 under the new camera. Let’s rename it to Slider Orthographic Camera while we’re at it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*am1QObsCH4GYTGjh3ZGcXA.gif" /></figure><p><strong>2. Implement the Slider.</strong> Follow the instructions <a href="https://lensstudio.snapchat.com/guides/scripting/helper-scripts/user-interface/">here</a> (make sure to use the Slider Orthographic Camera from Step 1) to get UI elements into your project. Once you have the UI pieces in your project, you can delete the Tap Hint and everything nested under UI Panel, except for the UI Color Picker. Your object hierarchy should look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*XHPK4lfMfgUB5ZMrAG3aAw.png" /></figure><p><strong>3. Modify the UI Color Picker component.</strong> Open the UIColorPicker script found in the Resources panel under UI &gt; Resources &gt; Scripts, and change line 48 from</p><p>//@input <strong>Component.MeshVisual</strong> colorRecipient</p><p>to</p><p>//@input <strong>Asset.Material</strong> colorRecipient</p><p>And change line 652 from</p><p>script.colorRecipient.mainPass.<strong>baseColor</strong> = currentColor;</p><p>to</p><p>script.colorRecipient.mainPass.<strong>maskColor</strong> = currentColor;</p><p>Note that <strong>maskColor </strong>is the name we gave to the Color Parameter input node in the Face Mask Segmentation Material graph from Step 3 of the previous section.</p><p>These code changes enable us to connect the color slider to our face mask color overlay. Select the UI Color Picker in the Objects panel and assign the segmentation material from Step 3 in the previous section:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*SM3M4qwrIZeEvggJRA8ttg.png" /></figure><p><strong>4. Organize the various components into two layers.</strong> Let’s put the color overlay components in one layer and the color slider components in another layer, so we can control when those layers appear in our Lens. This allows us to display the color overlay at all times and only show the color slider in capture mode. Everything under Mask Orthographic Camera should go in a layer called “Mask”, and everything under the Slider Orthographic Camera should go in a layer called “Slider”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*9fFhcg8CRLGmLMLVBc_wYA.gif" /></figure><p>Make sure you expand the UI Panel component and assign everything under that to the “Slider” layer, as well:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*BqZpeRQ0a3iRcXBMkdBu-Q.png" /></figure><p><strong>5. Organize render targets in the Scene Config.</strong> This step enables us to display the color slider UI element in live mode when recording a Snap, but have it disappear in capture mode before sending the snap. We’ll set this up in the Scene Config panel. The Scene Config panel lives behind the Resources panel in the lower left. Click on Scene Config so it’s visible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/1*tLXxC8ff_K9883vGElCCDw.png" /></figure><p>In the Scene Config, click on the Render Target button to the right of Capture Target. In the window that pops up, rename Render Target to Capture Render Target, and make a new Render Target called Live Render Target.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*BVjL-XbwUKOgqev8QVlejg.gif" /></figure><p>Back in the Scene Config, set the Capture Target to Capture Render Target, and set the Live Target to Live Render Target.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*ISBZNSlPIGsR6gjmPEbq6g.png" /></figure><p>Make sure that the render target for the Mask Orthographic Camera is set to Capture Render Target, and that the render target for the Slider Orthographic Camera is set to Live Render Target.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*eJQElH6CUiPG8aLsu_ZTTg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*qaci58GTbhxEcn0DPgDGWA.png" /></figure><p>Finally, set the input texture of the Capture Render Target to be the Device Camera Texture, and set the input texture of the Live Render Target to be the Capture Render Target.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/734/1*hljM8xv94-CZpMLUQ9O6Qw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/734/1*mKsF70-hWk1Bj7TUIbPtyA.png" /></figure><p>Let’s test out our new lens on a phone! You can easily <a href="https://lensstudio.snapchat.com/guides/general/pairing-to-snapchat/">pair your device with Lens Studio</a> and preview lenses on Snapchat. Here’s what the face mask colorizer looks like in action:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/270/1*G-fqQOSSQdMb-A7uaUGz0Q.gif" /></figure><h3>Conclusion</h3><p>And that’s it! We just built our very own face mask color changer Lens with a custom neural network, implemented in a Lens project via SnapML. We worked with a few different processes and tools, so let’s quickly recap:</p><ul><li>In Fritz AI Studio, starting out with only 50 manually-labeled seed images, we generated a data Snapshot of 4000 automatically-labeled images that we used to train our custom face mask segmentation model. We exported this Lens Studio-compatible model and an accompanying project template.</li><li>And for those who wanted to explore the nitty gritty of working with Lens Studio, we also worked through a step-by-step of the Lens building process, with special attention paid to working with custom neural networks.</li></ul><p>Stay tuned for more as our team continues to experiment with Lens Studio and Snap ML. And if you’re interested in building custom ML models to use directly in Lenses, <a href="https://bit.ly/3ihzYVw">let us know</a>!</p><p><em>Editor’s Note:</em><a href="http://heartbeat.fritz.ai/"><em> </em><strong><em>Heartbeat</em></strong></a><strong><em> </em></strong><em>is a contributor-driven online publication and community dedicated to exploring the emerging intersection of mobile app development and machine learning. We’re committed to supporting and inspiring developers and engineers from all walks of life.</em></p><p><em>Editorially independent, Heartbeat is sponsored and published by</em><a href="http://fritz.ai/"><em> </em><strong><em>Fritz AI</em></strong></a><em>, the machine learning platform that helps developers teach devices to see, hear, sense, and think. We pay our contributors, and we don’t sell ads.</em></p><p><em>If you’d like to contribute, head on over to our</em><a href="https://heartbeat.fritz.ai/call-for-contributors-october-2018-update-fee7f5b80f3e"><em> </em><strong><em>call for contributors</em></strong></a><em>. You can also sign up to receive our weekly newsletters (</em><a href="https://www.deeplearningweekly.com/"><strong><em>Deep Learning Weekly</em></strong></a><em> and the </em><a href="https://www.fritz.ai/newsletter/?utm_campaign=fritzai-newsletter&amp;utm_source=heartbeat-statement"><strong><em>Fritz AI Newsletter</em></strong></a><em>), join us on</em><a href="https://join.slack.com/t/fritz-ai-community/shared_invite/enQtNTY5NDM2MTQwMTgwLWU4ZDEwNTAxYWE2YjIxZDllMTcxMWE4MGFhNDk5Y2QwNTcxYzEyNWZmZWEwMzE4NTFkOWY2NTM0OGQwYjM5Y2U"><em> </em></a><a href="http://fritz.ai/slack"><strong><em>Slack</em></strong></a><em>, and follow Fritz AI on</em><a href="https://twitter.com/fritzlabs"><em> </em><strong><em>Twitter</em></strong></a><em> for all the latest in mobile machine learning.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba541b8ca5ae" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring SnapML: Working with Custom Neural Networks in Lens Studio]]></title>
            <link>https://heartbeat.comet.ml/exploring-snapml-working-with-custom-neural-networks-in-lens-studio-57459a51cb3d?source=rss-3ee419b4b108------2</link>
            <guid isPermaLink="false">https://medium.com/p/57459a51cb3d</guid>
            <category><![CDATA[heartbeat]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[snapchat]]></category>
            <category><![CDATA[mobile-machine-learning]]></category>
            <category><![CDATA[lens-studio]]></category>
            <dc:creator><![CDATA[Matthew Moellman]]></dc:creator>
            <pubDate>Mon, 03 Aug 2020 13:11:45 GMT</pubDate>
            <atom:updated>2021-10-18T17:10:29.116Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*K_AX_RDh4T8-gg2mdrGimA.jpeg" /></figure><p>The promise of being able to drop your own custom neural networks into Lens Studio as Lenses, which can then be deployed to millions of devices, is potentially game-changing.</p><p>But working with a tool this powerful and versatile inevitably involves some nuances you’ll need to consider while building. While working through a demo project (stay tuned for a tutorial soon), I was able to identify some key areas where working with custom ML models required some tweaking, fine-tuning, and adaptation.</p><p><a href="https://heartbeat.comet.ml/exploring-snapml-a-technical-overview-45d37114fe81">Last time</a>, we covered SnapML at a high-level — how it works, the benefits we think it can provide ML teams, and some strengths/areas for improvement. This time, we’ll dive into a few particular considerations you’ll need to keep in mind when doing custom work with SnapML and Lens Studio. Specifically, the following are considerations I came across while working to customize a<a href="https://lensstudio.snapchat.com/templates/ml/ml-templates-library/"> SnapML template</a> for my own use case, which seems to be a good place to start.</p><h3>The Project</h3><p>To help make this more tangible, I’ll be discussing my process while trying to implement a custom segmentation model in Lens Studio. Luckily, Snap includes some template projects to get started — and for my use case, they have a pizza segmentation template that I figured I’d be able to use as an easy starting point.</p><p>As I’d mentioned in the overview, <a href="https://lensstudio.snapchat.com/guides/machine-learning/ml-overview/">the SnapML docs</a> make it seem pretty easy to use a custom model built from a template in a Lens. They have a drag-and-drop interface and tools to adjust the inputs (e.g. stretching/scaling the camera feed, applying basic pre-processing) and outputs. But in reality, I ended up struggling once it came time to integrate the model into my Lens Studio project. These difficulties centered on two areas:</p><ul><li>Model Input and Output Formatting</li><li>Selecting Channels</li></ul><h3>Model Input and Output Formatting</h3><p>I wanted to use a custom segmentation model that I’d previously trained. I was hoping to use their<a href="https://lensstudio.snapchat.com/templates/ml/custom-segmentation"> pizza segmentation demo Lens</a> for my project as a starting point. The only difference between Snap’s model and mine was that mine has two output channels. But with my lack of Lens Studio / computer graphics knowledge, it ended up being very difficult to use the output from the one channel I wanted to.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Hu-4CxDN0VGB52L2" /><figcaption><em>A look at the Lens Studio interface, with the pizza segmentation template loaded in.</em></figcaption></figure><p>My model expects as input a 3-channel image with the RGB values transformed to floats between -0.5 and 0.5, instead of ints from 0–255. It outputs a 2-channel segmentation map, whose values are floats between 0 and 1. The first channel represents a background class (which enables me to train models with multiple classes of objects), and the second channel contains predictions for the class of objects I’m segmenting.</p><p>Lens Studio expects output values to be between 0 and 255, so I needed to both scale my outputs and select only the second channel. We can handle the input/output value transformations either in Lens Studio or in the model architecture, but the output channel must be selected in Lens Studio, because the Slice layer is not currently supported.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*uT97TRav24EHQdaX" /><figcaption>Lens Studio’s interface for adjusting model inputs and outputs</figcaption></figure><h4><strong>Option 1: Transform inputs and outputs in Lens Studio</strong></h4><p>In Lens Studio, it’s possible to adjust scale and bias parameters when you import the model. In my case, I wanted values from 0–255 to be transformed into values from -0.5 to 0.5, so I set the input scale parameters to be 0.0039… (= 1 / 255) and the bias parameters to be -0.5 for all 3 channels.</p><p>For the outputs, I set the scale parameters to be 255 to stretch my output values so they ended up between 0 and 255 instead of 0 and 1. This way of importing models was convenient for testing whether or not my model was importing correctly and making reasonable predictions.</p><p>After a few times though, it felt tedious to set all these parameters manually (and something about approximating 1 / 255 as 0.0039 bugs me, but I digress..), so I decided to work around this by using Option 2.</p><h4><strong>Option 2: Bake input/output transformations into the model architecture.</strong></h4><p>Once I was confident that my model architecture would import successfully, I wrapped the model with Lambda layers to transform my inputs and outputs:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6de7a2fc82a6829e04a959d9df4bc64f/href">https://medium.com/media/6de7a2fc82a6829e04a959d9df4bc64f/href</a></iframe><h3>Selecting channels</h3><p>Segmentation models output data arrays, which contain probabilities of each pixel belonging to a certain class (e.g. pizza). Effects in Snap Lenses, however, are composed of various filters that operate on images in standard RGB color spaces.</p><p>One of the most confusing parts I had working with Lens Studio was mapping model outputs to the image effects inside Lens Studio. To make model outputs useful in your Lenses, you’ll need to adopt the language of 3D graphics and learn a bit about textures, materials, and shaders. It’s worth familiarizing yourself with these concepts before getting started:</p><p><a href="https://lensstudio.snapchat.com/guides/">Lens Studio</a></p><p>Essentially, output is treated as a <strong>texture</strong> by Lens Studio. Textures have four channels (RGBA), and a <strong>material</strong> can be applied to manipulate their values.</p><p>In my specific case, I had to use this Split Vector node in the Material Editor to select the second channel of my model output. Unfortunately, it appears to be set up to handle inputs with 4 channels. As such, it seemed to simply repeat my two channels to fill up the 4 channels it expected. This ended up working for me this time around, but I don’t know if it could handle a model with 10 output channels, just as an example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/0*jS3JFk1yRuyGIhDv" /></figure><p>It remains to be seen how this dynamic might affect different project and model types, but it’ll be important to understand these concepts and how they interact with your custom models.</p><h3>Conclusion</h3><p>ML engineers looking to work with SnapML should be encouraged — though there are parts of the framework that are a bit difficult to work with, it’s a powerful set of tools once you’ve cracked the code a bit, so to speak.</p><p>But these same ML engineers would also be well-served by digging a bit deeper into Lens Studio, as it has some unique concepts and processes that might not be as familiar for some. Additionally, a thorough understanding of your own model’s input/output parameters, along with a strong sense of what you want to ultimately do with that model, will help ease the challenges I’ve identified here.</p><p>Stay tuned for more from our team, <a href="https://heartbeat.comet.ml/working-with-snapml-templates-in-lens-studio-an-overview-97a0b08ee719">including closer looks at Snap’s template projects</a>, and more specifics on working with ML inside SnapML and Lens Studio.</p><p><em>Editor’s Note: </em><a href="https://heartbeat.comet.ml/"><em>Heartbeat</em></a><em> is a contributor-driven online publication and community dedicated to providing premier educational resources for data science, machine learning, and deep learning practitioners. We’re committed to supporting and inspiring developers and engineers from all walks of life.</em></p><p><em>Editorially independent, Heartbeat is sponsored and published by </em><a href="http://comet.ml/?utm_campaign=heartbeat-statement&amp;utm_source=blog&amp;utm_medium=medium"><em>Comet</em></a><em>, an MLOps platform that enables data scientists &amp; ML teams to track, compare, explain, &amp; optimize their experiments. We pay our contributors, and we don’t sell ads.</em></p><p><em>If you’d like to contribute, head on over to our</em><a href="https://heartbeat.fritz.ai/call-for-contributors-october-2018-update-fee7f5b80f3e"><em> call for contributors</em></a><em>. You can also sign up to receive our weekly newsletters (</em><a href="https://www.deeplearningweekly.com/"><em>Deep Learning Weekly</em></a><em> and the </em><a href="https://info.comet.ml/newsletter-signup/"><em>Comet Newsletter</em></a><em>), join us on</em><a href="https://join.slack.com/t/fritz-ai-community/shared_invite/enQtNTY5NDM2MTQwMTgwLWU4ZDEwNTAxYWE2YjIxZDllMTcxMWE4MGFhNDk5Y2QwNTcxYzEyNWZmZWEwMzE4NTFkOWY2NTM0OGQwYjM5Y2U"><em> </em></a><a href="https://join.slack.com/t/cometml/shared_invite/zt-49v4zxxz-qHcTeyrMEzqZc5lQb9hgvw"><em>Slack</em></a><em>, and follow Comet on </em><a href="https://twitter.com/Cometml"><em>Twitter</em></a><em> and </em><a href="https://www.linkedin.com/company/comet-ml/"><em>LinkedIn</em></a><em> for resources, events, and much more that will help you build better ML models, faster.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=57459a51cb3d" width="1" height="1" alt=""><hr><p><a href="https://heartbeat.comet.ml/exploring-snapml-working-with-custom-neural-networks-in-lens-studio-57459a51cb3d">Exploring SnapML: Working with Custom Neural Networks in Lens Studio</a> was originally published in <a href="https://heartbeat.comet.ml/">Heartbeat</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring SnapML: A Technical Overview]]></title>
            <link>https://heartbeat.comet.ml/exploring-snapml-a-technical-overview-45d37114fe81?source=rss-3ee419b4b108------2</link>
            <guid isPermaLink="false">https://medium.com/p/45d37114fe81</guid>
            <category><![CDATA[lens-studio]]></category>
            <category><![CDATA[heartbeat]]></category>
            <category><![CDATA[mobile-ml]]></category>
            <category><![CDATA[snapchat]]></category>
            <category><![CDATA[augmented-reality]]></category>
            <dc:creator><![CDATA[Matthew Moellman]]></dc:creator>
            <pubDate>Wed, 22 Jul 2020 15:11:24 GMT</pubDate>
            <atom:updated>2021-10-11T16:25:32.443Z</atom:updated>
            <content:encoded><![CDATA[<h4>Our high-level technical overview of Lens Studio’s new framework for custom machine learning models</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*y3pDQqjE_degwPqybup7Vg.jpeg" /></figure><p>For years now, Snapchat has been at the forefront of mobile machine learning — their popular Lenses, which often <a href="https://heartbeat.comet.ml/combining-artificial-intelligence-and-augmented-reality-in-mobile-apps-e0e0ad2cfddc">combine on-device ML models with augmented reality</a>, have become shining examples of the power and flexibility of on-device machine learning.</p><p>Given our respect and admiration for Snap’s work in this area, our team was thrilled to hear about the recent release of <a href="https://lensstudio.snapchat.com/guides/machine-learning/ml-overview/">SnapML, Snap’s new ML framework</a> inside their development platform Lens Studio (released with 3.0).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/0*SitF79Lmh7BwllzC" /><figcaption>Sample Lenses built with custom neural networks, SnapML, and Lens Studio (<a href="https://lensstudio.snapchat.com/">image source</a>)</figcaption></figure><h3>How SnapML Works</h3><p>Engineers, dev teams, and other creators can drop custom ML models directly into Snapchat — these models must be, generally speaking, compatible with the ONNX model format. Some standard data pre- and post-processing methods can be applied in Lens Studio. Alternatively, these methods can be baked into the model architecture.</p><p>Custom models are integrated using Snap’s ML Component, which defines model inputs and outputs and how the model should run (i.e. every frame, upon a particular user action, in real-time, etc.).</p><p>These models must be built outside of Lens Studio and imported. To help with that, <a href="https://lensstudio.snapchat.com/guides/machine-learning/lens-templates/">there are templates available (via Jupyter Notebooks) for a number of use cases</a>: classification, object detection, style transfer, custom segmentation, and ground segmentation. You can use these as a starting point for your own custom models — especially if your use case matches one of these ML tasks.</p><p>Perhaps most importantly, because these models can be used directly in Snapchat as Lenses, they can quickly become available to millions of users around the world.</p><h3>What SnapML could mean for ML engineers and teams</h3><p>The above functionality, combined with Snap Lenses as a promising distribution channel, presents ML teams with a number of intriguing value props:</p><ul><li><strong>Reduced barriers to entry</strong>: The ability to drop custom neural networks into Lens Studio, and then distribute them to Snap’s millions of users, is an exciting possibility. This means that teams have both a promising channel and a deployment target that doesn’t require developing an entire mobile application from scratch.</li><li><strong>Easier experimentation</strong>: Since teams won’t have to work through a full app release cycle to see their on-device models in action, it’ll be easier to experiment with brand- and product-based Lenses powered by immersive ML features.</li><li><strong>Democratized on-device ML</strong>: Part of our vision at Fritz AI is to build a product that helps democratize mobile machine learning, both through the underlying technology and the tools that make it possible. In <a href="https://heartbeat.comet.ml/lens-studio-3-0-introduces-snapml-for-adding-custom-neural-networks-directly-to-snapchat-c2c32ed95b2b">our recent interview with one of SnapML’s early partners and creators</a>, Hart Woolery of 2020CV, he noted that SnapML’s potential reminds him, in some ways, of how YouTube helped democratize video creation.</li></ul><p><a href="https://heartbeat.comet.ml/lens-studio-3-0-introduces-snapml-for-adding-custom-neural-networks-directly-to-snapchat-c2c32ed95b2b">Lens Studio 3.0 introduces SnapML for adding custom neural networks directly to Snapchat</a></p><h3>Our Technical Overview</h3><p>To get a sense of what it’s like to build custom experiences with SnapML in Lens Studio, I worked through a simple demo project. While I’ll cover some of the unique challenges of that project in my next blog post, I wanted to take a bit of time to provide a high-level technical overview of the new platform — what it does well, where things get tricky, and some nuances that are essential to consider when getting started.</p><h4>Things SnapML Does Well</h4><p>First, a look at things SnapML does really well:</p><ul><li>One of my favorite features involves <strong>controlling <em>how</em> the model is run</strong> (e.g. every frame, when a user takes an action, in another thread). This allows degrees of control that really allow you to fine-tune the experience to fit your project requirements, while also experimenting with interactive and immersive elements.</li><li>You can also loop a sample video inside Lens Studio to <strong>see what your Lens does as you’re creating and editing it.</strong> I used my own images in the preview pane, as well as live camera feed. I didn’t try using my own video clip, but I imagine that works, too. This kind of real-time visualization is a nice touch and one that makes experimentation even easier.</li><li>There’s also <strong>an impressive OTA model download UX</strong>. Scan a QR code to pair your device, click a button in Lens Studio to send the model to your phone, get a notification when it downloads, and it shows up as a Lens. I love how simple this ended up being, as model delivery can be a pain, especially when you have to work through a full app update to release a new ML feature.</li><li>As mentioned above, the under-the-hood work that Snap ML does is also a positive. There are still some lingering issues here (more on this below), but things like model conversion and pre- and post-processing help speed up project workflows significantly.</li></ul><h4>Areas for Improvement in SnapML</h4><ul><li><strong>Data can be difficult. </strong>In the demo projects and notebooks provided by Snap, they’ve used COCO as their data source. Using COCO as the data source means that the category of objects you want to use for model training must be part of the COCO dataset, it requires writing code to extract the images/annotations of interest, and you have no control over the quality of the annotations. As such, working with unique or custom data is currently a bit of a pain point.</li><li><strong>Model training. </strong>Snap uses PyTorch to train a custom segmentation model in their example Jupyter Notebook. They recommend starting with a <a href="https://heartbeat.comet.ml/exploring-the-mobilenet-models-in-tensorflow-d9d21774cdab">MobileNet architecture</a>, but you can’t quite use the pre-packaged MobileNet model from PyTorch out-of-the-box. There’s a bit of extra code to write, in which you have to swap out layers to ensure compatibility.</li><li><strong>Model input and output formatting. </strong>Once you have a trained model, <a href="https://lensstudio.snapchat.com/guides/machine-learning/ml-overview/">the SnapML docs</a> make it seem pretty easy to use that model in a lens. They have a drag-and-drop interface and tools to adjust the inputs (e.g. stretching/scaling the camera feed, applying basic pre-processing) and outputs. But in practice, this was more difficult than I’d imagined. We’ll cover why in another blog post soon.</li></ul><h3>What’s Next</h3><p>All-told, SnapML is an incredible leap forward for mobile machine learning, and edge AI more broadly. V1 is impressive and promising, and the areas for improvement we explored above seem more like the growing pains of any ambitious piece of software’s initial release, and less like a major roadblock.</p><p>In addition to this overview, we’ve taken a closer look and done some hands-on work with the new framework as part of our continuing series. You can find these resources below:</p><ul><li><a href="https://heartbeat.comet.ml/lens-studio-3-0-introduces-snapml-for-adding-custom-neural-networks-directly-to-snapchat-c2c32ed95b2b">Lens Studio 3.0 introduces SnapML for adding custom neural networks directly to Snapchat</a></li><li><a href="https://heartbeat.comet.ml/working-with-snapml-templates-in-lens-studio-an-overview-97a0b08ee719">Working with SnapML Templates in Lens Studio: An Overview</a></li></ul><p><em>Editor’s Note: </em><a href="https://heartbeat.comet.ml/"><em>Heartbeat</em></a><em> is a contributor-driven online publication and community dedicated to providing premier educational resources for data science, machine learning, and deep learning practitioners. We’re committed to supporting and inspiring developers and engineers from all walks of life.</em></p><p><em>Editorially independent, Heartbeat is sponsored and published by </em><a href="http://comet.ml/?utm_campaign=heartbeat-statement&amp;utm_source=blog&amp;utm_medium=medium"><em>Comet</em></a><em>, an MLOps platform that enables data scientists &amp; ML teams to track, compare, explain, &amp; optimize their experiments. We pay our contributors, and we don’t sell ads.</em></p><p><em>If you’d like to contribute, head on over to our</em><a href="https://heartbeat.fritz.ai/call-for-contributors-october-2018-update-fee7f5b80f3e"><em> call for contributors</em></a><em>. You can also sign up to receive our weekly newsletters (</em><a href="https://www.deeplearningweekly.com/"><em>Deep Learning Weekly</em></a><em> and the </em><a href="https://info.comet.ml/newsletter-signup/"><em>Comet Newsletter</em></a><em>), join us on</em><a href="https://join.slack.com/t/fritz-ai-community/shared_invite/enQtNTY5NDM2MTQwMTgwLWU4ZDEwNTAxYWE2YjIxZDllMTcxMWE4MGFhNDk5Y2QwNTcxYzEyNWZmZWEwMzE4NTFkOWY2NTM0OGQwYjM5Y2U"><em> </em></a><a href="https://join.slack.com/t/cometml/shared_invite/zt-49v4zxxz-qHcTeyrMEzqZc5lQb9hgvw"><em>Slack</em></a><em>, and follow Comet on </em><a href="https://twitter.com/Cometml"><em>Twitter</em></a><em> and </em><a href="https://www.linkedin.com/company/comet-ml/"><em>LinkedIn</em></a><em> for resources, events, and much more that will help you build better ML models, faster.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=45d37114fe81" width="1" height="1" alt=""><hr><p><a href="https://heartbeat.comet.ml/exploring-snapml-a-technical-overview-45d37114fe81">Exploring SnapML: A Technical Overview</a> was originally published in <a href="https://heartbeat.comet.ml/">Heartbeat</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>