<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Prisma Labs Blog - Medium]]></title>
        <description><![CDATA[Prisma Labs deep learning blog - Medium]]></description>
        <link>https://medium.com/prismalabs?source=rss----18edcfbe88a5---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Prisma Labs Blog - Medium</title>
            <link>https://medium.com/prismalabs?source=rss----18edcfbe88a5---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 02:03:34 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/prismalabs" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Real-time Portrait Segmentation on Smartphones]]></title>
            <link>https://medium.com/prismalabs/real-time-portrait-segmentation-on-smartphones-39c84f1b9e66?source=rss----18edcfbe88a5---4</link>
            <guid isPermaLink="false">https://medium.com/p/39c84f1b9e66</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[mobile-app-development]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <dc:creator><![CDATA[Anton Lebedev]]></dc:creator>
            <pubDate>Wed, 17 Oct 2018 15:29:55 GMT</pubDate>
            <atom:updated>2018-10-19T19:48:36.042Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8WsYJPKlUz6wO2lJOOY74g.png" /><figcaption>Demonstration of segmentation value: original image and simulated bokeh effect</figcaption></figure><p>Various image effects have been receiving increasing attention in recent years. A popular example is bokeh, a blur on an out-of-focus region in a photograph. This effect is achieved by using a fast camera lens with a wide aperture. Unfortunately, it is almost impossible to reproduce this kind of effect with mobile phone cameras because they do not meet the necessary technical specifications. However, if each image pixel is classified into person and background categories, the bokeh effect can be simulated by blurring just the background. Classification of each pixel into categories is called semantic segmentation, and it can be used in various ways, such as changing the image background or applying separate filters for foreground and background.</p><p>Some devices may use stereo cameras that are able to extract depth information in order to segment an image. The approach in this article, however, aims to build a segmentation system that would be able to infer the desired information from a single RGB image. This would allow us to apply portrait effects on a frontal camera for a broader set of devices.</p><p>Significant progress was made over the years in the field of computer vision and, particularly, in semantic segmentation. It was achieved with the help of Convolutional Neural Networks. Such neural networks are effective at abstracting information from an image and inferring its qualities, including segmentation maps. However, most architectures require a lot of computational power and good parallelization on the device side. This requirement and the widespread popularity of neural networks encouraged device manufacturers to create new frameworks. The frameworks enable neural networks to be evaluated on a mobile GPU. These solutions, in most cases, achieve greater performance than ones based on CPUs.</p><p>Despite the popularity of deep learning in computer vision, all of the segmentation models that we have found have some disadvantages. In most cases, they are too slow to be inferred in real-time on mobile devices. Another complication is compatibility with mobile frameworks — not all layers are currently supported. For this reason, we designed a special architecture which fits all the restrictions.</p><p>Finally, the last challenge is the training dataset. To achieve the best performance, we collected a dataset for this task and created a specific augmentation pipeline.</p><h3>Dataset</h3><p>Neural networks need training, so the major part of building this type of Neural networks need training, so a major part of building this type of algorithm is a dataset. We wanted segmentation maps to be as accurate as possible, so all of the images in our dataset were manually labelled with a professional editing program. Furthermore, we wanted the dataset to be diversified. A lack of diversity may cause the algorithm to make structural mistakes. For example, if a model can’t handle a specific pose, it may be that the dataset is not representative enough. We used two ways to improve our dataset quality: augmentation and collection of extra images. The amount of images in our dataset incrementally increases over time. At the moment, we have about 13 thousand images in a training set and an additional thousand in a test set, with people in a variety of poses, containing full-body, portrait and selfie photos.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GhO19E9ZSVqtc66LBdxmIw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mhuF88JeavuHmEOyBsss2g.png" /><figcaption>Image augmentation: original image and two examples of random augmentation</figcaption></figure><h3>Architecture</h3><p>Neural networks need an architectural design. There are plenty of different layers and methods that can be used to combine them, each with its own advantages. Before specifying our network, we need to understand what features of the architecture are the most desirable. We want it to be fast, lightweight and compatible with most mobile frameworks. To achieve this, we sacrifice convergence rate and skip the regularization layers, preventing overfitting by using a small model size and massive augmentation.</p><p>Furthermore, it is crucial to use layers which are compatible with target platforms. Each framework has its own set of implemented operations, and we have to use the intersection over all these sets. We use the following layers: convolution, transposed convolution, relu, sigmoid and element-wise addition.</p><p>The last architectural challenge is efficiency. One of the key parameters is input image shape. We can use our model on an image scaled to lower resolution and upscale the output to the original shape. The lower the input shape is, the faster is the inference. However, making the resolution too low causes degradation of the resulting semantic mask quality due to upscale artefacts. The highest resolution that we were able to process in real-time was 256x256.</p><p>The final topology of the network is based on a U-Net encoder-decoder structure. In our experience, it is the best starting point for image segmentation with a low number of classes. Nevertheless, our architecture incorporates a few significant changes.</p><p>To improve convergence and make the model more accurate, we replaced the concatenation of features after upsampling with element-wise addition.</p><p>The next key feature is depth-wise separable convolution. Comparing with ordinary convolutions, it allows the same quality to be achieved with less computation cost and memory requirements. We add a residual block with depth-wise separable convolutions as the base block in our architecture instead of the original Convolution + ReLU blocks. Another feature of our architecture is asymmetry. A decoder contains more blocks than an encoder. This allows us to achieve better accuracy with the same computational cost. Finally, to achieve real-time inference, we sought to reduce the number of layers and feature maps as much as possible.</p><p>The final architecture is depicted in the figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y0S1deISIdDnbDhpqD4h4g.png" /><figcaption>Segmentation network topology</figcaption></figure><h3>Training tricks</h3><p>The next important stage is training. We have a two-step training process that leads to a better quality result. At first, we pre-train our model on all pictures from the dataset. The next step is a training model designed for selfie- and portrait- subsets that uses the result of the previous step as initial values for weights. In order to broaden our dataset during both steps, we massively use the augmentation. Examples of this augmentation are image rotation, curve adjustments and shade simulation. Our augmentation pipeline is built on two main concepts. The first is not to spoil the images too much — after the augmentation, the image should seem real. The second is to try to mask the flaws of the dataset. For example, in our experiments, motion blur imitation made a significant impact on the stability of video processing.</p><p>Another point we want to mention are output artefacts. The neural network tends to generate checkerboards — a grid that can be seen after applying background subtraction. This type of artefact does not influence the mIoU metric, but creates a poor visible experience. To solve this issue we add additional residual blocks on the final shape and give extra training time.</p><h3>Results</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TM1FcHfV1QzlmMM2SKI9SQ.png" /><figcaption>Segmentation output: original image, background subtraction, foreground subtraction</figcaption></figure><p>Finally, we get a model for portrait segmentation that is well-balanced between quality and speed. It takes up 3.7 mb in fp32 onnx format. Our algorithm is transferable to a majority of frameworks, including CoreML, MetalPerformanceShaders, SNPE, Huawei Kirin, OpenVINO. Specifically, on an iPhone 7 and a Mi Mix 2S, this model succeeds in having a runtime faster than the 30 fps threshold. The pipeline includes getting a pixel buffer from a camera, preprocessing, inference, postprocessing and rendering of the final image with a removed background on the screen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P1nAVIApsenH8OFVE74UTQ.png" /><figcaption>One more example of segmentation</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-bktbAMn_hytyPhg_EBkaw.png" /><figcaption>Bokeh simulation: image with and without background blur</figcaption></figure><h3>Post Scriptum</h3><p>The presented portrait segmentation system was created in collaboration with my remarkable colleagues. Unfortunately, there is no official co-authoring option on Medium, but they are mentioned below:</p><p><a href="https://medium.com/u/8719e21251a6">Lebedev Anton</a>, <a href="https://medium.com/u/c58dddf77305">Konstantin Semyanov</a>, Artur Chakhvadze — training pipeline and network architecture<br><a href="https://medium.com/u/76f1b5cbc425">Roman Kucev</a>, <a href="https://medium.com/u/434ba368713e">Pavel Voropaev</a> — data collection and annotation process management<br><a href="https://medium.com/u/d94709aa368c">Maxim Skorokhodov</a>, <a href="https://medium.com/u/88a5235e2797">Vyacheslav Tarasov</a> — Android integration<br><a href="https://medium.com/u/c1025351a31e">Oleg Poyaganov</a>, <a href="https://medium.com/u/215d6c63b088">Andrey Volodin</a> — iOS integration<br><a href="https://medium.com/u/51c63007b14d">MR</a> — review of this article</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39c84f1b9e66" width="1" height="1" alt=""><hr><p><a href="https://medium.com/prismalabs/real-time-portrait-segmentation-on-smartphones-39c84f1b9e66">Real-time Portrait Segmentation on Smartphones</a> was originally published in <a href="https://medium.com/prismalabs">Prisma Labs Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AR Synchronization System by Prisma]]></title>
            <link>https://medium.com/prismalabs/ar-synchronization-system-by-prisma-3c41ff55a2a7?source=rss----18edcfbe88a5---4</link>
            <guid isPermaLink="false">https://medium.com/p/3c41ff55a2a7</guid>
            <category><![CDATA[augmented-reality]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[framework]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Pavel Voropaev]]></dc:creator>
            <pubDate>Mon, 07 May 2018 18:21:23 GMT</pubDate>
            <atom:updated>2018-05-07T18:21:23.105Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eQRDXRuTLO16nU2Q." /><figcaption>Two synchronized devices</figcaption></figure><p>AR is taking the world by storm in providing whole new experiences for users. Well-known frameworks such as ARKit from Apple and ARCore from Google open up whole new ways to interact with real-world objects and app content. However, all AR solutions are single-session-oriented. The system usually knows close to nothing about the environment it will run in. This results in all AR things only living within the AR session life cycle. In other words, once you close the app, all of the objects appear, disappear, or reappear in completely diﬀerent areas on the screen. Moreover, there is no way for two or more users to see the same AR objects sharing the same real world points without the use of hacks like external markers (i.e. QR-codes), etc. This is what AR synchronization was made for.</p><h4>What are the possibilities?</h4><p>Our system silently analyzes the environment around the user while they are interacting with an AR object. We scan multiple key features from the user’s space: its geometry, objects of interest, etc. We then serialize this information and send it to any other current and newly entering users to the shared environment. Exchanging environment data among users provides the possibility to place AR objects into exact real world points, allowing users to see the same things from diﬀerent devices, simultaneously.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zIHj4JlYn20Ckn3Xqb4mTg.png" /><figcaption>Two person draw in the same AR space simultaneously</figcaption></figure><p>So far, with our technology, you can:</p><ul><li>Play multiplayer AR games (shoot a fireball at a friend!)</li><li>Create collaborative AR art such as 3D digital drawing in the shared environment</li><li>Create custom markers in the environment and share to other users (for example, digital pop-up instructions to show hotel guests how to use a safe box or shower)</li><li>Locate city sights using additional GPS synchronization</li></ul><h4><strong>How does it work?</strong></h4><p>There are two major steps.</p><p>The first step is to scan the environment. It usually takes a while (up to 30 seconds on average for a sufficient data dump). As the user looks around, objects around him are tracked and saved, the room’s geometry is scanned, and key feature points are located. It’s all done using our stabilization/scanning framework.</p><p>The second step is to match same object for two or more environments. A unique descriptor for every geometry point is calculated around the user and saved as a point’s features. A set of computer vision algorithms and machine-learning techniques are used to accurately find the diﬀerences between two or more coordinate systems.</p><p>There are multiple scenarios of doing that. The simplest one is when two users are simultaneously located in the same place. At first, users are prompted to look around and collect some data. After enough information is received, the data dump is serialized and exchanged via Bluetooth/Local Wi-Fi and then matched with a familiar object within the two room dumps. A peer-to-peer connection is usually established silently as a background process once two or more devices appear near to each other. The approach usually takes up to 30 seconds for users to look around in order to get an accurate data dump.</p><p>However, two users don’t have to be simultaneously in one location. In another scenario, one user marks the environment around them for others entering the environment at a later time. For example, the first users puts helpful location pins pointing to a TV, refrigerator, shower, safe box, etc — and provides AR video instructions by drawing AR welcome signs (“Welcome! :)”) in the living room, locating city sights nearby, and much more.</p><p>The framework then saves this dump once and sends it to a server. When any visitor comes into the marked environment, a data dump is first downloaded from a server to the visitor’s device. The visitor is then asked to look around a bit. The data dumps are packed very eﬃciently, which results in them being extremely lightweight (containing &lt; 1MB of information). In this scenario, it takes about 25 seconds for the user to look around in order to get an accurate match and enabling them to see all the prepared AR objects.</p><p>After a good environment match, all users will see the same AR scene corresponding to the same exact place in the real world.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FHq24ITr0omE%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DHq24ITr0omE&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FHq24ITr0omE%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/97e26f7f46896cbce5ae6f333b7f08af/href">https://medium.com/media/97e26f7f46896cbce5ae6f333b7f08af/href</a></iframe><h4><strong>What’s inside?</strong></h4><p>The AR synchronization system is shipped as an external framework that can be easily plugged into any application. The framework itself is pretty easy to use and provides a lot of boilerplate code. It has all the necessary mechanisms for scanning environment data, serializing/deserializing, and for matching this information between diﬀerent users.</p><p>It also will contain handy methods describing the current data dump’s quality as well as encouraging the user to be more active and to look around more. On every match, the framework also gives you an estimated transform, allowing you to start presenting AR content earlier.</p><p><strong>It doesn’t have any strong performance penalties since it works in the background.</strong></p><h4>The future of collaborative AR</h4><p>The AR session synchronization can open up whole new possibilities for collaborative AR experiences. Whether you’re an avid gamer, artist, or business professional looking to accommodate your clients, Prisma Labs is committed to bringing you the best technology for the most creative projects. Tell us what you can do with our homebuilt AR synchronization technology!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c41ff55a2a7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/prismalabs/ar-synchronization-system-by-prisma-3c41ff55a2a7">AR Synchronization System by Prisma</a> was originally published in <a href="https://medium.com/prismalabs">Prisma Labs Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Prisma + Huawei: A partnership for the future of mobile-based AI capabilities]]></title>
            <link>https://medium.com/prismalabs/prisma-huawei-a-partnership-for-the-future-of-mobile-based-ai-capabilities-421476d8092e?source=rss----18edcfbe88a5---4</link>
            <guid isPermaLink="false">https://medium.com/p/421476d8092e</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[mobile]]></category>
            <category><![CDATA[cameras]]></category>
            <category><![CDATA[neural-networks]]></category>
            <dc:creator><![CDATA[Pavel Voropaev]]></dc:creator>
            <pubDate>Mon, 09 Apr 2018 15:43:20 GMT</pubDate>
            <atom:updated>2018-04-09T15:21:11.353Z</atom:updated>
            <content:encoded><![CDATA[<p>Prisma Labs, a b2b company providing a wide range of camera effects, and famous for its flagship app Prisma is initiating a partnership with Huawei, one of the leading mobile phone companies to develop new AI-based camera effects specifically for Huawei smart phones.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ucSUiMnjCSGgNdZY2ZiGHg.png" /><figcaption>Google, Amazon, Microsoft, Prisma</figcaption></figure><p>Several days ago, Huawei launched a new P20 smartphone that raises the bar on modern smartphone camera quality. Most of its new exciting effects and improvements have been realized through the application of AI algorithms. The era of neural networks restricted to running on servers has passed and AI algorithms are currently being made to run seamlessly on mobile devices. In addition, these high-end algorithms can be processed without an internet connection and at record-high speeds.</p><p>As the first step of our extensive collaboration, we have developed Prisma filters tailor-made for Huawei smartphones, enabling them to run much faster on these devices, compared to those of other manufacturers. We were able to accomplish this leap forward through deep hardware integration with the Kirin 970 — the new Huawei processor. Our work demonstrates that the new Kirin processor makes it possible to process neural networks with a really fast inference. In other words, the ongoing mutual effort allows for a significant improvement in AI algorithm performance on Huawei devices. As we work on the development of a very wide range of AI effects such as segmentation (bokeh effect), face recognition, scene recognition, hands tracking for photos and video, as well as the AR experience here at Prisma, our partnership could become the next big leap forward for user experience with smartphone cameras.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*paJvjFkqUpfMiS0MJHIbJw.png" /><figcaption>Fast Prisma processing with Kirin 970</figcaption></figure><p>We are open to new opportunities and we strive to help companies improve user experience and user engagement. Get in touch with us through our website at <a href="https://prismalabs.ai/">https://prismalabs.ai</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=421476d8092e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/prismalabs/prisma-huawei-a-partnership-for-the-future-of-mobile-based-ai-capabilities-421476d8092e">Prisma + Huawei: A partnership for the future of mobile-based AI capabilities</a> was originally published in <a href="https://medium.com/prismalabs">Prisma Labs Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DIY Prisma app with CoreML]]></title>
            <link>https://medium.com/prismalabs/diy-prisma-app-with-coreml-6b4994cc99e1?source=rss----18edcfbe88a5---4</link>
            <guid isPermaLink="false">https://medium.com/p/6b4994cc99e1</guid>
            <category><![CDATA[coreml]]></category>
            <category><![CDATA[style-transfer]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[ios]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Oleg Poyaganov]]></dc:creator>
            <pubDate>Thu, 07 Sep 2017 17:21:54 GMT</pubDate>
            <atom:updated>2017-09-07T17:21:54.294Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/336/1*DgxpmB64QsEYZVQk_jCSZw.jpeg" /><figcaption>Example iOS app</figcaption></figure><p>WWDC 2017 brought us an understanding of what Apple thinks about machine learning and its applications right on mobile devices. CoreML framework makes it really easy to bring ML models into your iOS app. At Prisma Labs, we love AI and, more specifically, AI on mobile devices. We implemented our own neural network inference engine for iOS and Android about a year ago and now we are excited that Apple has presented its own native version of the similar approach. In this blog post, I’ll show you how to implement a small app for style-transfer using only open source models and CoreML.</p><p>We have shared a tool with the community recently for converting Torch7 models to CoreML: <a href="https://github.com/prisma-ai/torch2coreml">https://github.com/prisma-ai/torch2coreml</a>. I’ll use it to convert Justin Johnson’s pre-trained models from his great work “<a href="http://cs.stanford.edu/people/jcjohns/eccv16/">Perceptual Losses for Real-Time Style Transfer and Super-Resolution</a>” using repo: <a href="https://github.com/jcjohnson/fast-neural-style">https://github.com/jcjohnson/fast-neural-style</a>.</p><p>First of all, let’s formalize what we are doing. We need to obtain CoreML (.mlmodel files) models for image-to-image neural networks, which apply some “style” to a 3-channel image input and get the output image. We want to create a small app with some of these models as an example.</p><h3>Requirements</h3><p>You’ll need to have macOS, Xcode 9, Torch7, PyTorch and torch2coreml installed. There are links further in the text that describe the installation process of each component.</p><h3>Step 1: Download pre-trained models</h3><p>Let’s clone fast-neural-style repo:</p><pre>git clone <a href="https://github.com/jcjohnson/fast-neural-style">https://github.com/jcjohnson/fast-neural-style</a></pre><p>This repo contains a code to train new models and to run inference (apply style-transfer to images) using Torch7. You need to have Torch7 installed and configured properly: <a href="http://torch.ch/docs/getting-started.html#_">http://torch.ch/docs/getting-started.html</a> (only CPU version without CUDA is sufficient for this tutorial).</p><p>Then we download pre-trained style-transfer models:</p><pre>cd fast-neural-style &amp;&amp; bash models/download_style_transfer_models.sh</pre><p>The downloaded models will appear in “models/eccv16” and “models/instance_norm” subdirectories. We’ll use “instance_norm” (neural networks with InstanceNormalization layers) models but everything in this tutorial is compatible with “eccv16” models. You can train your own models as well, as it is described in fast-neural-style repo, and use it for your further experiments.</p><h3>Step 2: Prepare models for conversion</h3><p>If we dig inside fast-neural-style models, we find that the model is stored inside the Lua table with corresponding parameters about preprocessing, etc. We need to extract it. This step of the tutorial contains some simple Lua code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/67170909ba0258e42812421dfe3cfa34/href">https://medium.com/media/67170909ba0258e42812421dfe3cfa34/href</a></iframe><p>The next problem is custom Torch7 layers (implemented with Lua) inside the model. They are not part of the standard torch.nn package distribution:</p><ul><li>ShaveImage</li><li>TotalVariation</li><li>InstanceNormalization</li></ul><p>Let’s look inside each of these layers and modify the model to be ready for conversion. We are interested only in forward pass, so only “updateOutput” methods are investigated.</p><p><a href="https://github.com/jcjohnson/fast-neural-style/blob/master/fast_neural_style/ShaveImage.lua">ShaveImage</a> is a simple layer which crops input, cutting some pixels from the left, the right, the top and the bottom of the image. We can simply change it to the standard torch’s <a href="https://github.com/torch/nn/blob/master/SpatialZeroPadding.lua">SpatialZeroPadding</a> layer with negative paddings.</p><p>If we look inside the <a href="https://github.com/jcjohnson/fast-neural-style/blob/master/fast_neural_style/TotalVariation.lua">TotalVariation</a> layer, we’ll see that it is just a no-op layer on the forward phase. Thus, we can safely remove this layer from the model.</p><p>We cannot modify <a href="https://github.com/jcjohnson/fast-neural-style/blob/master/fast_neural_style/InstanceNormalization.lua">InstanceNormalization</a> layer as it has no analogue in torch.nn package, but we’ll convert it to the corresponding CoreML layer because it is supported in the latest coremltools python package. Thus, we leave this layer in the model as it is.</p><p>You can find the full script for preparing the model here: <a href="https://github.com/prisma-ai/torch2coreml/blob/master/example/fast-neural-style/prepare_model.lua">https://github.com/prisma-ai/torch2coreml/blob/master/example/fast-neural-style/prepare_model.lua</a></p><h3>Step 3: Implement converter to CoreML</h3><p>For this step of the tutorial we will use torch2coreml python package. Let’s write some python code. We have some models from the previous step that are prepared by means of Lua script. We’ll implement a specialized python converter for fast-neural-style torch models.</p><p>PyTorch is required to be installed. You can follow installation steps here: <a href="http://pytorch.org/">http://pytorch.org/</a>.</p><p>You also need to install the latest version of torch2coreml python package for python 2.7 (coremltools library only supports this version of python as the time of writing):</p><pre>[sudo] pip install -U torch2coreml</pre><p>The torch2coreml library can convert the model from the file or from the already loaded PyTorch model (it uses PyTorch internally to infer input/output shapes of some layers and to parse .t7 files). We need to load Torch7 model using PyTorch by ourselves because of the custom InstanceNormalization layer that we need to implement in python using parameters stored in the Lua model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f61e16ca3863daf7ee4e4e44fe33d80a/href">https://medium.com/media/f61e16ca3863daf7ee4e4e44fe33d80a/href</a></iframe><p>We need to turn on the “unknown_classes” argument as we have InstanceNormalization layers inside the model. Otherwise it will raise an exception.</p><p>The “Convert” function from torch2coreml runs inference on the PyTorch model, which is why we need to replace InstanceNormalization with its working analogue. Inside PyTorch we can find <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/instancenorm.py#L127">InstanceNorm3d</a> class, which is completely the same as Justin Johnson’s implementation in Lua. So, let’s implement the PyTorch legacy Module (as we work only with Torch7 equivalents inside PyTorch) using this class.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6cee0627540cc92bfeb695e39bde38f0/href">https://medium.com/media/6cee0627540cc92bfeb695e39bde38f0/href</a></iframe><p>Next, we replace every unknown InstanceNormalization object inside our model with instances of our newly implemented class. From this point, we have the fully working PyTorch model and it is ready to be converted.</p><p>We have no implementation for converting our custom InstanceNormalization layer inside torch2coreml library, but, fortunately, it has a mechanism for adding our own converter function for unknown layers.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/535e6e1b1dc593e6edfd8aa48a4955b8/href">https://medium.com/media/535e6e1b1dc593e6edfd8aa48a4955b8/href</a></iframe><p>Now everything is ready to convert prepared torch7 models into CoreML. Use this full python script for converter: <a href="https://github.com/prisma-ai/torch2coreml/blob/master/example/fast-neural-style/convert-fast-neural-style.py">https://github.com/prisma-ai/torch2coreml/blob/master/example/fast-neural-style/convert-fast-neural-style.py</a>. It has some additional modifications, for example, to fix the loading of the SpatialFullConvolution layer.</p><p>At this point, you have .mlmodel files which are ready to be embedded inside the iOS app. You can run generated CoreML models from python to test that style-transfer is applied properly to images. I have created a simple script for this purpose:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8532062551626397a56e444bff9f6fb6/href">https://medium.com/media/8532062551626397a56e444bff9f6fb6/href</a></iframe><p>Be mindful to use images with compatible input size for your models, or you can simply add image scaling before invoking the “predict” method of MLModel.</p><p>Example output for a Mosaic model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*7qm9B_INm917HHI-ZCTnNg.jpeg" /></figure><h3>Step 4: iOS app</h3><p>I have implemented a simple app using 4 style-transfer models from the previous step. You can find a detailed explanation of how to use CoreML inside the iOS app in Apple tutorials and documentation as it is beyond the scope of the current article. It is quite simple and intuitive! You can find the full iOS app source code (the first screenshot in this tutorial is this exact app) here: <a href="https://github.com/prisma-ai/torch2coreml/tree/master/example/fast-neural-style/ios">https://github.com/prisma-ai/torch2coreml/tree/master/example/fast-neural-style/ios</a>.</p><h3>Conclusion</h3><p>We have implemented converter to CoreML for original Justin Johnson’s fast-neural-style models using the torch2coreml package. The obtained models can be used in iOS and macOS apps. You can use the torch2coreml package for not only style-transfer, but for other model types as well. Feel free to contribute.</p><p>The full source code for this tutorial can be found here: <a href="https://github.com/prisma-ai/torch2coreml/tree/master/example/fast-neural-style">https://github.com/prisma-ai/torch2coreml/tree/master/example/fast-neural-style</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6b4994cc99e1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/prismalabs/diy-prisma-app-with-coreml-6b4994cc99e1">DIY Prisma app with CoreML</a> was originally published in <a href="https://medium.com/prismalabs">Prisma Labs Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>