<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pierre Pontevia on Medium]]></title>
        <description><![CDATA[Stories by Pierre Pontevia on Medium]]></description>
        <link>https://medium.com/@the_ponpon_?source=rss-c67417c659b2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*lmZxqr2hdVTwGN7H.png</url>
            <title>Stories by Pierre Pontevia on Medium</title>
            <link>https://medium.com/@the_ponpon_?source=rss-c67417c659b2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 02:14:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@the_ponpon_/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Developing magicplan-AI (Part 3/3)]]></title>
            <link>https://medium.com/@the_ponpon_/developing-magicplan-ai-part-3-3-a0ea3270a96e?source=rss-c67417c659b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/a0ea3270a96e</guid>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[coreml]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[augmented-reality]]></category>
            <dc:creator><![CDATA[Pierre Pontevia]]></dc:creator>
            <pubDate>Fri, 07 Sep 2018 22:13:35 GMT</pubDate>
            <atom:updated>2018-09-07T22:13:35.744Z</atom:updated>
            <content:encoded><![CDATA[<h3>magicplan-AI, a journey into the wonderful world of Deep Learning (Part 3/3)</h3><h3>Previously in the wonderful journey of magicplan-AI…</h3><p>In <a href="https://medium.com/@the_ponpon_/magicplan-ai-a-journey-in-the-wonderful-world-of-deep-learning-part-2-3-fdf3a62b71ff">part 2</a>, we showed how you could reach a good level detection accuracy on a Deep Learning model running on powerful GPU, when you have the right expertise on Deep Learning training.</p><p>Unfortunately, this is not enough when you want to implement the feature on a smartphone and have to deal with really limited hardware resources both in terms of memory and computing power.</p><h3>Part 3 — Embedding the feature into a smartphone</h3><p>Previous work allowed to design and train a “good enough” door / window object detection model. However, even just for the inference part, this architecture can only run on powerful NVIDIA GPU. It is incompatible with the smartphone limited hardware resources, both in terms of memory requirements and processing time.</p><p>This is a huge problem that kept us stuck for a while.</p><h4>The remote approach</h4><p>At first, the contemplated solution was to offload the GPU computation on the cloud.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LHMrOGTpHLEJtB3XT3GdWg.png" /><figcaption>Remote GPU based solution</figcaption></figure><p>The main advantage of this approach is that the server has the required GPU memory and computation power to run properly the model. So, this solution is available day one.</p><p>However, 3 factors made this approach problematic:</p><ol><li>magicplan is designed to run both online and offline. Relying on an remote connection for windows / doors detection would change this paradigm,</li><li>the upload and download durations create an unwanted and unpredictable lag that can be really annoying during the real-time capture session,</li><li>there was an uncertainty in the cost of deploying GPU servers on the cloud in order to scale with the potential demand, once in production.</li></ol><p>Nevertheless, the approach was very useful to “simulate”, locally on a private network and in real capture conditions, the reliability of the model and validate that the user experience was acceptable.</p><p>Eventually, the combination of 3 solutions allowed us to overcome this critical barrier:</p><ol><li>Applying quantising approach to reduce memory footprint through lowering floating precision,</li><li>Applying the Teacher / Student approach to our model to “shrink” it to an acceptable memory size, while keeping the same detection accuracy,</li><li>Moving part of the model from the TensorFlow standard framework to the Apple CoreML accelerated framework to optimise computing performance on the smartphone.</li></ol><h4>Applying quantising approach</h4><p>Quantisation is the idea of moving from floating point to integer arithmetic, thus saving in memory and computation time. This was a quick win as TensorFlow offers some tools to revisit a given model to perform quantisation very easily.</p><p>More on the topic:</p><ul><li><a href="https://arxiv.org/abs/1510.03009"><em>Zhouhan Lin, Matthieu Courbariaux, Roland Memisevic, and Yoshua Bengio. Neural networks with few multiplications. CoRR, abs/1510.03009, 2015.</em></a></li></ul><h4>Applying the Teacher / Student approach</h4><p>We have to thank the <a href="https://mila.quebec/en">MILA laboratory</a> with whom we collaborated in a <a href="https://www.nrc-cnrc.gc.ca/eng/irap/index.html">IRAP</a> funded initiative as they recommended us this approach as a way of shrinking the model and it worked!</p><p>It was key to have MILA expertise guiding us on this solution, a solution we would have most likely not considered otherwise.</p><p>This advanced approach is quite fascinating. To make it short, it relies on the idea that small models are worse at generalising than large models during the training procedure. So, in order to have good training for a small model, it helps to have a large model “distilling” knowledge from its internal layers into some internal layers of the small model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B97HzAbmw0VD5mjXqzZ2XQ.png" /><figcaption>Teacher — Student implementation</figcaption></figure><p>More on the topic:</p><ul><li><a href="https://arxiv.org/abs/1503.02531"><em>Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In NIPS Workshop, 2014</em></a><em>.</em></li><li><a href="https://arxiv.org/abs/1412.6550"><em>Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015.</em></a></li></ul><h4>Moving to Apple CoreML GPU accelerated framework</h4><p>While Deep Learning frameworks on PC with GPU acceleration are quite mature, this is not true when it comes to porting these frameworks to iOS or Android:</p><ul><li>Embedded versions of the main frameworks (like <a href="https://www.tensorflow.org/mobile/">TensorFlow Mobile</a>) are still in their early stages and not taking full advantage of the smartphone HW.</li><li>On the other end, there is <a href="https://developer.apple.com/documentation/coreml?language=objc">Apple CoreM</a>. CoreML is the official Apple Machine Learning framework for iOS, optimised for on-device performance. Unfortunately, nowadays, it only covers a limited subset of computation nodes implemented in TensorFlow.</li><li>In addition, there is still no standard interoperability format allowing one to port easily a model from one framework to another up to the embedded implementation, even if initiatives like <a href="https://onnx.ai/">ONNX</a> or <a href="https://www.khronos.org/nnef">NNEF</a> are making their way.</li></ul><p>As a result, the only way to get the best on iOS is to manually refactor the full graph into several parts to easily port the relevant subgraph to CoreML.</p><p>After all these operations, the resulting SSD model has the performance of the best R-CNN model validated on large GPU while occupying 14 MB and performing an inference in 250 ms on iPhone X. Quite an achievement!</p><h4>Lessons learnt</h4><p>When it comes to embedding models in smartphones, the reality is that the embedded frameworks and interoperability tools are still in the early stages, in stark opposition with the experience we had with the POC on GPU powered PCs.</p><p>In addition, advanced Deep Learning training techniques are required to obtain good results. You need a Deep Learning specialised team to deal with these advanced training procedures as well as playing with interoperability and complex acceleration codes.</p><p>This is where a collaboration with top AI academic organisations can be decisive in choosing the right directions and options to reach the targets.</p><h3>Conclusion</h3><p>Our journey in Deep Learning has been an amazing but challenging experience.</p><p>Truly, the 80 / 20 rule applies:</p><ul><li>it is easy to get some encouraging results thanks to the availability of mature prototyping tools and off-the-shelf models at start,</li><li>when it comes to a real app, you are much more on your own for deploying the technology on a smartphone and dealing with advanced training techniques to produce smartphone constraints compatible models.</li></ul><p>We are now in the beta phase of magicplan-AI. That means that the feature is live on magicplan 7.2 for iOS!</p><p>It is available to anybody who wants to <a href="https://www.magic-plan.com/project-magicplan-ai/">register to the magicplan-AI beta</a>.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FUgBFa6IMW60%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DUgBFa6IMW60&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FUgBFa6IMW60%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/0f2e4b016cd8056f50b8e5a4d9b4b1a5/href">https://medium.com/media/0f2e4b016cd8056f50b8e5a4d9b4b1a5/href</a></iframe><p>The purpose of this beta is to improve the training set as well as to understand better the user experience and identify what works and what needs to be changed.</p><p>There is still some significant work to make it an official feature but we believe that eventually AI will become part of the core of magicplan, just like AR has been in magicplan for the last 5 years.</p><p>Whether it is through an extension of its usage (to electrical outlets or corner detection), or by implementing a continuous training approach to the models, Sensopia Research team has a lot of exciting Deep Learning work ahead!</p><p>And to that end, <a href="https://www.linkedin.com/jobs/view/856086775/">we are hiring</a>!</p><h3>Acknowledgments</h3><p>I want to thank Jonathan Aigrain and Vahid Ettehadi, our two top research engineers for their outstanding work at making magicplan-AI happen. It is their achievement first.</p><p>I also want to thank Sylvain Laroche from the <a href="https://www.nrc-cnrc.gc.ca/eng/">National Research Council Canada</a> for the trust he has always demonstrated in Sensopia capabilities. The NRCC has helped us at critical stages of the company over the years, allowing our small structure to achieve ambitious projects while mitigating the financial risks.</p><p>Finally, our gratitude goes to Mike Pieper (<a href="https://mila.quebec/en/">MILA R&amp;D and Technological Transfer Team</a>) for all the good advices he gave us. That made the difference at the end.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a0ea3270a96e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Developing magicplan-AI (Part 2/3)]]></title>
            <link>https://medium.com/@the_ponpon_/developing-magicplan-ai-part-2-3-fdf3a62b71ff?source=rss-c67417c659b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/fdf3a62b71ff</guid>
            <category><![CDATA[coreml]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[augmented-reality]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[object-detection]]></category>
            <dc:creator><![CDATA[Pierre Pontevia]]></dc:creator>
            <pubDate>Tue, 28 Aug 2018 21:18:07 GMT</pubDate>
            <atom:updated>2018-09-07T22:24:17.071Z</atom:updated>
            <content:encoded><![CDATA[<h3>magicplan-AI, a journey into the wonderful world of Deep Learning (Part 2/3)</h3><h3>Previously in the wonderful journey of magicplan-AI…</h3><p>In <a href="https://medium.com/@the_ponpon_/magicplan-ai-a-journey-in-the-wonderful-world-of-deep-learning-part-1-3-83bcdd91228e">part 1</a>, we saw how today’s Deep Learning tools and data ecosystems make it easy to have an early prototype to assess the feasibility of a common Deep Learning task.</p><p>That said, it is one thing to have a workable prototype showing the potential of the approach, it is another thing to reach a reliable enough level of detection to allow the feature to be put in the hands of millions of users.</p><h3>Part 2 — Iterating on data and model to reach acceptable performance</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FUgBFa6IMW60%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DUgBFa6IMW60&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FUgBFa6IMW60%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/547115c210375a7f24d86f2a3198e13a/href">https://medium.com/media/547115c210375a7f24d86f2a3198e13a/href</a></iframe><p>As shown in this video, the objective of magicplan-AI is to let the user add a door or a window on a given wall, both at the correct position and with the correct size by simply taking a picture of the object while capturing the room.</p><p>This capture needs to be reliable in terms of location and size and real-time to avoid downgrading the quality of the already in-production room capture.</p><p>By reliable, we mean several things:</p><ul><li>ability to detect the existence of a door or a window on a picture,</li><li>ability to classify correctly as window or door the detected object,</li><li>ability to provide an accurate bounding box for the detected object.</li></ul><h4>Introducing relevant performance metrics</h4><p>To measure the performance of such a system, we use a <a href="https://en.wikipedia.org/wiki/F1_score">metrics called F1</a> , that “summarises” in one single value the combination of most of above requirements (important factors such as true positive, false positive and false negative).</p><p>Note: there are other metrics like the mAP (<a href="https://medium.com/@jonathan_hui/map-mean-average-precision-for-object-detection-45c121a31173">mean Average Precision</a>) that are more complete, but for the sake of simplicity, I will stick to F1 in this story.</p><p>It is important to understand that, depending on the usage scenario for a given object detector, errors of different nature can have very dissimilar impact.</p><p>A powerful illustration of this is a cancer cell detector, for which the impact of not detecting a real cancerous cell has catastrophic consequences compared to the impact of wrongly classifying a sane one.</p><p>The same asymmetry exists in magicplan capture scenario:</p><ul><li>wrongly classifying a correctly detected and framed window as a door is easy to correct by the user with the proper UI (one click operation),</li><li>on the opposite, poorly adjusting bounding box of a correctly detected and classified window will oblige the user to switch to edition mode and input the correct measurement obtained by alternate means (very time consuming and disruptive).</li></ul><p>This asymmetry in the cost of object detection errors needs to be reflected in the used metrics by handling specifically each error type.</p><p>Once the metrics are defined, we can rely on a quantitative approach to evaluate the different iterations of our object detector:</p><ul><li>our POC “SSD Inception V2” model trained with ImageNet database had a F1 score of 0.62,</li><li>qualitatively, the POC model was failing to detect some types of doors and lacked accuracy in the bounding box.</li></ul><p>Three methods have been used to improve the F1 score:</p><ol><li>improving / augmenting the training dataset,</li><li>playing with the loss function,</li><li>exploring different architectures.</li></ol><h4>Improving / augmenting the training dataset</h4><p>In the world of Deep Learning, data is key and there are a lot of identified techniques to improve the training database.</p><p>The first one is obviously to increase the size of the training dataset by adding more images or data augmentation techniques.</p><p>But surprisingly, for us, the most significant improvement came from cleaning the annotated ImageNet database.</p><p>There are several reasons to that:</p><ol><li>not all images with doors or windows are relevant to magicplan interiors scenario use cases — we call this a <strong>domain shift issue</strong>,</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*xG-78N1hXM3P0GFFnkF5PQ.png" /><figcaption>Not the type of door we are interested in!</figcaption></figure><p>2. door and windows are generic words that you can find in situations that have nothing to do with a house interior — we call this a <strong>semantic shift issue</strong>,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*H2mrxdd5Yxzf8B1EO8dygQ.png" /><figcaption>A plane window is not a house window!</figcaption></figure><p>3. crowd-sourcing annotation has its limits and mistakes can be included in the annotation — we call this a <strong>mis-annotation issue,</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*R_qniWt-URyUyG24D2tovA.png" /><figcaption>Wrong annotation!</figcaption></figure><p>4. while it is ok to annotate only one window in a picture containing several windows for a classification task, it is not ok to forget to annotate visible windows for an object detection task — we would call this a <strong>partial-annotation issue</strong>,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*KFJWrWaMlHhPtLztxVOnbQ.png" /><figcaption>Somebody forgot to annotate the windows!</figcaption></figure><p>5. we realized that doors when shot open can be problematic for magicplan-AI as we actually are looking for the door frame, not the mobile part — we call this a <strong>task annotation shift issue.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/332/1*bcH50t9OLNx6ubztPvgb1w.png" /><figcaption>Not the part of the door we want to detect!</figcaption></figure><p>For each type of issue, a fix was identified and applied.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X3xPF1vDJbshauNRoPgKGg.png" /><figcaption>Training set issues and fixes</figcaption></figure><p>Revisiting the training dataset and removing, re-annotating or re-weighting images, improved F1 score by more than 23%, reaching an interesting 0.85.</p><h4>Playing with the loss function</h4><p>Training correctly a model in Deep Learning is as much about the right dataset as it is about the right correction when an error is found during the training.</p><p>Fortunately for us, a large literature is available on identifying the right Loss function to use. Even better, on the particular case of Object detection, <a href="https://research.fb.com/facebook-open-sources-detectron">Facebook Detectron project</a> identified some key improvement in the way of applying the right Loss function, called Focus Loss, that were very easy to implement for us.</p><p>As a result, combining the training database quality improvement with the introduction of a better fitted loss function, we were able to significantly improve the F1 score as illustrated below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*3iHsp0GywEZO4l9LehMRaQ.png" /><figcaption>F1 score evolution according to Training set fixes</figcaption></figure><h4>Exploring different architectures</h4><p>Academic research has been quite active in the Object Detection field and several architectures are available for Deep Learning for Object Detection.</p><p>They can be grouped along two axes:</p><p>A — the type of feature extractor that processes the input image:</p><ul><li>Multiple architectures from lite (<a href="https://arxiv.org/abs/1704.04861">MobileNet</a>) to heavy (<a href="https://arxiv.org/abs/1602.07261">Inception</a>, <a href="https://arxiv.org/abs/1409.1556">VGG</a>, <a href="https://arxiv.org/abs/1512.03385">ResNet</a>)…</li><li>the bigger the feature extractor in terms of parameters, the better the descriptors but the more memory and time it takes to perform</li></ul><p>B — the number of steps to do the full detection:</p><ul><li>direct forward approach (<a href="https://arxiv.org/abs/1506.02640">YOLO</a>, <a href="https://arxiv.org/abs/1512.02325">SSD</a>) where one single network will detect the bounding boxes and classify them at the same time,</li><li>two steps approach (<a href="https://arxiv.org/abs/1506.01497">Faster RCNN</a>, <a href="https://arxiv.org/abs/1605.06409">R-FCN</a>) where a first network detects potential rough bounding boxes candidates while the second performs classification and bounding boxes fine tuning.</li></ul><p>As expected, the more complex the architecture, the better the performance (see graphic below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/943/1*LUdE-CnibX8cTCHnsZzxwQ.png" /><figcaption>Object Detection Architectures and performances (COCO dataset) — <a href="https://medium.com/@phelixlau/speed-accuracy-trade-offs-for-modern-convolutional-object-detectors-bbad4e4e0718">Source</a></figcaption></figure><p>However, what we discovered quite early is that even for the inference task (the task of running the model to perform the object detection on an image — not the training task which requires much more resources), not all architectures fit the constraints of running on a mobile device.</p><p>Two reasons to that:</p><ul><li>contrary to modern GPU boards with more than 10Gb of RAM, even the latest iPhone X has only 3Gb of RAM,</li><li>real-time constraint means that we can’t afford an object recognition lasting more than 1.0 second without creating a really annoying lag in the user experience.</li></ul><p>Some architectures do not fit in memory on the device. Some other do BUT it takes several seconds to process one object detection, which is not acceptable in the magicplan real-time capture scenario.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*St147d0ea0ZF8zw3Oc0x1Q.png" /><figcaption>Evaluation of several architectures</figcaption></figure><h4>Lessons learnt</h4><p>Contrary to the first “quick &amp; easy” stage, being able to play with all the options in the “off the shelf” models requires several conditions:</p><ul><li>a good methodology to objectively measure the progress,</li><li>a good understanding of what makes a good training convergence and what can break it,</li><li>a good understanding of the underlying neural nets architecture and nodes,</li></ul><p>In our case, this could not have happened without the presence of two full time PhDs in artificial intelligence / deep learning in the team who master these challenges.</p><h3>Coming next</h3><p>At this point we have a best in class model doing a good job in object detection but too big to run on any modern smartphone.</p><p><a href="https://medium.com/@the_ponpon_/developing-magicplan-ai-part-3-3-a0ea3270a96e">In last part</a>, we will describe more in detail the work required to move from a PC based solution to a smartphone embedded one.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fdf3a62b71ff" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Developing magicplan-AI (Part 1/3)]]></title>
            <link>https://medium.com/@the_ponpon_/developing-magicplan-ai-part-1-3-83bcdd91228e?source=rss-c67417c659b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/83bcdd91228e</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[coreml]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[augmented-reality]]></category>
            <dc:creator><![CDATA[Pierre Pontevia]]></dc:creator>
            <pubDate>Tue, 21 Aug 2018 19:46:51 GMT</pubDate>
            <atom:updated>2018-08-29T13:40:40.946Z</atom:updated>
            <content:encoded><![CDATA[<h3>magicplan-AI, a journey into the wonderful world of Deep Learning (Part 1/3)</h3><h3>Introduction</h3><p>Computer vision is undergoing a spectacular revolution thanks to AI advances, particularly in Deep Learning. Every day, a new article shows capabilities you could not imagine 5 years ago. Whether it comes from the GAFA teams or from the top labs’ spinoffs, it looks like AI for image understanding is making its way in the enterprise.</p><p>But what does this mean for a small company with more classical computer vision at the core of its product? <br>What does it take to do the big dive into Deep Learning? <br>How robust is this new approach to production requirements such as accuracy and speed on a smartphone app?</p><p>My name is Pierre Pontevia. I am a founder and Chief Research Officer for <a href="http://www.magic-plan.com/">magicplan</a>. Today I would like to share with you our journey toward implementing a Deep Learning based feature into our app.</p><p>In this article, we will not discuss the Singularity, nor beating any Go master or the latest breakthroughs in Generative Networks. We will instead share the practical steps necessary for a team of engineers to take a Deep Learning feature out of the “lab” and embed it in an app used by millions of smartphone owners.</p><h3>Quick introduction to magicplan</h3><p><a href="http://www.magic-plan.com/">magicplan</a> is an <a href="https://itunes.apple.com/ca/app/magicplan/id427424432?mt=8">iOS</a> and <a href="https://play.google.com/store/apps/details?id=com.sensopia.magicplan&amp;hl=en_CA">Android</a> app allowing anybody to create a floor plan of his home in a matter of minutes and is as simple as taking a picture .</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F6I0UMLcWW3E%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D6I0UMLcWW3E&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F6I0UMLcWW3E%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/264eb9e4d04fd0c0eaddcd833f87dc03/href">https://medium.com/media/264eb9e4d04fd0c0eaddcd833f87dc03/href</a></iframe><p>With more than 16 million downloads, magicplan is the reference for floor plans and renovation apps.</p><p>From day one, our app implemented “computer vision” based reality capture and camera tracking capabilities making it one of the first apps to offer true AR experience on a smartphone.</p><p>But today, this is not enough and our customers are asking for the ability to extract more information during the capture process.</p><p>This is where Deep Learning enters the game, providing magicplan with a new dimension of room understanding, as illustrated in this video:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FUgBFa6IMW60%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DUgBFa6IMW60&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FUgBFa6IMW60%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/547115c210375a7f24d86f2a3198e13a/href">https://medium.com/media/547115c210375a7f24d86f2a3198e13a/href</a></iframe><p>You can see how a user captures, in real-time, location and dimensions of the doors and windows of a room, literally only by taking pictures of them.</p><p>This is how it works: each time a user takes a shot of a wall, magicplan processes the taken picture with a special type of Deep Learning network that detects, locates and frames windows and doors present anywhere on the shot.</p><p><strong>The benefits in terms of convenience and time saving for the user are tremendous.</strong></p><p>In this blog, we share our experience about what it took to make this video a reality. The story is split in 3 main parts, each describing a major stage of our approach:</p><ol><li>Developing a POC to assess feasibility,</li><li>Iterating on data and model to reach acceptable performance,</li><li>Embedding the feature into a smartphone.</li></ol><h3>Part 1 — Developing a POC to assess feasibility</h3><p>As head of a small research team in a small startup, my first duty is to ensure we are working on something impactful on the product. That means we need to identify quickly and at a minimal cost whether a potential innovative technology is worth being investigated or not.</p><p>A Proof of Concept (POC) is key to achieve this. By POC, we mean translating an approach / algorithm / solution described mostly in academic articles into some executable code that we will run to evaluate the performance of the algorithm under circumstances as close as possible to the end case we want to solve.</p><p>Several factors helped us to really quickly develop and evaluate the POC:</p><ul><li>Almost all the tools and frameworks used to perform Deep Learning are open-source, lowering significantly the barrier of accessing a professional / reliable environment,</li><li>The academic world is very active in the field of “Deep Learning for Computer Vision” and researchers share what they develop with the community, simplifying drastically the work required to translate an idea into a code,</li><li>The GAFAs have done a tremendous job of structuring and sharing the most popular Deep Learning models with the AI community, accelerating the task of testing a particularly popular model on a specific sub-domain,</li><li>Data is key in Deep Learning. Fortunately, there are some large and free annotated databases available on the Internet for training models,</li><li>Deep Learning is computation-intensive. The good news is that Deep Learning hardware is getting cheaper.As an alternative, you can start using GPU on the cloud and avoid investments at the beginning (when you are not sure how promising the idea is).</li></ul><p>Helped by these facilitators, we structured the POC in 4 main steps:</p><ul><li>identifying the field of academic research covering our needs,</li><li>getting familiar with the most popular Deep Learning framework,</li><li>leveraging on-the-shell pre-trained relevant models from the model zoos available on the internet,</li><li>testing them on our specific challenge: the capturing of doors and windows.</li></ul><h4>The academic research field</h4><p>In our case, it was easy to identify ‘object detection (or localisation)’ as the research domain we wanted to dig in. Interestingly, ‘object localisation’ is just the second level in the scale of complexity of what you can understand in an image as described in the table below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/943/1*b7qpDTRtPyr1VgJdY3m8Qg.png" /><figcaption>Scene understanding levels (<a href="https://arxiv.org/pdf/1704.06857.pdf">source</a>)</figcaption></figure><h4>Deep Learning frameworks</h4><p><a href="https://www.tensorflow.org/">TensorFlow</a> (owned by Google) is the most used Deep Learning framework today. That means there is a high chance any work described in an article has been implemented or reproduced using TensorFlow. As a side note on the framework selection, popularity is an important criterion but not the only one. Ease of use and/or maturity in production tools are also important.</p><h4>Accessing pre-trained models</h4><p>The TensorFlow team maintains a Model Zoo in an additional <a href="https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md">repository</a>. There you can find multiple python modules offering APIs to implement the most popular Deep Learning models, including object detection models. These models come with pre-trained configurations. The training has been done on available data sets such as <a href="http://cocodataset.org/#home">COCO</a>. That saved us time and developments in the early stages of the evaluation.</p><p>The ‘object detection’ implementation in the library offers the ability to use ‘Transfer Learning’ through fine-tuning. In this approach, you start with an existing model, pre-trained on millions of images (such as COCO dataset), and you re-train on a smaller set of images (a couple hundreds images are enough for early testing) of your sub-domain (windows and doors pictures taken from the interiors in our case).</p><p>This approach saved us the burden of two significant challenges during the POC:</p><ul><li>the fact that usually, deep learning needs TONS of data in order to generalise correctly on a given problem. And it is quite unlikely that you have this volume of data available at the beginning of your investigation,</li><li>the fact that training on millions of images takes a lot of time (and money) if you do not have the proper hardware resources at hand. This is likely when you are a beginner in the world of Deep Learning.</li></ul><p>The challenge of finding several thousands of already annotated images of doors and windows was overcome by the availability of the <a href="http://www.image-net.org/">ImageNet</a> training dataset.</p><p>ImageNet offers access to more than 16 millions images indexed with a semantic tree of over 35 000 different categories. By just selecting the images containing the keywords ‘door’ or ‘window’, we were able to gather around 3500 images, enough to do a proper “Transfer Learning” based training.</p><h4>Testing on our use case</h4><p>Keeping in mind the real-time constraints of the magicplan capture scenario, we chose to start with a simple and popular model called “<a href="https://arxiv.org/abs/1512.02325">SSD Inception V2</a>” (more details in next part), to see how well it was at dealing with the task.</p><p>We re-trained the model with 3300 images split into a training dataset and a validation dataset as recommended for a proper training. The remaining 200 images were used as a test set to measure the progress of our different iterations.</p><p>In a matter of weeks, we were able to achieve a first acceptable level of performance with the trained model. Far from being perfect, the results were good enough to take the decision of investing more time and resources into the Deep Learning based object detection for windows and doors.</p><h4>Lessons learnt</h4><p>The main lesson of this first stage is that, when it comes to implementing features relying on validated Deep Learning techniques (such as object detection), you can and should use all the existing Deep Learning resources available on the Internet to maximise the efficiency of developing and testing your prototype.</p><p>That said, it is one thing to have a workable prototype showing the potential of the approach. It is another thing to make it part of an app downloaded by millions of users every day.</p><h3>Coming next</h3><p>In part 2 of 3 (<a href="https://medium.com/@the_ponpon_/developing-magicplan-ai-part-2-3-fdf3a62b71ff">link</a>), we will describe more in detail the work required to move from a prototype to a usable feature in term of performance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=83bcdd91228e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>