<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Chuan Li on Medium]]></title>
        <description><![CDATA[Stories by Chuan Li on Medium]]></description>
        <link>https://medium.com/@c_61011?source=rss-9f394e8bf587------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*lRPunmUQAwjwcI4D</url>
            <title>Stories by Chuan Li on Medium</title>
            <link>https://medium.com/@c_61011?source=rss-9f394e8bf587------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 17:51:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@c_61011/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why My Multi-GPU training is slow?]]></title>
            <link>https://medium.com/@c_61011/why-multi-gpu-training-is-not-faster-f439fe6dd6ec?source=rss-9f394e8bf587------2</link>
            <guid isPermaLink="false">https://medium.com/p/f439fe6dd6ec</guid>
            <dc:creator><![CDATA[Chuan Li]]></dc:creator>
            <pubDate>Fri, 20 Jul 2018 23:53:39 GMT</pubDate>
            <atom:updated>2018-07-24T01:24:48.898Z</atom:updated>
            <content:encoded><![CDATA[<ul><li>Many deep learning tutorials are not incentivized to showcase the advantage of a multi-GPUs system.</li><li>The fix: Use a bigger model, larger batch size and convolution layers.</li></ul><h3>Prerequisites</h3><ul><li>Basic understanding of <a href="http://cs231n.github.io/convolutional-networks/">convolution neural networks</a></li><li>A multi-GPU machine for testing the code, like <a href="https://lambdal.com/products/quad">this one</a>.</li><li>A working deep learning environment. You can it up using this <a href="https://lambdal.com/lambda-stack-deep-learning-software">one-line installation</a>.</li></ul><h3>Table of Contents</h3><ul><li><a href="https://medium.com/p/f439fe6dd6ec#4096">The miscarriage of justice</a></li><li><a href="https://medium.com/p/f439fe6dd6ec#6020">The red flags</a></li><li><a href="https://medium.com/p/f439fe6dd6ec#4b1a">The fixes</a></li><li><a href="https://medium.com/p/f439fe6dd6ec#6cca">The code</a></li></ul><p><strong>The miscarriage of justice</strong></p><p>Lambda Labs has served many customers with hardware for machine learning. One common question we got from our customers is: “Why training with multiple GPUs is not faster on my machine?”. For example:</p><p>“I paralleled this awesome <a href="https://github.com/fchollet/deep-learning-with-python-notebooks/blob/master/3.7-predicting-house-prices.ipynb">Keras tutorial</a>. However, it is not faster than running it with a single GPU. Even worse, it is actually slower. What’s wrong?”</p><p>This is where many people start to worry about if there is something wrong with their GPUs, the operation system or the deep learning frameworks they installed. However, in almost every case we’ve encountered, the source of the problem is the demo code used they use — those demo code simply weren’t designed to show case the advantage of multiple GPUs.</p><p>This article is about clarifying such doubts and showing an example of the right way to compare mulit-GPU with single-GPU for training a deep neural networks. But first, let’s brush up some basics:</p><ul><li>A single GPU also runs computation in parallel.</li><li>Multiple GPUs only help when the requested computation has already saturated a single GPU. For example, in the context of deep learning, we can increase the demand of computation by using a larger network (more computation per data sample), or using a larger batch size (more samples per training step).</li></ul><p>Next, we will see why the above Keras example can not unleash the power of multiple GPUs for training a deep neural networks.</p><p><strong>The red flags</strong></p><p>In order to NOT benefit from multiple GPUs, you can:</p><ul><li>Use a very small network.</li><li>Use very small batch size.</li><li>Create overhead for updating parameters across multiple GPUs.</li></ul><p>Let’s check these red flags one by one in the above Keras example. The model used in the example is a three-layer perceptron:</p><pre># Define the model<br>model = models.Sequential()<br>model.add(layers.Dense(64, activation=&#39;relu&#39;,<br>                       input_shape=(train_data.shape[1],)))<br>model.add(layers.Dense(64, activation=&#39;relu&#39;))<br>model.add(layers.Dense(1))</pre><pre>parallel_model = keras.utils.multi_gpu_model(model, NUM_GPU)<br>parallel_model.compile(optimizer=&#39;rmsprop&#39;, loss=&#39;mse&#39;, metrics=[&#39;mae&#39;])</pre><pre># Train the model<br>parallel_model.fit(train_data, train_targets,<br>                   epochs=80, batch_size=16, verbose=0)</pre><p>BTW the input is a 13 dimensional vector (hand-crafted features for predicting house price). So the total number of parameters is 13 x 64 + 64 x 64 + 64 x 1= 4992. To be fair, a network at this scale doesn’t need a GPU to train — <strong>red flag number one</strong>.</p><p>In the meantime, the example used batch size of 128. There is no world where such a small training job will saturate the computation power of a beefy GPU like <a href="https://www.tomsguide.com/us/nvidia-gtx-1080-ti-benchmarks,review-4241.html">this one</a>. And running the job with multiple-GPUs just make all of them idle for most of the time — <strong>red flag number two</strong>.</p><p>But why some times multi-GPU training is slower?</p><p>Well, training with multiple GPUs also introduces overhead. In particular, the need for parameter update across multiple GPUs, either through the CPU in a <a href="https://www.tensorflow.org/performance/performance_models">parameter-server setting</a>, or directly between the GPUs in a <a href="https://www.tensorflow.org/performance/performance_models">replicated setting</a>. The more GPUs you use, the bigger overhead they bring. This won’t be a problem if the GPU computation cycle (one forward step plus one backward step) takes long enough to “hide” the overhead. However, if the GPU computation cycle is short, it can be the case that parameter update takes longer and becomes the bottleneck of the training. This is more likely to happen to networks that use a lot densely connected layers — they introduce a lot of non-shared parameters, which is not a very efficient way to increase the GPU computation cycle — <strong>red flag number three</strong>.</p><p><strong>The fixes</strong></p><p>Here is what can be done to show case the advantage of training a deep neural networks with multiple GPUs:</p><pre>model = models.Sequential()  <br>model.add(keras.layers.Conv2D(128, 3, activation=&#39;relu&#39;,                                  <br>                              input_shape=(CHANNEL, HEIGHT, WIDTH)))    <br>model.add(keras.layers.MaxPooling2D(pool_size=2, strides=2,   <br>                                    padding=&#39;valid&#39;))  <br>model.add(keras.layers.Conv2D(256, 3, activation=&#39;relu&#39;))  model.add(keras.layers.MaxPooling2D(pool_size=2, strides=2, <br>                                    padding=&#39;valid&#39;))  <br>model.add(keras.layers.Conv2D(512, 3, activation=&#39;relu&#39;))  model.add(keras.layers.MaxPooling2D(pool_size=2, strides=2, <br>                                    padding=&#39;valid&#39;))  <br>model.add(keras.layers.Conv2D(64, 3, activation=&#39;relu&#39;))  model.add(keras.layers.MaxPooling2D(pool_size=2, strides=2, <br>                                    padding=&#39;valid&#39;))  <br>model.add(keras.layers.Reshape((-1,)))  model.add(keras.layers.Dense(NUM_CLASSES, activation=&#39;softmax&#39;))<br><br>parallel_model = keras.utils.multi_gpu_model(model, NUM_GPU)<br>parallel_model.compile(loss=&#39;categorical_crossentropy&#39;,                        <br>                       optimizer=&#39;rmsprop&#39;)<br>parallel_model.fit(x, y, epochs=NUM_EPOCHS,                   <br>                   batch_size=BATCH_SIZE * NUM_GPU)</pre><p>Compared to the origin Keras example, this one uses a larger networks (more layers), a larger batch size (scaled with number of GPUs) and convolutional layers. It runs almost three times faster on a four-GPU setting in comparison to the single-GPU setting.</p><p><strong>The code</strong></p><p>You can find the full example from this <a href="https://github.com/lambdal/wiki/blob/master/deeplearning/benchmarks/keras_test.py">link</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f439fe6dd6ec" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transfer Learning’s Best Practice for image classification]]></title>
            <link>https://medium.com/@c_61011/transfer-learnings-best-practice-for-image-classification-112b2bbbf818?source=rss-9f394e8bf587------2</link>
            <guid isPermaLink="false">https://medium.com/p/112b2bbbf818</guid>
            <dc:creator><![CDATA[Chuan Li]]></dc:creator>
            <pubDate>Fri, 20 Jul 2018 02:21:48 GMT</pubDate>
            <atom:updated>2018-07-24T00:01:47.429Z</atom:updated>
            <content:encoded><![CDATA[<ul><li><a href="https://arxiv.org/abs/1805.08974">The Latest empirical study</a> suggested the best practice of transfer learning for image classification is to <strong>prepare a few dozen training images for each class and use fine-tuning</strong>.</li><li>ImageNet accuracy is a good predictor of fine-tuning performance. NASNat-A-Large gives the best ImageNet performance, so use it for fine-tuning.</li></ul><h3>Prerequisites</h3><ul><li>Understand the problem of <a href="http://cs231n.github.io/classification/">image classification</a> and the <a href="http://cs231n.github.io/convolutional-networks/#overview">deep learning</a> way of addressing it.</li></ul><h3>Table of Content</h3><ul><li><a href="https://medium.com/p/112b2bbbf818/#0c4f">Transfer learning: the problem</a></li><li><a href="https://medium.com/p/112b2bbbf818/#93da">Fixed features, fine-tuning and trained from scratch</a></li><li><a href="https://medium.com/p/112b2bbbf818/#7bad">ResNets: the best fixed feature extractor?</a></li><li><a href="https://medium.com/p/112b2bbbf818/#03b1">ImageNet: the oracle of fine-tuning</a></li><li><a href="https://medium.com/p/112b2bbbf818/#0272">Don’t train from scratch</a></li></ul><p><strong>Transfer learning: the problem</strong></p><p>In the beginning there was a long winter for A.I.</p><p>Then came the ImageNet. It led to the invention of AlexNet, VGGNet, GoogLeNet and eventually A.I.’s <a href="http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/">super-human performance</a> in image classification. This made people believe that a vast amount of training data is the key to powerful machine learning models. However, even with the explosion of digital content on the internet, very few has the resource to sourcing data at ImageNet’s scale.</p><p>A more practically important question is, whether or not there is a path to get strong machine learning models without spending weeks on collecting, labeling, and cleaning a huge amount of data.</p><p>One promising path seems to be using transfer learning. The idea is simple: adapting the model trained with a big dataset (for example, ImageNet) to your own problem. The hypothesis for this to work is:</p><p><em>Models that perform better on ImageNet are likely to perform better on other vision tasks.</em></p><p>Or is this true?</p><p><a href="https://arxiv.org/abs/1805.08974">A recent study</a> from Google suggested the answer is “yes” — for most of the case. This blog is a summary of the study, which shed light on today’s best practice of transfer learning for image classification.</p><p><strong>Fixed features, fine-tuning and trained from scratch</strong></p><p>There are two types of transfer learning: use fixed features and fine-tuning. Both involve restoring weights from a pre-trained ImageNet model, and re-training the network for the new classes of interests. The difference is, the fixed features approach freezes early layers and only trains the last layer. In contrast, the fine-tuning approach trains all layers. Intuitively, the fixed features approach is less prone to over-fitting while the fine-tuning approach is better at handling new classes that are irrelevant to ImageNet.</p><p>The Google researchers set themselves the goal of systematically studying the pros and cons of these two approaches, together with the case that training the network from scratch (no transfer learning).</p><p><a href="https://medium.com/p/112b2bbbf818#6dc6">Figure 1</a> summarizes their findings: First, <strong>Dataset similar to ImageNet benefit more from transfer learning than dataset un-similar to ImageNet.</strong> The first column of <a href="https://medium.com/p/112b2bbbf818#6dc6">Figure 1</a> shows an example: the t-SNE embeddings of the penultimate layer of the same Inception-V4 network have very different characters on these two datasets: the embedding of FGVC aircraft dataset has no structure while the embedding of CIFAR-10 (similar to ImageNet) has clear separation between different classes.</p><p>Second, <strong>both fine-tuning and trained from scratch led to significantly better features </strong>(column 2&amp;3, <a href="https://medium.com/p/112b2bbbf818#6dc6">Figure 1</a>, where the classes are clearly separated). This behavior also generalizes to many datasets. So the message is do not fix the early layers in transfer learning. However there is one caveat: Using fixed features can be better if there is very little training data. We will come back to this in details later.</p><p>Third, although training from scratch achieved parity with fine-tune, it is at the cost of significantly more training data and longer training time. So <strong>one should prefer fine-tuning over trained from scratch</strong>. Again there is an exception: training from scratch can give better accuracy if the new dataset has little similarity with ImageNet. However this does not happen very often considering the scale of ImageNet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/1*NZKiVmKvTnaAVOKMLCnNMQ.png" /><figcaption>Figure 1. 2D embedding (t-SNE) of features from the penultimate layer of Inception for different approaches. Image from <a href="https://arxiv.org/abs/1805.08974">https://arxiv.org/abs/1805.08974</a></figcaption></figure><p>In summary, my takeaway of their study is:</p><p><strong><em>Always prepare yourself at least a few dozen of training images for each class and use fine-tune.</em></strong></p><p><strong>ResNets: the best fixed feature extractor?</strong></p><p>If for whatever reason, you can’t get a few dozen examples for each class. Then fixed features is a better choice. Why? Because it is safer to make decisions based on the past experience if you don’t have enough observation in a new situation.</p><p>An interesting discovery of the study is that <a href="https://arxiv.org/abs/1512.03385">ResNets</a> seems to be the best fixed feature extractor. However this is completely empirical based and there is no theory behind it. Moreover, ImageNet accuracy does not correlate well with the transfer accuracy in the case of fixed feature extractor: the <a href="https://en.wikipedia.org/wiki/Pearson_correlation_coefficient">Pearson correlation coefficient</a> between the two is only 0.24. This meaning in practice there is no way to choose the best fixed feature extractor based on their ImageNet performance.</p><p>My own experience is that Inception-V4 and NASNet-A-Large are also very good fixed feature extractors and they <a href="https://github.com/lambdal/lambdarun/tree/master/tutorials/image_classification">sometimes perform better</a> than ResNets. In conclusion, if you want to use the fixed feature approach, at least try ResNets, Inception-V4 and NASNet-A-Large.</p><p><strong>ImageNet: the oracle of fine-tuning</strong></p><p>But there is a world where ImageNet accuracy correlate well with transfer accuracy — fine-tuning. The <a href="https://en.wikipedia.org/wiki/Pearson_correlation_coefficient">Pearson correlation coefficient</a> between the two is 0.86. Remember it was 0.24 in the fixed feature case.</p><p>This gives us an other reason to use fine-tune: Use it as the an oracle of fine-tuning. <strong>One can expect the best fine-tuning performance to come from the model that has the best ImageNet performance</strong>.</p><p>BTW, the best model on ImageNet is NASNet-A-Large. It has won 9 of the 12 transfer tasks in the study (<a href="https://medium.com/p/112b2bbbf818#df6e">Figure 2</a>). So <strong>choose NASNet-A-Large for fine-tuning</strong>!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/1*Vr3JdBZAz52c11PbV-K6Rw.png" /><figcaption>Figure 2. Twelve transfer tasks and the best network for each of them. Image from <a href="https://arxiv.org/abs/1805.08974">https://arxiv.org/abs/1805.08974</a></figcaption></figure><p><strong>Don’t train from scratch</strong></p><p>There is really no reason to prefer train from scratch over fine-tuning. Just consider the difference between the two: how weights are initialized. Fine-tuning uses weights from a pre-trained model and trained from scratch uses random weights. As long as there is similarity between the original task and the transferred task, weights from a pre-trained model should do better than random numbers.</p><p>In consequence, the direct benefit of fine-tuning is <strong>accelerated training</strong>. The study compared the number of epochs required to reach 90% of the maximum accuracy, and found an on average 17-fold speedup in fine-tuning (<a href="https://medium.com/p/112b2bbbf818#ad3e">Figure 3</a>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*bQgM9RefaOEPJBDZBeHD-g.png" /><figcaption>Figure 3. Fine-tuning (orange) needs a lot fewer epochs to reach high accuracy than training from scratch (blue). Image from <a href="https://arxiv.org/abs/1805.08974">https://arxiv.org/abs/1805.08974</a></figcaption></figure><p>Another reason to not train from scratch is it has poor performance when presented with insufficient amount of training samples (<a href="https://medium.com/p/112b2bbbf818#56da">Figure 4</a>). This suggests transfer learning is less prone to over-fitting if it can leverage knowledge learnt from a large, relevant task.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/1*2hU-XEN-RwOYQ_ZYvNytgg.png" /><figcaption>Figure 4. Training from scratch (blue) needs a lot more data to catch up with the performance of fine-tuning (orange). Also notice that fixed features (green) has certain advantage when only extremely small amount of data is available (far left side of the plots). Image from <a href="https://arxiv.org/abs/1805.08974">https://arxiv.org/abs/1805.08974</a>.</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=112b2bbbf818" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transfer learning with Mask R-CNN]]></title>
            <link>https://medium.com/@c_61011/transfer-learning-with-mask-r-cnn-f50cbbea3d29?source=rss-9f394e8bf587------2</link>
            <guid isPermaLink="false">https://medium.com/p/f50cbbea3d29</guid>
            <category><![CDATA[life]]></category>
            <dc:creator><![CDATA[Chuan Li]]></dc:creator>
            <pubDate>Tue, 17 Jul 2018 17:28:19 GMT</pubDate>
            <atom:updated>2018-07-18T02:47:42.883Z</atom:updated>
            <content:encoded><![CDATA[<ul><li>We see unprecedented performance in today’s object detection algorithms.</li><li>It is extremely easy to leverage a pre-trained model to work with your own object detection problem.</li></ul><h3>Prerequisites</h3><ul><li>Basic understanding of <a href="http://cs231n.github.io/convolutional-networks/">convolution neural networks</a> .</li></ul><h3>Table of Contents</h3><ul><li><a href="https://medium.com/@c_61011/transfer-learning-with-mask-r-cnn-f50cbbea3d29#99fa">Object detection: the problem</a></li><li><a href="https://medium.com/@c_61011/transfer-learning-with-mask-r-cnn-f50cbbea3d29#9615">Single-stage v.s. Multi-stage</a></li><li><a href="https://medium.com/@c_61011/transfer-learning-with-mask-r-cnn-f50cbbea3d29#3191">Mask R-CNN in a nutshell</a></li><li><a href="https://medium.com/@c_61011/transfer-learning-with-mask-r-cnn-f50cbbea3d29#920c">Performance</a></li><li><a href="https://medium.com/@c_61011/transfer-learning-with-mask-r-cnn-f50cbbea3d29#4770">Nigiri v.s. Maki: transfer learning with Mask R-CNN</a></li><li><a href="https://medium.com/@c_61011/transfer-learning-with-mask-r-cnn-f50cbbea3d29#fea0">Code</a></li></ul><p><strong>Object detection: the problem</strong></p><p><strong><em>Jump to </em></strong><a href="https://medium.com/@c_61011/transfer-learning-with-mask-r-cnn-f50cbbea3d29#4770"><strong><em>Transfer learning</em></strong></a><strong><em> if that is what you are here for.</em></strong></p><p>Object detection is a two-fold task: figuring out “what” are in an image and “where” are they.</p><p>The “what” part is recognition: for each input image, outputs a probability distribution across a set of pre-defined class labels, such as dog, cat etc. (<a href="https://www.flickr.com/photos/a_peach/8631368705/#d0ae">Figure 1</a>, left). The “where” part is localization: knowing an image to be one of the pre-defined classes, predicts the location of the instance (<a href="https://www.flickr.com/photos/a_peach/8631368705/#d0ae">Figure 1</a>, right. In this case, predicting the blue bounding box associated with the dog.).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/658/1*G1L4Gsnyn-r1syRHTmcZiw.png" /><figcaption>Figure 1. Image recognition v.s. object detection. Input image credit to flikcr user a_peach</figcaption></figure><p>Separately, recognition and localization are difficult tasks; combined, they making object detection a very hard chicken-egg problem.</p><p>Of course, one should allow multiple classes and instances in a single image, which only makes it more complicated.</p><p><strong>Single-stage v.s. Multi- stage</strong></p><p>Fortunately, the renaissance of neural networks has made object detection unprecedentedly promising. To make things compact, this blog only discusses the cutting edge solutions (as in early 2018). To learn the progress of object detection since 2014, we refer you to this <a href="https://medium.com/comet-app/review-of-deep-learning-algorithms-for-object-detection-c1f3d437b852">review</a>.</p><p>There is one catch before diving into technical details: today’s object detection has branched out into two main streams: single-stage detection (SSD, YOLO) and multi-stage detection (the R-CNN family). The takeaway is:</p><p><em>A multi-stage detector first detects foreground objects, then recognizes each of them as specific classes (</em><a href="https://medium.com/p/f50cbbea3d29#5999"><em>Figure 2</em></a><em>, right). In comparison a single-stage detector does not bother the foreground object detection but instead uniformly samples foreground objects using a grid (</em><a href="https://medium.com/p/f50cbbea3d29#5999"><em>Figure 2</em></a><em>, left).</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*M7Mj0_-mgtkyd57eVbI_9g.png" /><figcaption>Figure 2. Grid-based proposal v.s. detection-based proposal. Single-stage detectors use the grid while multi-stage detectors use a dedicate detector (the first stage) to search for potential foreground objects.</figcaption></figure><p>Performance-wise, a single-stage detector is often faster because of, obviously, less computation by only look at fixed locations on the grid. On the other hand, a multi-stage detector is more accurate because it does not only look at fixed locations on the grid but also locations that are likely to have foreground objects in the second stage of the detection.</p><p><strong>Mask R-CNN in a nutshell</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/815/1*Uezma5ZkX8AaDbdT5vN8Lg.png" /><figcaption>Figure 3. Mask R-CNN. Image from <a href="https://arxiv.org/abs/1703.06870">https://arxiv.org/abs/1703.06870</a></figcaption></figure><p>Mask R-CNN is widely recognized as the state of the art multi-stage object detector (as in early 2018). As you can see from <a href="https://medium.com/p/f50cbbea3d29#b44d">Figure 3</a>, it recognizes the objects’s classes (“person” and “ball”), the locations (the bounding box) and the shapes (the colorful masks). And it does so for over 80 different classes to a fairly accuracy degree — an achievement that is completely unthinkable a few years back.</p><p>But in a nut-shell, it is still a smart regressor, or a couple of regressors that are carefully wired together. Let’s go through them one by one:</p><ul><li><strong>RoI</strong>: Region of Interest proposal is the first regressor. First, imagine a (fully convolutional) network that outputs a low resolution “foreground” map. Each pixel in this map represents the probability of existing an foreground object. Then, change the output dimension of this network from one (the foreground probability) to five, so it also predicts the bounding box for each region in the image as if it has a foreground object. That is our foreground object detection and the entire first stage.</li><li><strong>Alignment</strong>: this is actually not a regressor. We list it here anyway because it is a very important step of the pipeline. The goal is to fetch content inside of the bounding boxes so it can be processed (recognized) by the second stage of the detection. There are several catches here: first, we only consider bounding boxes with high foreground probabilities; second, each bounding box is represented by four floating-point numbers so it is not aligned with the pixelated image information, making it difficult to fetch the content; third, bounding boxes have different sizes but the later computation requires a fixed size input (for batch process and for the use of densely connected layers). The answers are: first, use non-maximum suppression to screen out bounding boxes that are unlikely to have any foreground object; second, use bi-linear interpolation to fetch the content so it aligns with the original image in sub-pixel resolution; third, the content, in the form of “neural patches”, are normalized to the same size.</li><li><strong>Prediction heads</strong>: This is a multi-task regressor that takes an neural patch and outputs three different predictions: class label, refined bounding box, and binary mask. The regressor is, no surprise, a neural network. It begins with a few layers that are shared by all different tasks, then a few layers that are dedicate to each individual task.</li></ul><p><strong>Performance</strong></p><p>The performance of Mask R-CNN is nothing but mind-blowing. There are some really cool qualitative demonstrations, like <a href="https://www.youtube.com/watch?v=OOT3UIXZztE">this</a> and <a href="https://www.youtube.com/watch?v=KYNDzlcQMWA">this</a>. Quantitatively, the method achieved the best detection accuracy with decent speed: Masked Average Precision of 35.7 on MSCOCO at 5 frames/second.</p><p>If you wonder what average precision is — it summarizes the shape of the precision/recall curve, and is defined as the mean precision at a set of equally spaced recall levels, like [0,0.1,…,1]. The “masked” way of computing average precision is related to how the detected object is matched against the ground truth object: masked AP matches them using the masks of the objects, while non-masked AP matches them using the bounding boxes of the objects.</p><p><strong>Nigiri v.s. Maki: transfer learning with Mask R-CNN</strong></p><p>The coolest thing about Mask R-CNN is that it can easily transfer into a bespoke solution for your own object detection problem.</p><p>Imagine an owner of a sushi restaurant who wants to improve the quality of the service. We can help him/she with a system that recognizes the sushi in each customer’s plate. Let’s say we start from something simple: trying to find all the Nigiri sushi and theMaki sushi in the plates.</p><p>Guess what, it is not too difficult to get decent results. All you need is fine tuning a pre-trained model with 70 annotated images. Let’s see how this is done:</p><p><strong>Data preparation</strong>: Download training and testing images (with annotations) from <a href="https://github.com/lambdal/lambdarun/blob/master/tutorials/object_detection">here</a>. BTW, we used VGG’s <a href="http://www.robots.ox.ac.uk/~vgg/software/via/">VIA</a> toolbox for annotation and it’s awesome:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nNkIklnF41tVzjOyoFayGw.png" /><figcaption>Figure 4. Annotating sushi with VGG’s via toolbox.</figcaption></figure><p><strong>Training</strong>: the idea is to reuse most part of the pre-trained network, and change its final output layer to detect sushi. Specifically, we need to make a few modifications to the original model:</p><ul><li>Modify the last layer in each prediction head so they cope with three output classes (Background, Nigiri and Maki) as opposed to 80 classes in MSCOCO.</li><li>Restore weights of all other (unmodified) layers from a pre-trained model.</li><li>Train the modified layers with the sushi images.</li></ul><p>Here are some results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/869/1*N79GorPUQFFWE7FEjySU2Q.png" /><figcaption>Figure 5. Sushi detection on testing images.</figcaption></figure><p>Not bad.</p><p><strong>Code</strong></p><p>You can find our opensource implementation <a href="https://github.com/lambdal/lambdarun/tree/master/tutorials/object_detection">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f50cbbea3d29" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>