<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kerem Turgutlu on Medium]]></title>
        <description><![CDATA[Stories by Kerem Turgutlu on Medium]]></description>
        <link>https://medium.com/@keremturgutlu?source=rss-25e7c0e46c50------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*sGvpMvcC26XrGmwt1pX0LA.jpeg</url>
            <title>Stories by Kerem Turgutlu on Medium</title>
            <link>https://medium.com/@keremturgutlu?source=rss-25e7c0e46c50------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 18:40:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@keremturgutlu/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Weakly Supervised Transfer Learning in Medical Imaging]]></title>
            <link>https://medium.com/@keremturgutlu/weakly-supervised-transfer-learning-in-medical-imaging-c89c5ca2d0be?source=rss-25e7c0e46c50------2</link>
            <guid isPermaLink="false">https://medium.com/p/c89c5ca2d0be</guid>
            <category><![CDATA[medical-imaging]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[transfer-learning]]></category>
            <category><![CDATA[weakly-supervised]]></category>
            <dc:creator><![CDATA[Kerem Turgutlu]]></dc:creator>
            <pubDate>Tue, 07 May 2019 05:34:11 GMT</pubDate>
            <atom:updated>2019-05-07T05:34:11.812Z</atom:updated>
            <content:encoded><![CDATA[<p>Also <strong>part 2</strong> of <a href="https://medium.com/@keremturgutlu/semantic-segmentation-u-net-part-1-d8d6f6005066"><strong>Semantic Segmentation — U-Net</strong></a></p><h3>Why Transfer Learning?</h3><p>It is now well known fact that bigger models combined with immense amounts of data prove to beat the previous state of the art at any given task. Unfortunately in some tasks or domains it’s not trivial to obtain a that amount of high quality data for training purposes. That’s why we still put faith in techniques which don’t rely on huge amounts of data but still generates competitive results.</p><p>First technique that comes to mind in such setting is transfer learning. Transfer learning is the general idea of using pretrained models to finetune on a downstream target task. In general Imagenet pretrained models for computer vision and language models for NLP are the most popular and widely used choices.</p><p>In my experience transfer learning is almost always helpful when there is not enough data for a given task. The usefulness of transfer learning is correlated with the learned features within a pretrained network and transferability of these features to the target downstream task. This also explains why Imagenet pretrained models are more helpful with natural images as compared to medical images.</p><p>One should know that it is not an immutable truth that all kinds of transfer learning would help at any circumstances. There are papers stating that transfer learning is not helpful for some tasks in their experiments or there are some arguments that with enough data transfer learning is not necessary. All of these claims might be true but I believe that every task-data-sample size trio is a unique case and we should conduct ablation studies without making any prior assumptions. Still, what I am sure though is that transfer learning can’t hurt any model’s performance. After all, the day worst case scenario would be as same as randomly initializing the model weights.</p><p>Lately, I conducted some experiments for my work investigating the effects of task dependent and task independent transfer learning for semantic segmentation. What I mean by task independent transfer learning is using a pretrained model that was trained on a different task rather than semantic segmentation, such as Imagenet classification. Contrarily, when I refer to task dependent transfer learning I mean using a pretrained model which was trained on the same task as the downstream task, for example COCO semantic segmentation task.</p><p>Below, blue curve represents Imagenet pretrained Dynamic Unet model, green curve represents randomly initialized Dynamic Unet model trained from scratch and orange curve represents a Fully Convolutional Network with Inception blocks, which is called DeepCut in the graph, again trained from scratch. All of the models are trained with varying sample sizes to see the effect of training sample size. We can see that Imagenet pretrained model (blue curve) outperforms at all sample sizes and gives very close results with only 100 samples compared to the case of training on 10x full data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XTzrp0T_X48cUkaO842dFw.png" /><figcaption><strong>No Pretraining</strong></figcaption></figure><p>Here, we see the same exact models but this time all of the models are pretrained on COCO segmentation data and later finetuned. Even though Imagenet-&gt;COCO-&gt;Downstream model outperforms at all sample sizes other models also perform pretty good with just 100 samples thanks to transfer learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ovCwyeFAAuJqIpetZa_-LQ.png" /><figcaption><strong>Pretrained on COCO</strong></figcaption></figure><p>These experiments made me realized that models benefit from transfer learning as long as learned features are meaningful for the downstream task even though pertaining task is different than the downstream task. For example, we know that image classification is a classification task that outputs class label probabilities but in fact it inherently learns object localizations as well. Class activation maps are an excellent way to prove this point. These features for example provide powerful priors for the fine-tuning process for localization task such as semantic segmentation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/0*SP98W1221wFIWAVu.jpg" /></figure><h3>Why Weakly Supervised Learning?</h3><p>Now that we know transfer learning is great at all circumstances, it’s time to give the bad news: <strong>unfortunately you won’t have a pretrained model for every deep learning task you would like to tackle</strong>. In my opinion there are two main reason why there might not be a pretrained model available:</p><ol><li>Your task might require a new/custom/unique architecture which is not yet pretrained on a large dataset or</li><li>It might be very expensive to collect a dataset for pretraining purposes</li></ol><p>For my project in collaboration with UCSF, we were lucky enough to have both of these problems :)</p><p>The goal of the project was to segment brain for skull stripping and ventricles for pre and post surgery volume calculations. You may search google for brain ventricles. Due to ethical reasons I won’t be posting any sample images even though all patients were anonymized. Volumetric changes in ventricles are important since it’s an indication of whether the patient should be released or not after a surgery. Decision process is a very manual task which requires careful assessment by looking at both the scans (before and after surgery) side by side. Another option is to use a commercial software such as <a href="https://www.mimsoftware.com/">MIM</a> and manually segment the ventricles to later compare by calculating both the volumes. Manual segmentation for just a single patient takes couple of hours even at the hands of the experts and depends on how much of an accuracy you care.</p><p>So, what we wanted to accomplish by this project was to come up with a deep learning model which would automatically segment the ventricles. Throughout the experiments both 2D and 3D CNN models were used and 3D models give superior results as they were also incorporating contextual information from multiple slides simultaneously. <strong>This unique architectural choice brings us to the first problem of not having a pretrained model</strong>. As a solution to this we might have trained our own pretrained models but <strong>in the field of medicine it’s impossible or too costly to obtain a dataset at Imagenet scale and this brings us to the second problem.</strong></p><p>Since transfer learning was not an option a lot of time was spent on acquiring data for couple of months. Without using transfer learning I’ve conducted experiments with different variants of 3D CNN models. As promised in a preceding <a href="https://medium.com/@keremturgutlu/semantic-segmentation-u-net-part-1-d8d6f6005066">blog post</a> I will explain each model in detail.</p><h4>3D Unet [<a href="https://arxiv.org/abs/1606.06650">paper</a>]</h4><p>In the preceding blog post I’ve explained Unet and Dynamic Unet architectures in depth. Here, 3D Unet is the same exact model but instead of using 2D operations such as 2D convolutions, 2D pooling, 2D dropout we use their 3D counterparts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qpMLBvBgx4mtxKzzdJ0xSw.png" /><figcaption>3D Unet</figcaption></figure><h4><strong>MeshNet [</strong><a href="https://arxiv.org/pdf/1612.00940.pdf"><strong>paper</strong></a><strong>]</strong></h4><p>MeshNet is an architecture that I came across to after searching for “brain segmentation deep learning models”. Sometimes you go with the most trivial way of research and see what other’s have done in a similar context :) It is a fully convolutional model which uses dilated convolutions to reduce number of parameters while maintaining same effective receptive fields.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XhlJ0Aonl-2wLg6qyaw7fw.png" /><figcaption>MeshNet</figcaption></figure><h4>3D ResUnet [<a href="https://arxiv.org/pdf/1802.10508.pdf">paper</a>]</h4><p>This model is based on 3rd place solution from <a href="https://www.med.upenn.edu/sbia/brats2018/data.html">BRATS Brain segmentation challenge</a>. It encapsulates ideas both from Unet and PreAct ResNet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mD5A_S9oAesJ-UK-Rz0rUw.png" /><figcaption>3D ResUnet</figcaption></figure><p>I am also providing <a href="https://gist.github.com/KeremTurgutlu/72dd8d5a5bf6e6272e499d854c479232">code</a> for all the above models implemented by myself in PyTorch using <a href="https://github.com/fastai/fastai">fast.ai</a>. In general I’ve experimented with different combinations of normalizations, activations, width and depth within these architectures.</p><h4>Experiments</h4><p>In our training sets we had <strong>human annotated MRI scans of 112 patients and CT scans of 107 patients</strong>. For each MRI and CT dataset <strong>1 validation and 2 test sets were also constructed.</strong> <strong>Sample sizes for these test sets were [15, 20] for MRI and [9,18] for CT</strong> respectively. <strong>Test 2 has more challenging and unique cases with more abnormalities, as well as pre and post surgery scans.</strong> Having different test sets in terms of difficulty is beneficial for understanding the generalization of our models in terms of rare edge cases.</p><p>At the beginning of the study we didn’t know how much data would be sufficient and we didn’t start any experiments before the human annotation was complete. So we didn’t have the chance to say; <em>“This is enough data now, thank you :)” </em>to the experts. What we only had as a dataset was what I’ve explained in the previous paragraph.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/1*fHlryTgj0E8p7bNSN-pIjQ.png" /></figure><p>Now, here comes the cool part. We realized that we can use <strong>machine generated weak labels for pretraining and human annotated strong labels for finetuning.</strong> I learned that it’s possible to label couple of images and then ask the commercial software to create similar masks for a bulk of MRI scans. Software probably uses some sort of heuristic algorithm (or maybe knn based??) in order to come up with these masks because they are not as good as the human annotator and the quality is visibly worse. After all, if the commercial software was good enough we wouldn’t be working on this project to start with. As a result we’ve gathered a 10x larger dataset of machine generated weak labels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*o_w_wwGEC_wDPDF1IP98tg.png" /></figure><p>These kind of imperfect labels are called<strong> “weak labels” </strong>because you are not 100% sure if it’s true and the process of training with weak labels is called <strong>“weakly supervised training” </strong>(a nice <a href="https://code.fb.com/ml-applications/advancing-state-of-the-art-image-recognition-with-deep-learning-on-hashtags/">post</a> from FAIR), and I will call our method <strong>“weakly supervised transfer learning” </strong>— we are not only training a model with coarse labels but also finetuning with more accurate ones.</p><p>I’ve created 11 baseline models with combinations of different architectures, normalization layers, activation functions, dropout rates and so on, all can be accessed from <a href="https://gist.github.com/KeremTurgutlu/72dd8d5a5bf6e6272e499d854c479232">this gist</a>. For each dataset I’ve trained pretrained networks using the weak labels. Later I used these pretrained networks without needing to change the architecture (since task remained same) and finetuned them on our strong human annotated labels. We only had weak labels for MRI modality for both brain(skull stripping) and ventricle labels, so same MRI pretrained models were used for CT finetuning as well.</p><p>Below are the results for brain and ventricle segmentation tasks.</p><p><strong>NOTL — represents no transfer learning</strong> <strong>and training with human annotated data directly from scratch.</strong></p><p><strong>WEAK — represents only training with weak labels from scratch.</strong></p><p><strong>TL — represents weakly supervised transfer learning from weak MRI labels to human annotated modality labels.</strong></p><p>Numbers are from the best models on test1 and test2 independently.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jSz9ihlJrc-P6XPzvz_7ZA.png" /></figure><p>Even though, at first glance applying transfer learning seems a bit beneficial for brain segmentation and seems not harm ventricle segmentation task I couldn’t find any statistical significance that proves weakly supervised transfer learning to be working with this setting.</p><p>Then I started thinking deeply and asking questions to myself:</p><ul><li><strong>Maybe number of samples we gathered from human annotators is already enough?</strong></li><li><strong>Maybe the given tasks are too easy to be solved with such small labeled data?</strong></li><li><strong>Maybe transfer learning from MR to CT modalities is not that of a good idea even though we normalized them with a similar fashion?</strong></li><li><strong>Maybe differences in modality distributions is not something learnable by transfer learning?</strong></li></ul><p>As a result we beat the commercial software in all test cases and human annotators in some test cases for our primary research.</p><p>But I was still curious to see how this weakly supervised transfer learning might help. So I conducted more experiments and varied sample sizes, and here are the final results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KzPYIP45RgqSbsAR4eTyZg.png" /></figure><p>Varying sample sizes show that for MRI -&gt; MRI; weakly supervised transfer learning helps substantially while MRI -&gt; CT results are not still very clear to be helpful. <strong>With only 10 samples we are performing very close to training with a 10x larger dataset which has test1 and test2 dice scores of 0.8575 and 0.7939.</strong></p><p>In my opinion, main takeaway is that each time we are dealing with a data-task-sample size trio we should approach the problem at hand differently and always think about what we can do differently before going towards the most obvious and expensive solution, such as collecting more labelled data.</p><h3>Future Work and Goals</h3><p>Even though weakly supervised transfer learning is very cool and shows it’s power when training data is very limited it’s still always not an option. Because, still accessing or generating weak labels might not be possible for some of the tasks. For example, I don’t think it would be possible to come up with coarse machine annotations for diverse brain tumors, chest X-ray readings, etc. Which means that we would need more general purpose encoders such as Imagenet pretrained models but for medical imaging. I believe it’s now time that we have a shared open source library with pretrained medical models for as many as modalities, body parts and even tasks.</p><p>I just registered to <a href="https://www.ukbiobank.ac.uk/">UK BioBank</a> with the goal in mind to make this dream come true. I hope that my application will be approved and we can have a leap in medical deep learning. UK BioBank is a great organization with an amazing goal in mind: to support bona fide researchers for public good. I highly recommend you to visit their page to witness the amazing work they got there.</p><p>I hope you enjoyed reading this, let’s keep experimenting to explore other cool stuff together ;)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c89c5ca2d0be" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding building blocks of  ULMFIT]]></title>
            <link>https://blog.mlreview.com/understanding-building-blocks-of-ulmfit-818d3775325b?source=rss-25e7c0e46c50------2</link>
            <guid isPermaLink="false">https://medium.com/p/818d3775325b</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Kerem Turgutlu]]></dc:creator>
            <pubDate>Sun, 10 Feb 2019 06:43:31 GMT</pubDate>
            <atom:updated>2019-02-11T21:41:05.191Z</atom:updated>
            <content:encoded><![CDATA[<h3>Understanding building blocks of ULMFIT</h3><p>Last week I had the time to tackle a Kaggle NLP competition: <a href="https://www.kaggle.com/c/quora-insincere-questions-classification">Quora Insincere Questions Classification</a>. As it’s easy to understand from the name, the task is to identify sincere and insincere questions given the question text. In short it’s a binary classification problem.</p><p>I recently completed Fast.ai Part 1 (2019). Library evolved drastically over the last 1 year. It is lot easier and faster to use and it has an amazing documentation. Since it has such a great documentation I rarely have a problem that requires me to pop up a question in the forums, though forums are still great to learn things beyond the library itself. If you want to apply for Part 2 (2019) as a diversity fellow or sponsor here is the <a href="https://www.fast.ai/">link</a>.</p><p>If you don’t know it already, NLP had a huge hype of transfer learning in this past 1 year, starting with <a href="https://arxiv.org/abs/1801.06146">ULMFit</a>, <a href="https://allennlp.org/elmo">ELMo</a>, <a href="https://arxiv.org/abs/1806.05662">GLoMo</a>, <a href="https://blog.openai.com/language-unsupervised/">OpenAI transformer</a>, <a href="https://arxiv.org/pdf/1810.04805.pdf">BERT</a> and recently <a href="https://ai.googleblog.com/2019/01/transformer-xl-unleashing-potential-of.html">Transformer-XL</a> for further improving language modeling capabilities of the current state of the art. I am not going to be able to explain all of these exceptional works except for ULMFIT. For a general overview for all the above models and more I encourage you to read this amazing <a href="https://lilianweng.github.io/lil-log/2019/01/31/generalized-language-models.html">blog post</a> in order to get a deeper understanding in language modeling methodologies and advancements.</p><p>First, my aim is not to explain how or why ULMFIT became to be a state-of-the-art model because there are many resources like this <a href="http://nlp.fast.ai/classification/2018/05/15/introducting-ulmfit.html">fast.ai blog post</a>, <a href="https://course.fast.ai/">part 1 (2019) course videos</a>, and many other blog posts floating around the internet. Rather, I will be digging deeper into references of ULMFIT and references of it’s references to get a better understanding of the fundamental building blocks that it adapts.</p><p>This might be little overwhelming for demystifying the underlings of this study but sometimes being little obsessed helps. I usually end up opening 87634172 tabs in my browser, reading the source code and spending quite some time bouncing back and forth between all these resources. I did the same when I got my hands dirty with ULMFIT for the Kaggle competition I’ve introduced in the beginning of this post. Since I’ve already spent quite some time digging deeper, I thought it would be cool to share what I distilled from it.</p><p>High level idea of ULMFIT is to train a language model using a very large corpus like <a href="https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/">Wikitext-103</a> (103M tokens), then to take this pretrained model’s encoder and combine it with a custom head model, e.g. for classification, and to do the good old fine tuning using discriminative learning rates in multiple stages carefully.</p><p>You should already be familiar with all of these, after all the aim here is to dig a little deeper.</p><pre>if not: print(&quot;shame :)&quot;); open(<a href="http://nlp.fast.ai/classification/2018/05/15/introducting-ulmfit.html">this</a>).read() or open(<a href="https://arxiv.org/pdf/1801.06146.pdf">this</a>).read()<br>else: continue</pre><h3>AWD-LSTM</h3><p>Here is an overview of our encoder + classifier model.</p><pre>SequentialRNN(<br>  (0): MultiBatchEncoder(<br>    (module): AWD_LSTM(<br>      (encoder): Embedding(60003, 300, padding_idx=1)<br>      (encoder_dp): EmbeddingDropout(<br>        (emb): Embedding(60003, 300, padding_idx=1)<br>      )<br>      (rnns): ModuleList(<br>        (0): WeightDropout(<br>          (module): LSTM(300, 1150, batch_first=True)<br>        )<br>        (1): WeightDropout(<br>          (module): LSTM(1150, 1150, batch_first=True)<br>        )<br>        (2): WeightDropout(<br>          (module): LSTM(1150, 300, batch_first=True)<br>        )<br>      )<br>      (input_dp): RNNDropout()<br>      (hidden_dps): ModuleList(<br>        (0): RNNDropout()<br>        (1): RNNDropout()<br>        (2): RNNDropout()<br>      )<br>    )<br>  )<br>  (1): PoolingLinearClassifier(<br>    (layers): Sequential(<br>      (0): BatchNorm1d(900, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>      (1): Dropout(p=0.4)<br>      (2): Linear(in_features=900, out_features=50, bias=True)<br>      (3): ReLU(inplace)<br>      (4): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>      (5): Dropout(p=0.1)<br>      (6): Linear(in_features=50, out_features=2, bias=True)<br>    )<br>  )<br>)</pre><p>Above is the layer by layer exposure of ULMFIT in fast.ai library. Model is composed of pretrained encoder (<strong>MultiBatchRNNCore — module 0</strong>) and a custom head, e.g. for binary classification task in this case (<strong>PoolingLinearClassifier — module 1</strong>).</p><p>Architecture that ULMFIT uses for it’s language modeling task is an <a href="https://arxiv.org/pdf/1708.02182.pdf">AWD-LSTM</a>. The name is an abbreviation of ASGD Weight-Dropped LSTM.</p><p>Here is a hint about maybe the most important differentiator of this model in form of a meme:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/0*pqOAsU3_UZd0Aruz.jpg" /></figure><p>AWD-LSTM literally has dropout at all the possible layers as long as it makes sense. Fast.ai <a href="https://docs.fast.ai/text.models.html">docs</a> precisely explains all of them (I also encourage you to check out the documentation):</p><p>If you go back and look at the ULMFIT architecture exposed earlier, you may one by one link the modules listed below. To simplify everything I will assume a vocabulary of <strong>V = [“I”, “went”, “to”, “trouble”, “of”, “exceling”, “for”, “you”, “guys”, “:)” ]</strong>. Don’t be surprised to see the <strong>smiley — :)</strong> as another token, any information you can represent is valuable. Tokenization is another important aspect in NLP which will not be covered here but for those who are particularly interested may visit <a href="https://docs.fast.ai/text.transform.html">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/326/1*ZwQ8wwewP6pfdfVE5uJbZw.png" /><figcaption>Embedding Lookup Table with n dimensions</figcaption></figure><ol><li><strong>Encoder Dropout</strong> (<em>EmbeddingDropout()</em>)<strong>: </strong>Zeroing embedding vectors randomly.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*-8dQdM7059IAt048k_7V5Q.png" /><figcaption><strong>encoder_dp</strong> module</figcaption></figure><p><strong>2.Input Dropout</strong> (<em>RNNDropout())</em><strong>: </strong>Zeroing embedding lookup outputs randomly.</p><p><strong>N.B. </strong>Here, the order of inputs fed into LSTM is <strong><em>batch size x sequence length x embedding size </em></strong>at each forward pass.</p><p>Dropout is one of the de-facto ways for regularizing deep learning models and preventing complex co-adaptations of hidden units (weights), a.k.a overfitting.[1,2]</p><p>AWD-LSTM paper emphasizes that until contribution applying naive-classical dropout techniques on a LSTM for language modeling tasks were ineffective as it disrupts the ability to retain long term dependencies or the effective dropout methods that might work would require inefficient re-implementations of CUDNN LSTM. Retaining long term dependencies is a very key point, after all the main differentiator of an LSTM over an RNN in practice is the capability of carrying out longer term dependencies by learning how much information to flow through the sequence.[3]</p><p>Let’s try to imagine what happens if we use a naive dropout layer on input sequences fed to LSTM and further convince ourselves why might it not be effective in practice.</p><p>If you go to PyTorch documents the plain dropout layer is defined as:</p><blockquote>During training, randomly zeroes some of the elements of the input tensor with probability <em>p</em> using samples from a Bernoulli distribution. Each channel will be zeroed out independently on every forward call.</blockquote><p>What we actually want to achieve is to zero elements in a way that same parts of hidden units are disabled throughout the sequence. After all at each forward pass by applying dropout we introduce a partially masked form of our model — <strong><em>f(x), </em></strong>and masked weights should be consistent for each and every input’s forward pass.</p><pre># shape : (1, 3, 5) - (bs, seq_len, emb_len)<br># taking a single sentence like &quot;I love dogs&quot; as an example</pre><pre># case 1: naive dropout <br>tensor([[ 1.7339,  0.2212, -2.5182, -0.0000,  2.2808],<br>        [-0.6162,  0.4100, -0.6159, -1.3283,  1.8158],<br>        [-2.5599, -1.5934, -2.1049,  0.8675, -0.6511]])</pre><pre># case 2: special RNN dropout<br>tensor([[ 1.7339,  0.0000, -0.0000, -1.7596,  2.2808],<br>        [-0.6162,  0.0000, -0.0000, -1.3283,  1.8158],<br>        [-2.5599, -0.0000, -0.0000,  0.8675, -0.6511]])</pre><p>As you can see above, the naive method will not ensure to disable the same weights of the RNN layer, in this case it’s <strong>input-to-hidden</strong>, but in contrary <em>RNNDropout()</em> will give the consistency throughout the whole sequence. In computer vision it’s trivial to randomly mask inputs as the forward pass will be stateless but in the case of recurrent networks we should carry out the exact same masked weights. I hope that now it’s a bit more clear why retaining longer term dependencies with naive method might be hard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2abJ9OoaUamQTV3cNwuT9A.png" /><figcaption>Rather than memorizing the functions let’s paste this image here just in case :)</figcaption></figure><p><strong>3. Weight Dropout(</strong><em>WeightDropout()</em><strong>) </strong>: Apply dropout to LSTM weights.</p><p>Previously in <strong>Input Dropout </strong>we’ve seen how dropout is applied to each input sequence independently but consistently throughout the sequence. In <strong>Weight Dropout </strong>dropout is applied to <strong>hidden-to-hidden </strong>matrix by zeroing out hidden units randomly. Weight masks are randomly refreshed at each batch-forward pass.</p><pre># PyTorch LSTM weights :(rnn.weight_ih_l0, rnn.weight_hh_l0)<br>rnn = nn.LSTM(5,10,1, bidirectional=False, batch_first=True)</pre><p>Weight Dropout is a generic dropout layer, which wraps a PyTorch module and zeroes the weight randomly given the name of the layer. So you can theoretically use it for any custom layer given the named attributes. Here you can notice that <strong>weight_ih_l0 </strong>is not registered for weight dropout.</p><p>My logical assumption would be, since input dropout is already present, adding another layer of dropout for <strong>input-to-hidden </strong>weight<strong> </strong>would be unnecessary and would also make it harder to control the level of dropout we desire over the <strong>input-to-hidden</strong> transformation. Also the ability to add randomness to every sample is much powerful than doing it only every batch-forward pass.</p><pre>WeightDropout(rnn, weight_p=0.5, layer_names=[&#39;weight_hh_l0&#39;])</pre><p>4. <strong>Hidden Dropout </strong>(<em>RNNDropout()</em>): Zeroing outputs of LSTM layers. This dropout is applied except for the last LSTM layer output.</p><p>This dropout layer randomly zeroes out raw outputs from previous LSTM layers in a stacked LSTM module. It can be thought as zeroing out the inputs of the stacked LSTM layers, e.g. in a 3 layer LSTM, the inputs to LSTM-2 and LSTM-3 would be randomly zeroed.</p><p>As it’s done in <strong>Input Dropout </strong>here in order to be consistent throughout the sequence the same parts are zeroed as before feeding to hidden features to the next LSTM layer. Same logic we’ve discussed in original sequence input applies to inputs of the upper LSTM layers.</p><p>5. <strong>Output Dropout </strong>(<em>RNNDropout()</em>): Zeroing final sequence outputs from encoder before feeding it to decoder.</p><p>This module is not visible when we look at the full ULMFIT classification model since this dropout is used before generating next word predictions by the language model decoder. Language model decoder is nothing but a fully connected linear layer which transforms outputs of encoder to token predictions in our vocabulary. As we are dealing with a sequential input to the decoder linear layer keeping sequential dropout consistency is important.</p><p><strong>Main Takeaway:</strong></p><ul><li>Ensure to apply dropout throughout a single input sequence in a consistent way and apply it anywhere possible. Apply random dropout per input sequence.</li><li>Also apply dropout on hidden-to-hidden weight of LSTM. Apply random dropout per batch-forward pass.</li></ul><p><strong>Wrapping Everything Up</strong></p><p>Since all the dropout layers are explained we can put it all together and visualize how each and every one of these take place.</p><p>Let’s define a dummy batch with the following sequences :</p><p><strong>batch = <em>[“I love cats”, “I love dogs”]</em></strong></p><ul><li>First step is to initialize an embedding matrix for each token and apply the encoder dropout</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*9FZw9_ulQjCYF8GAn1CuUA.png" /></figure><ul><li>Then we will do a look up for our batch and apply the input dropout</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*PMjSkgyK403dR4YpJdAVFQ.png" /></figure><ul><li>Next step is to allow random dropout at hidden-to-hidden weights for each LSTM layer. Here I am defining a <strong>2 layer LSTM with 5 hidden units</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dO-rgbMROYsBK45W_6TfBA.png" /></figure><p>You may notice how same tokens, like “love”, have different input features after input dropout. Here <strong>Whh </strong>is<strong> hidden-to-hidden</strong> weight and it’s shared throughout the full sequence forward pass.</p><ul><li>Lastly we take a look at hidden dropout which is the input for the upper layer LSTMs in the stack</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TyxaTcQvkAqwgtBr5tcz9Q.png" /></figure><p>If we want to look at output dropout, you will notice it’s nothing but the same type of dropout applied to final raw outputs of the encoder stacked LSTM which noted as decoder_inp {1,2,3}.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u_vv4FqBZH6DrCGTy4juHA.png" /></figure><p><strong>Variable Length BPTT</strong></p><p>Adding dropout to everywhere isn’t the only regularization adapted by ULMFIT from AWD-LSTM and there is much more to see from these bag-of-tricks. Let’s take a step back from classification and explore language modeling part since it’s the first part of the whole fine-tuning process. Even though ULMFIT already offers a pretrained language model on English Wikitext-103, we still need to fine tune it for our custom dataset in order to better adapt to the linguistic properties of our corpus. You would agree that language used in movie reviews would differ a lot from the language that legal documents have even though they are both in English.</p><p>In the original AWD-LSTM paper authors emphasize on how dataset usage in traditional language modeling is inefficient. The reason being having a fixed window to back-propagate will always have the same words contributing for the update with same weight of gradients flowing from last word to the first. Randomizing windows selected at each step is what is meant by variational length BPTT. This randomization acts as a regularization method and allows a theoretical 100% utilization of the dataset.</p><p>The way this phenomena implemented in fast.ai or at least using dataset efficiently is as following:</p><p>All the text is concatenated as a one big array, you may think it as a stream of text. Then it’s chunked into pieces of batch size sequence and each sequence is BPTT long. A special callback class called LanguageModelPreLoader takes care of all the utilization and the optimization. It shuffles the index and adds randomness to the starting index of the sequence selected which allows a theoretical usage of all the tokens to start with.</p><p><strong>MultiBatchEncoder</strong></p><p>We’ve seen how dataset utilization is allowed in language modeling, now let’s take a look at another smart way of processing a full sentence for classification task. <em>MultiBatchEncoder() </em>is a wrapper over the encoder which allows processing a full sentence but at the same time respecting the BPTT that we’ve specified. Again we can visualize this using spreadsheets.</p><p>We will have the following batch for simplicity:</p><p><strong>batch = <em>[“I love cats very much”, “I love dogs very much”], bptt = 3, output_dim=2</em></strong></p><p>What we will do is to slide over this batch along the sequence dimension by multiples of bptt and at the end to concatenate outputs from the encoder. This will give as <strong>batch x sequence length x output_dim (emb_size if tie_weights)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*k7dLdcQ-piYhrKRmkz4JNA.png" /><figcaption>Top Down Visualization of How MultiBatchEncoder Works</figcaption></figure><p>This allows you to process a full sentence by recursively passing chunks of sequences to the encoder (stacked LSTM). You might ask why can’t you just pass the whole sequence but set a BPTT instead? One reason is vanishing an exploding gradients due to too many recursive operations while updating the encoder. Another one is GPU memory allocation for storing gradients, more recursive operations means a larger computational graph. Lastly, having a smaller graph means less operations; faster training and cheaper update for a given parameter. For more on BPTT see this wonderful <a href="https://machinelearningmastery.com/gentle-introduction-backpropagation-time/">post</a>. Be cautious what we’ve been referring as BPTT is referred as truncated BPTT in that post.</p><h3>QRNN</h3><p>Earlier we’ve discussed how ULMFIT is constructed with stacked LSTMs and even bidirectional LSTMs (only difference being a concatenated output from both sides). Still there is one more encoder layer that you might consider instead of LSTM, QRNN — Quasi-Recurrent Neural Network, a work from some of the same authors from AWD-LSTM. Main motivation of QRNN is to introduce convolutions to parallelize the operations which is not possible with dependent LSTM layers. It is also stated that QRNN not only allows 16x faster computation speed but also has better performance compared to stacked LSTM in various tasks such as language modeling, sentiment classification, and character-level neural machine translation.[5]</p><p>CNNs have been used for sequence related tasks and it’s also known that they are inherently good at extracting features such as n-grams. Say you have a 2D CNN layer with a kernel size of 3x300 [n_seq x emb_size], this layer would allow you to extract features from 3 consecutive word tokens, or in other words 3-gram features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s6eRf9YeEGrjzUJQT9X_sg.png" /><figcaption>A view of how multiple filters applied for 2–3–4 gram features</figcaption></figure><p>But the nature of how these convolutions operate don’t allow them to utilize the full order information of the sequence. Even with state-of-art attention language models such as BERT order is encoded in a way to be consumed as an additional information and it’s known to be highly valuable.</p><p>QRNN addresses some of the problems CNNs and RNNs have; convolutions being time invariance and LSTMs being non-parallelized. We can say that QRNN combines best of two worlds: parallel nature of convolutions and time dependencies of LSTMs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RUSb9FmjTmzWPkyTT4HEPA.png" /><figcaption>LSTM, CNN and QRNN architectures compared</figcaption></figure><p>First set of operations in a QRNN layer we will output Z — candidate output, F— forget gate and O — output gate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/522/1*BG5PgKq__peQ8mFzLyNADQ.png" /></figure><p>In language modeling tasks convolution filters should only operate on previous tokens since the task is to predict the next token. QRNN uses a technique called masked convolutions which was introduced in Pixel RNN from Google Deep Mind in order to do so. This allows to build an autoregressive which doesn’t leak information from the future.[7]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lO8IbxfT4zd3pf7tyyqllw.png" /></figure><p>In QRNN paper masked convolutions said to be implemented by padding the input to the left by kernel size minus one. Let’s take a look at an example. Again we are starting of with a dummy sequence after embedding lookup <strong>[“I”, “love”, “cats”, “and”, “dogs”].</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RPSm7PTA3zoZ2FMiyav26A.png" /><figcaption>Wz, Wf and Wo for generate gate outputs</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/1*dIvi-GZlWjNOq3M6bTD83A.png" /><figcaption>We extract Z, F, O gates by doing a padded sum product of transposed Wz, Wf and Wo</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/1*jI6471fe8lPx6DPvK-bgjw.png" /><figcaption>Then apply <strong>tanh, sigmoid </strong>and<strong> sigmoid</strong> activations respectively</figcaption></figure><p>The operations above give us Z, F, O gates. Now we will look at how <strong>fo-pooling</strong> takes place. This is the part where time dependency takes place.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*YZ2kOVshE6aZ1eaBBrGzAA.png" /><figcaption>fo-pooling equation</figcaption></figure><p>There is also an option to use f-pooling which discards output gate by setting output_gate=False .</p><blockquote>Although the recurrent parts of these functions must be calculated for each timestep in sequence, their simplicity and parallelism along feature dimensions means that, in practice, evaluating them over even long sequences requires a negligible amount of computation time.</blockquote><p>Let’s have a look how ct and ht are computed at each step:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jd58VBHZOzDBuQaYfoPRGA.png" /><figcaption>sequential ct and hn generation</figcaption></figure><p>In order to capture larger n-grams kernel size can be increased when computing Z,F,O. Similarly to approximate more complex functions, multiple layers can be stacked just like stacked LSTMs. Multiple convolution plus pooling will surely give more powerful models. You may check <a href="http://neuralnetworksanddeeplearning.com/chap4.html">this</a> for a better understanding of universal approximation theorem. Yet we don’t want our model to overfit so we will still need some sort of regularization within the model itself.</p><p>Earlier while explaining AWD-LSTM we’ve seen how dropout scheme is important for not to disrespect the long term dependencies and how it’s important to have consistent dropout throughout the sequence forward pass. Due to QRNN’s lack of recurrent weights authors suggest applying dropout during the calculation of F gate because as you may also notice F gate is shared across all timestep calculations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/694/1*dSlWK9Fij2QUdBn--VFo1Q.png" /><figcaption>Dropout amount is also referred as “zone out” in the original paper</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gUnchyed5A14awUtLfjIgQ.png" /><figcaption>Here is the modified version of our sample illustration with dropout added</figcaption></figure><p>Remember that other dropout methods that we’ve seen for AWD-LSTM are also applicable here and might be good for further regularizing your model. QRNN is just an optional replacement for LSTM layers.</p><pre>AWD_LSTM(<strong>vocab_sz</strong>:int, <strong>emb_sz</strong>:int, <strong>n_hid</strong>:int, <strong>n_layers</strong>:int, <strong>pad_token</strong>:int, <strong>bidir</strong>:bool=<strong><em>False</em></strong>, <strong>hidden_p</strong>:float=<strong><em>0.2</em></strong>, <strong>input_p</strong>:float=<strong><em>0.6</em></strong>, <strong>embed_p</strong>:float=<strong><em>0.1</em></strong>, <strong>weight_p</strong>:float=<strong><em>0.5</em></strong>, <strong>qrnn</strong>:bool=<strong><em>True</em></strong>)</pre><p>In fast.ai implementation, the following matrices <strong>[Wz, Wf, Wo]</strong> are concatenated into a one matrix for performance considerations.<em> WeightDropout() </em>is applied to this matrix.</p><p><strong>End Notes</strong></p><p>I hope that I covered some of the important building blocks of ULMFIT and give you guys a better understanding through simple visualizations on how dropout is applied almost everywhere and how a QRNN layer works. For me, writing this blog post let me better understand how these state of the art NLP models work. So, I would suggest everyone to read the article, the source code, write about it and while doing so to try implementing these ideas yourself. For me using Microsoft Excel was easy and fast since all of these already are implemented in fast.ai.</p><p>For more stayed tune or just go explore the library and papers yourself. As I am about to publish this blog post I’ve seen that fast.ai started supporting <strong>Transformer</strong> and <strong>Transformer XL </strong>a day ago<strong>, </strong>two attention based models. I will try to do a benchmarking using all these options: A<strong>WD-LSTM, AWD-QRNN, Transformer </strong>and<strong> Transformer XL</strong> on <a href="https://www.kaggle.com/c/quora-insincere-questions-classification">Quora Insincere Questions Classification</a> dataset and edit this post in the coming days.</p><p><strong>Excel Work:</strong></p><p>If you want to check out the spreadsheet work I’ve created a copy of it in Google sheets <a href="https://docs.google.com/spreadsheets/d/1-Jr79UNo2T46StimpG-ZWdNPBuaOIQ-APWMaaMWUbYQ/edit?usp=sharing">here</a>. Since functions are not copied it’s static but might still be helpful for illustration purposes.</p><p><strong>References:</strong></p><ol><li><em>Improving neural networks by preventing co-adaptation of feature detectors </em>G. E. Hinton∗ , N. Srivastava, A. Krizhevsky, I. Sutskever and R. R. Salakhutdinov</li><li><em>Dropout: A Simple Way to Prevent Neural Networks from Overfitting </em>Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov</li><li>Understanding LSTM Networks <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/">http://colah.github.io/posts/2015-08-Understanding-LSTMs/</a></li><li><em>Regularizing and Optimizing LSTM Language Models </em>Stephen Merity, Nitish Shirish Keskar, Richard Socher</li><li><em>Quasi-Recurrent Neural Networks </em>James Bradbury, Stephen Merity , Caiming Xiong &amp; Richard Socher</li><li><em>A Convolutional Neural Network for Aspect Sentiment Classification </em>Yongping Xing and Chuangbai Xiao and Yifei Wu and Ziming Ding</li><li><em>Pixel Recurrent Neural Network </em>Aaron van den Oord Nal Kalchbrenner Koray Kavukcuoglu</li><li>Fast.ai documentation <a href="https://docs.fast.ai/">https://docs.fast.ai/</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=818d3775325b" width="1" height="1" alt=""><hr><p><a href="https://blog.mlreview.com/understanding-building-blocks-of-ulmfit-818d3775325b">Understanding building blocks of  ULMFIT</a> was originally published in <a href="https://blog.mlreview.com/">ML Review</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tensor Decomposition: Fast CNN in your pocket]]></title>
            <link>https://medium.com/@keremturgutlu/tensor-decomposition-fast-cnn-in-your-pocket-f03e9b2a6788?source=rss-25e7c0e46c50------2</link>
            <guid isPermaLink="false">https://medium.com/p/f03e9b2a6788</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Kerem Turgutlu]]></dc:creator>
            <pubDate>Thu, 05 Jul 2018 10:41:19 GMT</pubDate>
            <atom:updated>2018-07-05T10:41:19.356Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Why do we even care about tensor decomposition ?</strong></p><p>Even though it’s an active area of research and there are many applications of tensor decomposition, such as signal processing, numerical algebra, computer vision, numerical analysis, machine learning, graph analysis and many more. We will mainly learn about a technique that will help us to train our image classifiers faster and which will compress our models so that they can fit devices in our pockets and run efficiently in them.</p><p>Speeding up the training and reducing the memory footprint is not only an interest for big tech companies who want to run AI as mobile services but also useful for practitioners with low resources. <a href="http://www.fast.ai/2018/04/30/dawnbench-fastai/">DAWNBench</a> is a great example of what you can achieve with limited resources as long as you know the right techniques.</p><p><strong>Back to Basics</strong></p><p>Before getting started, this post will assume that the reader is already familiar with basic deep learning concepts, convolutional neural networks and basic linear algebra.</p><p>If you are already familiar with matrix decomposition then you can think tensor decomposition as a higher order matrix decomposition. If you are not familiar with it or need a refresher here we will briefly go over some of the basic ideas. In general we decompose matrices to solve systems of linear equations, to reduce the data dimension, to fill missing values, to find approximations for eigenvalues and for many other cool applications.</p><p><strong>Some of the popular matrix decomposition techniques are:</strong></p><p><strong>SVD</strong></p><p>Singular Value Decomposition (SVD) is a very fundamental algorithm to decompose any given matrix into <strong>UΣV*</strong>. Here U and V* are orthogonal matrices.</p><p>It is the generalization of the eigendecomposition of a positive semidefinite normal matrix. We can use QR shift factorization to find singular values and solve for SVD faster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_pYWy9GeJzHGq93j8TObQg.png" /></figure><p><strong>PCA</strong></p><p>Principal component analysis is a very popular practical method used in statistics and data science and it can be thought as a by product of SVD. In SVD we decompose our matrix into orthogonal and diagonal components and PCA tries to identify orthogonal vectors which maximizes the variance explained in data.</p><p><strong>Let X = UΣV* then principal components after ordering singular values can be written as:</strong></p><p>X1 = u1σ1v1*</p><p>X2 = u2σ2v2*</p><p>…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/1*XgAxftrCHNwaSSvaCi6zcg.png" /></figure><p><strong>NMF</strong></p><p>Non-negative matrix factorization (NMF) is widely used in recommender systems and topic modelling. NMF will decompose a non-negative data matrix into two components containing non-negative values by minimizing the reconstruction error, e.g. Frobenius norm. In terms of optimization, constraining ourselves into non-negativity is not fundamental but it increases the interpretability of the results obtained by the factorization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*75E815RPh2HqxMw4O6aGYA.png" /></figure><p><strong>LU</strong></p><p>LU decomposition uses Gaussian elimination to decompose a matrix into following two matrices: L for lower triangular and U for upper triangular.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/532/1*fZ_2Ezol8gus-K4WMMACbQ.jpeg" /></figure><p><strong>QR</strong></p><p>QR is a very fundamental algorithm in linear algebra it is used to answer 3 out of 4 main problems that linear algebra tries to solve. It is used in <strong>Least Squares Problem, Eigenvalue Problem </strong>and<strong> Finding SVD Problem. </strong>For all these problems QR factorization is the optimal algorithm.</p><p>There are mainly three ways of computing QR decomposition, Gram-Schmidt process, householder reflections and Givens rotations. QR decomposes a matrix into orthogonal and upper triangular matrices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/586/1*EDXjxWJ7euR2fSs1a0b3yA.png" /></figure><p><strong>CHOLESKY</strong></p><p>We can use cholesky decomposition to solve for <strong>Ax = b</strong>, <strong>Least Squares Problem</strong> though still QR is more optimal compared to Cholesky. Cholesky factorization is a special case of LU where A is positive definite and algorithm is stable. It will decompose into a lower triangular matrix and conjugate transpose of that matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/401/1*PtaRHE4wRmt77TyJ4mZCRg.png" /></figure><p><strong>Tensor Decomposition</strong></p><p>Both CP (CANDECOMP/PARAFAC) and Tucker decompositions can be considered to be higher order generalizations of the matrix singular value decomposition (SVD) and principal component analysis (PCA).</p><p><strong>Canonical Rank:</strong> Minimal possible R in CP decomposition.</p><p>In 2d low-rank approximation can be computed in a stable way by using SVD or if the matrix is large by rank revealing algorithms. This is not the case for the CP decomposition when d &gt; 2, as there is no finite algorithm for determining canonical rank of a tensor. Therefore, most algorithms approximate a tensor with different values of R until the approximation error becomes small enough.</p><p><strong>Terminology</strong></p><p>Here is some terminology about tensors.</p><p><strong>NLS:</strong> Non-linear least square which minimizes L2-norm of the approximation residual for a user defined fixed R.</p><p><strong>a:</strong> vector</p><p><strong>A:</strong> matrix</p><p><strong>X:</strong> tensor</p><p><strong>fiber:</strong> Fixing every index but one in a tensor.</p><p><strong>row:</strong> mode-2 fiber</p><p><strong>column:</strong> mode-1 fiber</p><p><strong>slice:</strong> 2 dimensional sections of a tensor</p><p>Third order tensors have column, row, and tube fibers denoted as x:jk, xi:k, xij:</p><p><strong>horizontal view:</strong> Xi::</p><p><strong>lateral view:</strong> X:j:</p><p><strong>frontal view:</strong> X::k</p><p><strong>||X||</strong> = norm of tensor is square root of sum of square of all of it’s elements</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*jSdgdilJFzjeEGKuy4QaSQ.png" /></figure><p><strong>Inner Product</strong> of two same sized tensors is the sum of the product of their entries:</p><pre>import torch<br>import tensorly as tl # using pytorch &gt;= 0.4 backend</pre><pre>A = torch.randint(0, 100, (3,3,3))<br>B = torch.randint(0, 100, (3,3,3))<br>tl.tenalg.inner(A, B) == torch.sum((A*B))</pre><p>&lt;X, X&gt; = X²</p><p>An N way tensor or array is <strong>rank one </strong>if it can be written as the outer product of N vectors:</p><p>X = a o b o c</p><p><strong>o:</strong> denotes outer product</p><p>If a tensor has same size along every dimension (every mode is same size) then it’s called <strong>cubical.</strong></p><p><strong>Matricization</strong>: Unfolding in a given dimension(s) to make a tensor a matrix.</p><p><strong>Important Matrix Operations</strong></p><p><strong>Kronecker Product:</strong> Generalization of outer product from vectors to matrices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*rFvgB_2yDFBFndLnA8MQxw.png" /></figure><p><strong>Khatri-Rao:</strong> Kronecker product using columns of each matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*qV9N13mI-XjcCbhcdPDrxw.png" /></figure><p>Now, we will look into a particular tensor decomposition algorithm called CP Decomposition.</p><p><strong>CP Decomposition</strong></p><p>The basic idea behind CP decomposition is that you define a rank R, then sum of outer product of rank 1 tensors is your approximation. Still, approximating rank of a tensor is a hard problem:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*gQnJBExMurrrjHZM1z3fSQ.png" /></figure><p>tensorly is a great library when working with tensors and it provides off the shelf functions for tensor decomposition. In this library CP decomposition optimizes Frobenius norm of difference of <strong>constructed tensor</strong> and the <strong>original tensor</strong> using <strong>ALS (Alternating Least Squares).</strong></p><p>Here is our objective function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/403/1*SdClCL8z1sMqmPTnpHKFUA.png" /><figcaption>Minimizing Frobenius Norm</figcaption></figure><p>But we can also use PyTorch and Adam optimizer or any other optimizer to implement CP decomposition ourselves. Here is a code snippet for decomposing a rank-3 tensor using CP decomposition via Adam optimizer, learning rate is adjusted by trial and error so it may change for your own case. I’ve also compared construction errors for different ranks between tensorly and my implementation, difference was negligible. So, we can say that below is a PyTorch version of the same optimization problem.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f6ea7f67e54316964b376bc63ac3e3e4/href">https://medium.com/media/f6ea7f67e54316964b376bc63ac3e3e4/href</a></iframe><p>Here is a plot of approximation errors for different rank decomposition. As you will notice as we increase the rank reconstruction error decreases, which is the expected behavior.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fc3pHyY8Ld9U0DN2ftaGHg.png" /></figure><p>In part 2 of this blog post, I will explain how we can use CP decomposition to decompose convolutional layers of a CNN and reduce the memory footprint of a image classifier like VGG, so stay tuned !</p><p>Thanks for reading the blog, hope you enjoyed it !</p><p><strong>References:</strong></p><ul><li>tensorly library: talk bit about tensorly too</li><li><a href="https://epubs.siam.org/doi/pdf/10.1137/07070111X">https://epubs.siam.org/doi/pdf/10.1137/07070111X</a></li><li><a href="https://arxiv.org/pdf/1412.6553.pdf">https://arxiv.org/pdf/1412.6553.pdf</a></li><li><a href="http://www.dsrc.rpi.edu/presentations/Acar_RPI11.pdf">http://www.dsrc.rpi.edu/presentations/Acar_RPI11.pdf</a></li><li><a href="https://arxiv.org/abs/1802.10410">https://arxiv.org/abs/1802.10410</a></li><li><a href="https://jacobgil.github.io/deeplearning/tensor-decompositions-deep-learning">https://jacobgil.github.io/deeplearning/tensor-decompositions-deep-learning</a></li><li><a href="https://medium.com/@nicolas_19145/state-of-the-art-in-compressing-deep-convolutional-neural-networks-cfd8c5404f22">https://medium.com/@nicolas_19145/state-of-the-art-in-compressing-deep-convolutional-neural-networks-cfd8c5404f22</a></li><li><a href="https://arxiv.org/pdf/1412.6553.pdf">https://arxiv.org/pdf/1412.6553.pdf</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f03e9b2a6788" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Semantic Segmentation — U-Net]]></title>
            <link>https://medium.com/@keremturgutlu/semantic-segmentation-u-net-part-1-d8d6f6005066?source=rss-25e7c0e46c50------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8d6f6005066</guid>
            <category><![CDATA[semantic-segmentation]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[unet]]></category>
            <category><![CDATA[data-science-bowl]]></category>
            <dc:creator><![CDATA[Kerem Turgutlu]]></dc:creator>
            <pubDate>Fri, 20 Apr 2018 10:58:13 GMT</pubDate>
            <atom:updated>2019-05-06T00:13:35.844Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Here again writing to my 6 months ago self…</em></p><p>In this post I will mainly be focusing on semantic segmentation, a pixel-wise classification task and a particular algorithm for it. I will be providing a walk-through on some of the cases I had and have been working on lately.</p><p>By definition, semantic segmentation is the partition of an image into coherent parts. For example classifying each pixel that belongs to a person, a car, a tree or any other entity in our dataset.</p><p><strong>Semantic Segmentation vs. Instance Segmentation</strong></p><p>Semantic segmentation is relatively easier compared to it’s big brother, instance segmentation.</p><p>In instance segmentation, our goal is to not only make pixel-wise predictions for every person, car or tree but also to identify each entity separately as person 1, person 2, tree 1, tree 2, car 1, car 2, car 3 and so on. Current state of the art algorithm for instance segmentation is Mask-RCNN: a two-stage approach with multiple sub-networks working together: RPN (Region Proposal Network), FPN (Feature Pyramid Network) and FCN (Fully Convolutional Network) [5, 6, 7, 8].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*pKKYS17lOwPsreUVTak37g.png" /><figcaption>Fig 4. Semantic Segmentation</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*C90jdvqH1-Kc67wEYwtFLQ.png" /><figcaption>Fig 5. Instance Segmentation</figcaption></figure><h3>Case Study: Data Science Bowl 2018</h3><p>Data Science Bowl 2018 just ended and I’ve learned a lot from it. Maybe the most important lesson I learned was, even with deep learning, a more automated technique compared to traditional ML, pre and post processing might be crucial to get good results. Those are important skills for a practitioner to obtain and they define the way you structure and model the problem.</p><p>I will not go through every little detail and explanation about this particular competition since there is great amount of discussion and explanation on both the task itself and the methods used throughout the competition <a href="https://www.kaggle.com/c/data-science-bowl-2018/discussion/54741">here</a> . But I will briefly mention the winning solution as it is related to the foundations of this post. [13]</p><p>Data Science Bowl 2018 just like other Data Science Bowls in the past was organized by Booz Allen Foundation. This year’s task was to identify nuclei of cells in a given microscopy image and to provide masks for each nucleus independently.</p><p>Now, take a moment or two to guess which type of segmentation this task demands; semantic or instance ?</p><p>Here is a sample masked image and it’s raw microscopy image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/1*Lj9cyAXoTtOnB_nM5fwMyw.png" /><figcaption>Fig 6. Masked Nuclei (left) and Original Image (right)</figcaption></figure><p>Even though it may sound like a semantic segmentation task at first, the task here is instance segmentation. We need to treat each nucleus in the image independently and identify them as nuclei 1, nuclei 2, nuclei 3, … similar to the example we had for car 1, car 2, person 1 and so on. Perhaps the motivation for this task is to track the sizes, the counts and the characteristics of nuclei from a cell sample over time. It is very important to automate this tracking process and further speed up the experimentation of running on different treatments for curing various diseases.</p><p>Now, you may think that if this article is about semantic segmentation and if Data Science Bowl 2018 is an example of instance segmentation task, then why am I keep talking about this particular competition. If you are thinking about this, then you are definitely right and indeed the end goal for this competition was not an example for semantic segmentation. But as we will keep going you will see how you can actually turn this instance segmentation problem into a multiclass semantic segmentation task. This was the approach that I’ve tried but failed in practice but also turned out to be the high level motivation for the winning solution too.</p><p>During this 3 months of period of the competition there were only two models (or variants of them) which were shared or at least explicitly discussed throughout the forums; Mask-RCNN and U-Net. As I’ve mentioned earlier Mask-RCNN is the state-of-the-art algorithm for object detection which detects individual objects and predicts their masks, as in instance segmentation. Mask-RCNN’s implementation and training is harder since it employs a two-stage learning approach, where you first optimize for an RPN (Region Proposal Network) and then predict bounding boxes, classes and masks simultaneously.</p><p>On the other hand U-Net is a very popular end-to-end encoder-decoder network for semantic segmentation [9]. It was originally invented and first used for biomedical image segmentation, a very similar task we had for Data Science Bowl. There was no silver bullet in the competition, and none of these two architectures alone without post or pre-processing or any minor tweaks in architectural design proved to have a top score. I didn’t have the chance to try Mask-RCNN for this competition, so I kept my experiments around U-Net and learned a lot about it.</p><p>Also, since our topic is semantic segmentation I will leave Mask-RCNN to other blog posts out there to explain. But if you still insist to try them in your own CV applications, here are two popular github repositories with implementations in <a href="https://github.com/matterport/Mask_RCNN">Tensorflow</a> and <a href="https://github.com/multimodallearning/pytorch-mask-rcnn">PyTorch</a>. [10, 11]</p><p>Now, we may continue with U-Net and dive deeper into it’s details...</p><p>Here is the architecture for start:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*dKPBgCdJx6zj3MpED3lcNA.png" /><figcaption>Fig 7. Vanilla U-Net</figcaption></figure><p>For those who are familiar with traditional convolutional neural networks first part (denoted as DOWN) of the architecture will be familiar. This first part is called down or you may think it as the encoder part where you apply convolution blocks followed by a maxpool downsampling to encode the input image into feature representations at multiple different levels.</p><p>The second part of the network consists of upsample and concatenation followed by regular convolution operations. Upsampling in CNNs may be a new concept to some of the readers but the idea is fairly simple: we are expanding the feature dimensions to meet the same size with the corresponding concatenation blocks from the left. You may see the gray and green arrows, where we concatenate two feature maps together. The main contribution of U-Net in this sense compared to other fully convolutional segmentation networks is that while upsampling and going deeper in the network we are concatenating the higher resolution features from down part with the upsampled features in order to better localize and learn representations with following convolutions. Since upsampling is a sparse operation we need a good prior from earlier stages to better represent the localization. Similar idea of combining matching levels is also seen in FPNs (Feature Pyramidal Networks). [7]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/765/1*Y5CRI3eoVsjf570nkWEcDg.png" /><figcaption>Fig 7. Vanilla U-Net Tensor Annotation</figcaption></figure><p>We can define one block of operations in down part as convolutions→ downsampling.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0ec72a988641cbf18879b923dadb5160/href">https://medium.com/media/0ec72a988641cbf18879b923dadb5160/href</a></iframe><p>Similarly we can define one block of operations in up part as upsampling → concatenation →convolutions.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cc4491f6f81bff2ca8e802886b0c88dc/href">https://medium.com/media/cc4491f6f81bff2ca8e802886b0c88dc/href</a></iframe><p>By inspecting the figure more carefully, you may notice that output dimensions (388 x 388) are not same as the original input (572 x 572). If you want to get consistent size, you may apply padded convolutions to keep the dimensions consistent across concatenation levels just like we did in the sample code above.</p><p>When such upsampling is mentioned you may come across to either one of the following terms: transposed convolution, upconvolution, deconvolution or upsamling. Many people including myself and PyTorch documentations don’t like the term deconvolution, since during upsampling stage we are actually doing regular convolution operations and there is nothing de- about it. Before going any further if you are unfamiliar with basic convolution operations and their arithmetic I highly recommend visiting <a href="https://github.com/vdumoulin/conv_arithmetic/blob/master/README.md">here</a>. [12]</p><p>I will be explaining upsampling methods from simplest to more complex. Here are three ways of upsampling a 2D tensor in PyTorch:</p><p><strong>Nearest Neighbor</strong></p><p>This is the simplest way of finding the values of missing pixels when resizing (translating) a tensor into a larger tensor, e.g. 2x2 to 4x4, 5x5 or 6x6.</p><p>Let’s implement this basic computer vision algorithm step by step using Numpy:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e456d4c6a53c40edffe57f84dd8a0e7f/href">https://medium.com/media/e456d4c6a53c40edffe57f84dd8a0e7f/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*IiEfK4NbvGrhWA7Nlsm7fg.png" /></figure><p><strong>[PyTorch]</strong> F.upsample(…, mode = “nearest”)</p><pre><strong>&gt;&gt;&gt; </strong>input = torch.arange(1, 5).view(1, 1, 2, 2)<br><strong>&gt;&gt;&gt; </strong>input<br><br>(0 ,0 ,.,.) =<br>  1  2<br>  3  4<br>[torch.FloatTensor of size (1,1,2,2)]<br><br><strong>&gt;&gt;&gt; </strong>m = nn.Upsample(scale_factor=2, mode=&#39;nearest&#39;)<br><strong>&gt;&gt;&gt; </strong>m(input)<br><br>(0 ,0 ,.,.) =<br>  1  1  2  2<br>  1  1  2  2<br>  3  3  4  4<br>  3  3  4  4<br>[torch.FloatTensor of size (1,1,4,4)]</pre><p><strong>Bilinear Interpolation</strong></p><p>Bilinear Interpolation Algorithm is less computationally efficient than nearest neighbor but it’s a more precise approximation. A single pixel value is calculated as the weighted average of all other values based on distances.</p><p><strong>[PyTorch] </strong>F.upsample(…, mode = “bilinear”)</p><pre><strong>&gt;&gt;&gt; </strong>input = torch.arange(1, 5).view(1, 1, 2, 2)<br><strong>&gt;&gt;&gt; </strong>input<br><br>(0 ,0 ,.,.) =<br>  1  2<br>  3  4<br>[torch.FloatTensor of size (1,1,2,2)]</pre><pre><strong>&gt;&gt;&gt; </strong>m = nn.Upsample(scale_factor=2, mode=&#39;bilinear&#39;)<br><strong>&gt;&gt;&gt; </strong>m(input)<br><br>(0 ,0 ,.,.) =<br>  1.0000  1.2500  1.7500  2.0000<br>  1.5000  1.7500  2.2500  2.5000<br>  2.5000  2.7500  3.2500  3.5000<br>  3.0000  3.2500  3.7500  4.0000<br>[torch.FloatTensor of size (1,1,4,4)]</pre><p><strong>Transposed Convolution</strong></p><p>In transposed convolutions we have weights that we learn through back-propagation. In papers I’ve come across all of these upsampling methods for various cases and also in practice you may change your architecture and try all of them to see which works best for your own problem. I personally prefer transposed convolutions since we have more control over it but you may go for bilinear interpolation or nearest neighbor for simplicity as well.</p><p><strong>[PyTorch]</strong> nn.ConvTranspose2D(…, stride=…, padding=…)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/948/1*-u7Cj5jpGUbWkdpT1Y-1Sg.png" /><figcaption>Fig 8. Examples for transposed convolution operation with different parameters. Credit goes to <a href="https://github.com/vdumoulin/conv_arithmetic">https://github.com/vdumoulin/conv_arithmetic</a> [12]</figcaption></figure><p>If we go back to our original case, Data Science Bowl, the main drawback of using a vanilla U-Net approach in the competition was the overlapping nuclei. As it’s seen in the image above if you create a binary mask and use it as your target, U-Net will surely predict something similar to this and you will have a combined mask for several nuclei which are overlapping or lie very close to each other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*ePiNH-RIVPaxNXH1WgFYVw.png" /><figcaption>Fig 9. Overlapping nuclei mask</figcaption></figure><p>Referring to overlapping instances problem, authors of U-Net paper used weighted cross entropy to emphasize learning the borders of cells. This method helped them to separate overlapping instances. Basic idea is to weight borders more and to push network towards learning gaps between close instances.[9]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*3dp84O3KCVCr1AsJt-QZWQ.png" /><figcaption>Fig 10. Weight map</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LPbooZuQG_IA0vmbxv96Eg.png" /><figcaption>Fig 11. (a) Raw image (b) ground truth different color for each instance (c) generated segmentation mask (d) pixel-wise weight map</figcaption></figure><p>Another solution to this kind of problem, an approach that was used by many competitors including the winning solution, is to convert binary masks into a multiclass target. The nice thing about U-Net is that you can structure your network to output as many channels as you want and represent any class in any channel by using 1x1 convolution at the final layer.</p><p>Quoting from the Data Science Bowl winning solution:</p><blockquote>2 channels masks for networks with sigmoid activation i.e. (mask — border, border) or 3 channels masks for networks with softmax activation i.e. (mask — border, border , 1 — mask — border)</blockquote><blockquote>2 channels full masks i.e. (mask, border)</blockquote><p>After making these predictions classical image processing algorithms like watershed can be used for post-processing to further segment individual nuclei. [14]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*DIbLJC1xv_ypx3QFwfXmag.png" /><figcaption>Fig 12. For visual purposes foreground (green) contour (yellow) background (dark) classes</figcaption></figure><p>This was the first official computer vision competition that I’ve had the courage to participate in Kaggle and it was a Data Science Bowl. Even though I’ve completed the competition only in top 20% (which is considered as an average score) I felt the pleasure of participating in a Data Science Bowl and learning the things that I could have never learn if I wasn’t actually participating and trying on my own. Active learning is far more fruitful than watching or reading about similar approaches from online sources.</p><p>As a deep learning practitioner who just started practicing months back with Fast.ai this was an important step for me towards my never ending journey and it was very valuable in terms of gaining experience. So, for those who feel intimated by challenges that you’ve never seen or solved before I highly recommend you to specifically go after these type of challenges in order to feel the great pleasure of learning something that you didn’t know before.</p><p>Another valuable lesson I’ve learned in this competition was that, in a computer vision (this applies to NLP too) competition it’s very important to check every single prediction by eye to see what’s working and what’s not. If your data is small enough you should go and check each and every individual output for sure. This will allow you to further come up with better ideas or even debug your code if something is wrong with it.</p><h3><strong>Transfer Learning and Beyond</strong></h3><p>So far we’ve defined the building blocks of vanilla U-Net and mentioned how we can manipulate targets to solve for instance segmentation. Now we can further discuss the flexibility of these type of encoder-decoder networks. By flexibility I mean the freedom you have over it and the creativity you can put on it’s design.</p><p>Anyone who practices deep learning at some point come across to transfer learning because it’s a very powerful idea. In short transfer learning is the concept of using a pretrained network which was trained on many samples for a similar task that we are facing but lacking the same amount of data. Even with enough data transfer learning can boost the performance up to some extent, not only for computer vision tasks but also for NLP too.</p><p>Transfer learning proved to be a powerful technique for U-Net like architectures as well. We’ve previously defined two major components of U-Net; down and up. Let’s rephrase these parts as encoder and decoder this time. Encoder part basically takes the input and encodes it in a low dimensional feature space which represents our input in a lower dimension. Now imagine replacing this encoder with your favorite ImageNet winner; VGG, ResNet, Inception, NasNet, … which ever you want. These networks are highly engineered to do one common thing: to encode a natural image in the best way possible to classify it and their pretrained weights on ImageNet are waiting for you to grab them online.</p><p>So why not use one of these architectures as our encoder and construct the decoder in a way that it will work the same way as the original U-Net, but better, on steroids.</p><p>TernausNet which is the winner architecture for <a href="https://www.kaggle.com/c/carvana-image-masking-challenge">Kaggle Carvana</a> challenge uses the same transfer learning idea with VGG11 as it’s encoder. [15, 16]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*mqdUlED6AuhZGip7Ov1o-g.png" /><figcaption>TernausNet by Vladimir Iglovikov and Alexey Shvets</figcaption></figure><h3>Fast.ai: Dynamic U-Net</h3><p>Inspired by TernausNet paper and by many other great resources, I wanted to generalize this idea of using pretrained or custom encoders for U-Net like architectures. So, I’ve came up with a general architecture: <a href="https://github.com/fastai/fastai/blob/master/fastai/vision/models/unet.py"><strong>Dynamic Unet</strong></a>.</p><p>Dynamic Unet is an implementation of this idea, it automatically creates the decoder part to any given encoder by doing all the calculations and matching for you. Encoder can either be a pretrained network off the shelf or any custom architecture you define yourself.</p><p>It is written in PyTorch and currently in Fast.ai library. You can refer to this <a href="https://github.com/KeremTurgutlu/deeplearning/blob/master/datasciencebowl2018/FASTAI%20-%20DSBOWL%202018.ipynb">notebook</a> to see it in action or look at <a href="https://github.com/fastai/fastai/blob/master/fastai/models/unet.py">source</a>. The main goal of Dynamic Unet is to save practioners’ time and allow easier experimentation with different encoders with the least amount of code possible.</p><p>In part 2 I will be explaining 3D Encoder Decoder models for volumetric data, such as MRI scans and give real world examples that I’ve been working on.</p><p><strong>References</strong></p><p>[5] Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks: <a href="https://arxiv.org/abs/1506.01497">https://arxiv.org/abs/1506.01497</a></p><p>[6] Mask R-CNN: <a href="https://arxiv.org/abs/1703.06870"><em>https://arxiv.org/abs/1703.06870</em></a></p><p>[7] Feature Pyramid Networks for Object Detection: <a href="https://arxiv.org/abs/1612.03144"><em>https://arxiv.org/abs/1612.03144</em></a></p><p>[8] Fully Convolutional Networks for Semantic Segmentation: <a href="https://people.eecs.berkeley.edu/~jonlong/long_shelhamer_fcn.pdf"><em>https://people.eecs.berkeley.edu/~jonlong/long_shelhamer_fcn.pdf</em></a></p><p>[9] U-net: Convolutional Networks for Biomedical Image Segmentation: <a href="https://arxiv.org/abs/1505.04597"><em>https://arxiv.org/abs/1505.04597</em></a></p><p>[10] Tensorflow Mask-RCNN: <a href="https://github.com/matterport/Mask_RCNN"><em>https://github.com/matterport/Mask_RCNN</em></a></p><p>[11] Pytorch Mask-RCNN:<em> </em><a href="https://github.com/multimodallearning/pytorch-mask-rcnn"><em>https://github.com/multimodallearning/pytorch-mask-rcnn</em></a></p><p>[12] Convolution Arithmetic: <a href="https://github.com/vdumoulin/conv_arithmetic/blob/master/README.md"><em>https://github.com/vdumoulin/conv_arithmetic</em></a></p><p>[13] Data Science Bowl 2018 Winning Solution, ods-ai: <a href="https://www.kaggle.com/c/data-science-bowl-2018/discussion/54741"><em>https://www.kaggle.com/c/data-science-bowl-2018/discussion/54741</em></a></p><p>[14] Watershed Algorithm <a href="https://docs.opencv.org/3.3.1/d3/db4/tutorial_py_watershed.html">https://docs.opencv.org/3.3.1/d3/db4/tutorial_py_watershed.html</a></p><p>[15] Carvana Image Masking Challenge: <a href="https://www.kaggle.com/c/carvana-image-masking-challenge">https://www.kaggle.com/c/carvana-image-masking-challenge</a></p><p>[16] TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image Segmentation: <a href="https://arxiv.org/abs/1801.05746"><em>https://arxiv.org/abs/1801.05746</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d8d6f6005066" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Debut: dicom-contour]]></title>
            <link>https://medium.com/@keremturgutlu/debut-dicom-contour-9ce74326bdbe?source=rss-25e7c0e46c50------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ce74326bdbe</guid>
            <dc:creator><![CDATA[Kerem Turgutlu]]></dc:creator>
            <pubDate>Sun, 11 Mar 2018 16:59:20 GMT</pubDate>
            <atom:updated>2018-03-11T16:59:20.147Z</atom:updated>
            <content:encoded><![CDATA[<p>pip install dicom-contour</p><p>Today we are announcing the release of a pip package called <strong>dicom-contour. </strong>This package aims to take the burden from scientists, radiologists, deep learning practitioners like me and many other people who are working with DICOM images in a Python environment. This package will allow you to speed up the cumbersome data preparation stage needed before feeding the appropriate data for your medical deep learning applications. There are already very good packages for working with DICOM files, like <strong>pydicom</strong>, but they are only the best for some specific tasks like reading, modifying and writing. With <strong>dicom-contour </strong>we are solving another very specific task which is not very trivial, I will explain it in detail throughout this post but here is an example of the things that you can do with this package.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/1*aTIuLxg0GhKD36x4ElCbyg.png" /><figcaption>Reading a DICOM Slice with Regular Packages v.s. What <strong>dicom-contour</strong> offers in addition</figcaption></figure><p>Let’s quickly go over and explain what a DICOM file is and why one might actually need this package.</p><p><strong>DICOM</strong> is one of the most widely used medical image file format along with others like <strong>Nifti</strong> and <strong>Minc</strong>. <strong>Digital Imaging and Communications in Medicine</strong> (<strong>DICOM</strong>) is a standard for storing and transmitting medical images enabling the integration of medical imaging devices such as scanners, servers, workstations, printers, network hardware, and picture archiving and communication systems (PACS) from multiple manufacturers. It has been widely adopted by hospitals, and is making inroads into smaller applications like dentists’ and doctors’ offices. It is often the case that these files are processed and manipulated in the hands of the experts using an expensive commercial software, which has many capabilities like viewing, thresholding, transforming and drawing contours around a region of interest.</p><p>So far so good, but when it comes to data science and deep learning it is often the case that we will use Python. Also, we are not quite interested in the unstructured/nested nature of DICOM files but rather we are mostly (almost 99% of the time) interested in Numpy arrays. Of course meta data about a patient is very valuable but arrays or tensors will be the core input for our project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*xGEPu9bTZBG3FcNHWktRyQ.png" /><figcaption>An example of nested DICOM file structure</figcaption></figure><p>It is pretty straightforward to read regular DICOM images with <strong>pydicom </strong>package<strong> </strong>and call <strong>pixel_array</strong> method to extract pixel intensities as Numpy arrays. But things get complicated when you have an additional contour file and you want to use it for tasks like <a href="http://blog.qure.ai/notes/semantic-segmentation-deep-learning-review">semantic segmentation</a>. A contour is typically used by medical experts in order to point out a certain part of a scan by drawing a continuous line around the region of interest, e.g. a lesion or a tumor on a brain MRI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DvkkvwWb9tUqTafBdjecJw.png" /><figcaption>Reading and extracting an image with pydicom</figcaption></figure><p>A common way to store a patient data, or at least this is what the export of a commercial software looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/181/1*DED3o6J3HshPFCLN2S42Pg.png" /><figcaption>Common way to store dicom data</figcaption></figure><p>You will have a folder for each patient which will have a DICOM file for each image slice and an additional contour file for contour information. But extracting a Numpy array from a contour file is not trivial.</p><p>There are multiple reasons for this:</p><ul><li><strong>A contour file might have multiple contours</strong>, meaning that it might hold the labels that were drawn by multiple experts. Since drawing a contour is still done manually, it is prone to human error. And most of the time a consensus of many experts is needed.</li><li><strong>Contour line information is stored in Cartesian coordinates</strong>. As it is mentioned above, contour is a continuous line drawn by an expert but a line in computer vision is a collection of pixels. But the way a commercial software exports each pixel is like the following; (x, y, z) — (height, width, depth) in mm. So we need to find a mapping from Cartesian coordinates to pixel positions in order to construct our Numpy arrays.</li><li><strong>Filenames under a patient folder are not conventional</strong>. Meaning that they are not as pretty as we demonstrated as an example above but they are rather named after machine generated numbers. So you also need to detect which file is the contour file.</li><li><strong>There is no package out there to read dicom files and return 3D Numpy</strong> <strong>arrays for both images and contours</strong>. Why do we want it ? Again, it is to have an input and output pair for our supervised segmentation tasks.</li></ul><p>Hence,<strong> dicom-contour </strong>takes all the suffering from you and serves as a collection of supplementary functions on top of <strong>pydicom </strong>in order to tackle this very specific problem we’d been discussing from the beginning of this post.</p><p><strong>Story behind dicom-contour and why it is released ?</strong></p><p>Since day 1 we had the idea of releasing the code we’ve written as an open source package but I was never convinced if it was really needed or if anyone in this area would ever use it. Just today I had the opportunity and pleasure of meeting a USF Data Science alumna who is currently working as a Bioinformatics Programmer in UCSF and applying deep learning models on cardiovascular images . She asked me about my current practicum(internship) in UCSF and wanted to know the biggest challenge or unusual problem I’ve faced so far.</p><p>In short in our practicum, we are currently working on developing a single multi-task 3D CNN for brain tumor segmentation. Along with segmenting the tumor, this network will also be predicting local failure and overall survival of a patient and the grade of the tumor simultaneously. So, you may think it as a network that will output a binary mask along with binary outputs for those other classification tasks.</p><p>Even though this sounds like a very challenging problem which requires encoder-decoder CNNs as well as back propagating from an aggregated complex loss function, it was not even a part of the answer that I gave to the question <strong>“What was the most challenging problem you’ve faced?”</strong>. My answer within a second without any further hesitation was: <strong>“Converting DICOM images with contour files into 3D numpy arrays for supervised learning.</strong>”</p><p>This sounds like a trivial problem I know, because converting A to B is probably the easiest thing you can do in computer science, not because the process is easy but because there is a plethora of open source resources that would be available to you for your specific task.</p><p>But I promise you, after days of online searches the best thing I’ve came across was a google group chat that happened ~5 years ago. Which didn’t quite give what I was looking for but rather it convinced me more into the idea of writing a pipeline for this process from scratch.</p><p>An example thread on this problem: <a href="https://stackoverflow.com/questions/46666762/getting-dicom-structure-contours-as-array-in-python">https://stackoverflow.com/questions/46666762/getting-dicom-structure-contours-as-array-in-python</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/847/1*ki_Hq4SKThF4QEh5_PfZJg.png" /><figcaption>A recent google search</figcaption></figure><p>Constructing the pipeline was not very easy since me and my co-worker are not experts in this field. We are just Data Science Master’s students who are passionate about collaborating in medical deep learning research as much as we can.</p><p>During this process we read DICOM documentations to get a better feel of the structure of the data and consulted experts in this field through <a href="http://www.fast.ai/">Fast.ai</a> forums, here is the actual <a href="http://forums.fast.ai/t/lung-cancer-detection-convolution-features-gradient-boost/1442/30?u=kcturgutlu">discussion thread</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/1*QIngzrRfwn5ECyhTOuuD5w.png" /><figcaption>Great support and help from the community</figcaption></figure><p>So, if we come back to the chat I’ve had today with our alumna, it was something like this:</p><ul><li><strong>What was the biggest challenge you’ve faced ?</strong></li><li>It was definitely converting DICOM images with contours into numpy array for deep learning models. I couldn’t find anything online.</li><li><strong>I know I am also working with DICOM but haven’t seen a code for that.</strong></li><li>Maybe I should package that for other people :)</li><li><strong>Yeah, maybe you should!</strong></li></ul><p>This is the summary of why today I am announcing this package which is available on pip, via pip install dicom-contour . A detailed tutorial is available here: <a href="https://github.com/KeremTurgutlu/dicom-contour">https://github.com/KeremTurgutlu/dicom-contour</a>.</p><p><strong>But still I will examples of functions in the package:</strong></p><p><strong>path: </strong>Path of the directory which has all the DICOM (.dcm) files including images and the contour filename.</p><p><strong>file: </strong>Contour filename.</p><p><strong>ROIContourSequence/index: </strong>Index of the contour sequence to get. Contour sequences can be multiple, e.g. different specialists labeling an image, so it should be specified here.</p><p>get_contour_file(path) : Returns the filename of contour file in the path.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/699/1*YJ8ESN0T152pj3G8mc2f4w.png" /></figure><p>get_roi_names(contour_data) : Returns available contour sequence names. There are usually multiple contours as we discussed before.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*HZplav7azIzyzw0PBz8L8Q.png" /></figure><p>cfile2pixels(file, path, ROIContourSequence) : Returns a list which have pairs of img_arr and contour_arr for a given contour file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/888/1*5hN_aBXIQUCO6drBeWnIYg.png" /></figure><p>slice_order(path) : Returns an ordered list of images/slices by their position. Order is important for 3D applications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/847/1*xiXq0112tDQagJtnKbOPwA.png" /></figure><p>get_data(path, index) : Returns a 3D Numpy array for both images and contours. This is mostly all you need to convert DICOM into Numpy array.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/559/1*P00ccjF0CEdJHfH6-IpsIw.png" /><figcaption>120 slices 512x512 resolution</figcaption></figure><p>create_image_mask_files(path, index, img_format=&#39;png&#39;) : Create image and corresponding mask files under the folders <strong>/images</strong> and <strong>/masks</strong> under the parent directory of path. This function will additionally fill contours to form masks which are more suitable for deep learning tasks. All images are individual slices here. You can later use these images to run 2D applications as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*TTthcTa7jZILjCNNYY2WIA.png" /></figure><p>I hope this package will help others like me, who are not familiar with DICOM images and contours. Enabling them to get on track rather fast and focus on more important things like optimizing their neural net to help people !</p><p>Many thanks to <a href="https://twitter.com/jeremyphoward">Jeremy Howard</a> for taking us from zero background and turning us into research level deep learning practitioners in just few months. If you are interested in deep learning, without a doubt this <a href="http://fast.ai/">MOOC</a> is one of the best resources out there.</p><p>I would here like to thank my co-auhtor <a href="https://www.linkedin.com/in/siavashmortezavi/">Siavash Mortezavi</a> for figuring out some of the most critical points of the project and looking at it from another angle. Please check his amazing github for cool introductions <a href="https://github.com/smortezavi">https://github.com/smortezavi</a>.</p><p>Special thanks to all <a href="http://www.fast.ai/">Fast.ai</a> family and experts in the field that helped me out even with my most boring questions.</p><p><strong>Future Work:</strong></p><p>I will be writing more about medical deep learning where I will mention both our 3D multi task learning project as well as the ongoing Data Science Bowl 2018. Stay tuned for more, in the meantime you may want to check this out:</p><p><a href="https://github.com/KeremTurgutlu/deeplearning">KeremTurgutlu/deeplearning</a></p><p>Linkedin: <a href="https://www.linkedin.com/in/kerem-turgutlu-12906b65/">https://www.linkedin.com/in/kerem-turgutlu-12906b65/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ce74326bdbe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[End-to-end Distributed ML using AWS EMR, Apache Spark (Pyspark) and MongoDB Tutorial with…]]></title>
            <link>https://medium.com/data-science/end-to-end-distributed-ml-using-aws-emr-apache-spark-pyspark-and-mongodb-tutorial-with-4d1077f68381?source=rss-25e7c0e46c50------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d1077f68381</guid>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[mongodb]]></category>
            <dc:creator><![CDATA[Kerem Turgutlu]]></dc:creator>
            <pubDate>Fri, 19 Jan 2018 04:41:58 GMT</pubDate>
            <atom:updated>2018-01-21T03:08:40.518Z</atom:updated>
            <content:encoded><![CDATA[<h3>End-to-end Distributed ML using AWS EMR, Apache Spark (Pyspark) and MongoDB Tutorial with MillionSongs Data</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J28sEr2RzBwMgvQYuMZl2g.jpeg" /></figure><p>In this post I will mention how to run ML algorithms in a distributed manner using Python Spark API pyspark. We will also learn about how to set up an AWS EMR instance for running our applications on the cloud, setting up a MongoDB server as a NoSQL database in order to store unstructured data (such as JSON, XML) and how to do data processing/analysis fast by employing pyspark capabilities.</p><p><strong>Requirements:</strong></p><ul><li>AWS account (or you can run on local)</li><li>Python 2.7 (for running the scripts that will be provided)</li></ul><h3><strong>1. Setting up an EMR instance</strong></h3><p>What is Amazon EMR ? (url: <a href="https://aws.amazon.com/emr/">https://aws.amazon.com/emr/</a>)</p><blockquote>Amazon EMR provides a managed Hadoop framework that makes it easy, fast, and cost-effective to process vast amounts of data across dynamically scalable Amazon EC2 instances. You can also run other popular distributed frameworks such as <a href="https://aws.amazon.com/emr/details/spark/">Apache Spark</a>, <a href="https://aws.amazon.com/emr/details/hbase/">HBase</a>, <a href="https://aws.amazon.com/emr/details/presto/">Presto,</a> and <a href="https://aws.amazon.com/blogs/big-data/use-apache-flink-on-amazon-emr/">Flink</a> in Amazon EMR, and interact with data in other AWS data stores such as Amazon S3 and Amazon DynamoDB.</blockquote><p>Even though this step will be optional in order to run the scripts and understanding the outline of what’s going on from top the bottom, usually the main reason why we might need distributed computing is the fact that our local machine specs are not capable of doing the processing, analysis or modeling we want.</p><p>We might as well launch multiple EC2 instances and configure master and worker nodes but all of these steps are actually taken care of EMR. EMR in short will allow us to automatically distribute jobs we want to run once we run them on our master instance. Otherwise we should be configuring and installing Spark on all nodes.</p><p>Below you can see how jobs are distributed through Spark framework. Driver runs the main process converting it into tasks and schedules them for executors. Then workers (executors) run these tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*94sLpx-6ScSxcm5dTAuO_w.png" /><figcaption>Driver and Executor Architecture in Spark</figcaption></figure><p>For launching our first EMR instance we need to login to aws.amazon.com and then console will appear.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/767/1*f6ZDvUsLfXdh-q97-D3aig.png" /><figcaption>select “EMR” here</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/1*TNdhep2SLP-Cx7onECGKPA.png" /><figcaption>click “Create Cluster”</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/1*yx8cBIjT7jqnSrTOISoMYg.png" /><figcaption>Here we should select “Spark” under Applications and define how many instances we want for our application. For example 1 for master and 2 for core nodes (executors which will run tasks). So here we will be distributing the job into 2 worker nodes.</figcaption></figure><p>You should also give a key-pair in order to ssh into nodes later and for running your application. Once you click create your cluster it will start bootstrapping and once everything is ready master and cores will be waiting. One key point about EMR is that it is not stoppable once you start it. Meaning you can’t stop and start EMR instance like regular EC2 instances. So it’s good practice to run all the applications and then terminate the EMR. If you try to stop without jobs get done you will need to start over again. Another common practice for data processing or analysis jobs is to use Amazon S3. EMR, S3, Spark get along very well together. You can store your data in S3, then read and process it without actually storing it in your nodes and after processing it through spark you can write it back to S3 and terminate EMR.</p><p>After EMR instance is ready you can go to your terminal and ssh into it by using your pem key and public DNS of the master.</p><pre>$ ssh -i .ssh/mykey.pem hadoop@ec2–xx–xxx–xxx-xx.us-west-2.compute.amazonaws.com</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*KKMUtETa-qLUHxJBuN3l2w.png" /><figcaption>EMR terminal</figcaption></figure><h3>2. Getting the Data</h3><p>In this tutorial I will be working with a subset of MillionSongs dataset but you any dataset over couple of GB would be support our point. Here is the link to get the data in many ways: <a href="https://labrosa.ee.columbia.edu/millionsong/pages/getting-dataset">https://labrosa.ee.columbia.edu/millionsong/pages/getting-dataset</a>. Even though they provide this dataset in an EC2 image, for completeness we will pretend that this is our own data that we try to import into AWS. Data we will be exploring here is called MillionSongSubset which has randomly selected subset of million songs, a collection of 10,000 songs. You can refer to “MILLION SONG SUBSET” link and download the tar.gz file from there. You can download the data to your local then scp to both master and worker nodes but this will take forever to scp. So I would recommend here to use a curlwget extension through Chrome and download tar.gz file directly to master and worker nodes. The extension that I am using on Chrome is called CurlWget. Try to store data /mnt/ in your nodes since it may take a quite big space and don’t forget to store it under the same path for each node.</p><p>Once you’ve downloaded data you will see there are two main folders data/A/ and data/B/. These folders have sub directories which eventually have single song data under each .h5 hierarchical data file, more information about data is provided in getting-the-dataset url.</p><h3>3. Converting H5 into Spark RDD with Pyspark</h3><p>So far we’ve launched our EMR instance and get the data into same path for all nodes, now we will convert data into Spark RDD in order to use pyspark and it’s distributed computing functionalities. RDD (Resilient Distributed Dataset) is the way that spark represents data and stores it in partitions. Data can come from either text, csv, json format or a database.</p><p>I couldn’t find a direct way to convert H5 files into an RDD but this link <a href="https://www.hdfgroup.org/2015/03/from-hdf5-datasets-to-apache-spark-rdds/">https://www.hdfgroup.org/2015/03/from-hdf5-datasets-to-apache-spark-rdds/</a> inspired me to write my own script to do this conversion in parallel using pyspark. The basic idea is to collect all h5 filepahts in a python list and create an RDD out of this list. Then we can easily do the conversion we want using map function on this RDD.</p><pre># 8 is number of partitions to distribute the data<br># For local applications it will be distributed to 8 threads</pre><pre>file_paths = sc.parallelize(h5_file_paths, 8)</pre><pre># Read and convert to python dict in parallel</pre><pre>songs_rdd = file_paths.map(lambda x: h5todict(x))<br>songs = songs_rdd.collect()</pre><p>Here we also collect songs as a python list in order to create our MongoDB database, since a part of the tutorial is to show MongoDB as well but we might just stick with our RDD which contains python dicts (in this case each element is a python dictionary which has a single song data in a hierarchical relationship). Having RDD is enough to continue processing, analysis and modeling via pyspark.</p><p><strong>Side note</strong>: The full script for conversion is provided here:<a href="https://github.com/KeremTurgutlu/distcomputing/blob/master/MillionSongs-Project/h5toMongo.py">https://github.com/KeremTurgutlu/distcomputing/blob/master/MillionSongs-Project/h5toMongo.py</a>. All nodes should have the required python packages installed such as h5py, json, numpy…</p><p>For this you need to ssh into your worker nodes as well as your master node. To ssh into worker nodes, you should have your pem key in master and ssh with that through master terminal. After that all you need is pip install the required packages which are not already provided.</p><h3>4. Setting up MongoDB</h3><p>In this step we will install and setup mongodb in our master node. You can follow the steps here <a href="https://docs.mongodb.com/manual/tutorial/install-mongodb-on-red-hat/">https://docs.mongodb.com/manual/tutorial/install-mongodb-on-red-hat/</a> for installation. After installation you should create a directory for the databases you create to be stored in. Again, creating them under /mnt as /mnt/mongo/db is better for disk usage. Also you will need to set permission to this directory, by running the command:</p><pre>$ sudo chown -R `id -un` /mnt/mongo/db/</pre><p>After this step run:</p><pre>$ mongod --dbpath /mnt/mongo/db</pre><p>Now MongoDB server should be listening to connections. On a separate terminal you can run <strong>mongo</strong> and start the mongo shell. In mongo shell you may view current databases with <strong>show dbs.</strong></p><p>Now, having an alive connection to MongoDB. Next, we need to install pymongo package which is the python API for MongoDB into all nodes. After that we may run the script provided through github repo on our master. For that script to run we should also have <a href="https://github.com/KeremTurgutlu/distcomputing/blob/master/MillionSongs-Project/user_definition.py">https://github.com/KeremTurgutlu/distcomputing/blob/master/MillionSongs-Project/user_definition.py</a> in the same path. In user definition we define executor-driver memories , max result and overhead memory in order not run into memory issues while running our application.</p><p>This script will read h5 files and create a mongoDB collections called ‘songs’ under database called ‘MillionSongs’ by default (you can of course specify other names for database and collection in user_definition.py). You may think of database as a regular SQL database, collection as a SQL table and document(dict of single song) as a row in the table here.</p><h3>5. Distributed Analysis AND ML</h3><p>At this point we have a MongoDB database named “MillionSongs” and under a collection named “songs” (or the database and collection you’ve specified). For your own data in order to have a database and collection for example from a json file that sits under your current directory, you may run this command from another terminal while mongod running in another terminal:</p><pre>$ mongoimport --db dbName --collection collectionName --file fileName.json</pre><p>With mongodb it’s very easy to filter, select, aggregate and to do many other operation by querying the data as we were doing in SQL. In our songs collection each song is represented as a document(a python dict) due to the way we’ve create and inserted them through h5tomongo.py script.</p><p>Our first task will be to predict what decade a particular song was released in. For more detailed explanations on data fields you may find this link helpful: <a href="https://labrosa.ee.columbia.edu/millionsong/pages/field-list">https://labrosa.ee.columbia.edu/millionsong/pages/field-list</a>. This task is promoted both in Kaggle and UCI websites. We will be using segments_timbre which is an MFCC-PCA like 2d feature about song’s texture, songs energy, songs danceability and songs year to complete our task.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/1*312K1gj2rACkifoos05B3A.png" /><figcaption>2d features for 12 segments</figcaption></figure><p><strong>First, let’s collect the data we want from mongodb:</strong></p><p>We define the fields that we want to be returned by defining a dictionary which have the hierarchical field names (for example year has the analysis -&gt; songs -&gt; year hierarchy in our documents), 1 for the ones that we want to be returned and 0 for the ones we don’t want. By default _id of the document is also returned that’s why we explicitly deselected . You can see that we define a filter inside find function where we create train_year_data and test_year_data. Test data will be the data that has year field missing (which is imputed as 0 in the original data). Our task may be to build a good prediction model with training data in order to further impute the missing years in this test set.</p><pre>from pymongo import MongoClient</pre><pre>client = MongoClient()<br>db = client[dbname]<br>collection = db[collection_name]</pre><pre>fields = {&#39;musicbrainz.songs.year&#39;:1,                    &#39;analysis.segments_timbre&#39;:1,                    &#39;analysis.songs.energy&#39;:1,                    &#39;analysis.songs.danceability&#39;:1,                    &#39;metadata.songs.title&#39;:1,                    &#39;metadata.songs.artist_name&#39;: 1,<br>&#39;_id&#39;:0}</pre><pre>train_year_data = collection.find({&#39;musicbrainz.songs.year&#39;:{&#39;$gte&#39;:1920}}, fields)<br>test_year_data = collection.find({&#39;musicbrainz.songs.year&#39;:{&#39;$eq&#39;:0}}, fields)</pre><p><strong>Second, let’s create our RDDs and Dataframes:</strong></p><p>Spark SQL supports dataframes which we will be using for feeding into machine learning algorithms, it also has a very convenient support with pandas dataframes. Transition from MongoDB collection to Spark RDD is as easy as:</p><pre># create train and test rdds<br>train_year_rdd = sc.parallelize(list(train_year_data))<br>test_year_rdd = sc.parallelize(list(test_year_data))</pre><p>We will create a 1d feature vector out of timbre_segments by taking the covariances over all segments and means of each segments. Adding up to total of 90 features for 12 segments. We will apply two custom functions through map in order to get the features we want to create our Spark SQL dataframes. Because train_year_rdd and test_year_rdd are still in hierarchical python dict format that we can’t use to create dataframes to feed into ML algorihtms. You can find the source code for these steps and further here: <a href="https://github.com/KeremTurgutlu/distcomputing/blob/master/MillionSongs-Project/predict_year.py">https://github.com/KeremTurgutlu/distcomputing/blob/master/MillionSongs-Project/predict_year.py</a></p><pre># create train and test dataframes</pre><pre>train_full_df = train_year_rdd.map(lambda d: getYearFeatures(d)).\<br>                               map(lambda d: getDFeatures(d)).toDF()</pre><pre>test_df = test_year_rdd.map(lambda d: getYearFeatures(d)).\<br>                        map(lambda d: getDFeatures(d, True)).toDF()</pre><p>Create a training and validation set:</p><pre># split to train and validation dataframes<br>train_df, val_df = train_full_df.randomSplit([0.7, 0.3])</pre><p>Create vector assembler:</p><p>Vector assembler take a list of desired column names that will be feed into ML algorithm, combine them and get them ready for optimized spark ML.</p><pre># define vector assembler and transform dataframes into correct format<br># cache train and validation dataframes for fast iterations</pre><pre>va = VectorAssembler(inputCols=x_cols, outputCol=&#39;features&#39;)<br>train_va=va.transform(train_df).select(&#39;features&#39;,&#39;decade&#39;).cache()<br>val_va=va.transform(val_df).select(&#39;features&#39;, &#39;decade&#39;).cache()<br>test_va=va.transform(test_df).select(&#39;features&#39;, &#39;decade&#39;).cache()</pre><p>Pyspark ML package provides a high variety of models, transformers and many options. It’s good to check out the documentation: <a href="http://spark.apache.org/docs/2.2.0/api/python/pyspark.ml.html">http://spark.apache.org/docs/2.2.0/api/python/pyspark.ml.html</a> for personal applications. Almost every distributed ML algorithm is provided through this API but not deep learning. For deeplearning you can checkout H20.ai’s sparkling water: <a href="https://www.h2o.ai/sparkling-water/">https://www.h2o.ai/sparkling-water/</a> or simply use a modern GPU, or even multiple with Pytorch’s nn.DataParallel.</p><p>Since we have our vector assemblers now we can do actual distributed training. I will show the code to run a logistic regression model but script provided in the repo runs both logistic regression and random forest.</p><pre># define model and fit </pre><pre>lr = LogisticRegression(featuresCol=&#39;features&#39;, labelCol=&#39;decade&#39;,regParam=0.01, maxIter=1000, fitIntercept=True)<br>lr_model = lr.fit(train_va)</pre><p>or we can simply do a grid search with cross-validating</p><pre># create cross validation<br>cv = CrossValidator().setEstimator(lr).setEvaluator(evaluator).setNumFolds(5)<br>#ParamGridBuilder() – combinations of parameters and their values.<br>paramGrid = ParamGridBuilder().addGrid(lr.maxIter, [1000]).addGrid(lr.regParam, [0.0001, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5]).build()<br>#setEstimatorParamMaps() takes ParamGridBuilder().<br>cv.setEstimatorParamMaps(paramGrid)<br>cvmodel = cv.fit(train_va)</pre><p>We evaluate the model’s by using multi-class F1 score since accuracy wouldn’t be good enough.</p><p><strong>For running the scripts provided:</strong></p><p>In order to run spark jobs through terminal you need to use spark-submit command. After cloning or getting the scripts you want into a folder and defining the right file_roots in h5toMongo.py and desired memory allocations defined in user_definition you may run:</p><pre>$ spark-submit h5toMongo.py</pre><p>and</p><pre>$ spark-submit predict_year.py</pre><p>I hope that you will find this brief tutorial helpful, there is a lot to cover and to learn through documentation and exprience. So please let me know if you have any suggestions, recommendations or questions.</p><p>Thanks for reading !</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d1077f68381" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/end-to-end-distributed-ml-using-aws-emr-apache-spark-pyspark-and-mongodb-tutorial-with-4d1077f68381">End-to-end Distributed ML using AWS EMR, Apache Spark (Pyspark) and MongoDB Tutorial with…</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Structured Deep Learning]]></title>
            <link>https://medium.com/data-science/structured-deep-learning-b8ca4138b848?source=rss-25e7c0e46c50------2</link>
            <guid isPermaLink="false">https://medium.com/p/b8ca4138b848</guid>
            <category><![CDATA[structured-data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[kaggle]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Kerem Turgutlu]]></dc:creator>
            <pubDate>Fri, 24 Nov 2017 20:35:08 GMT</pubDate>
            <atom:updated>2017-11-26T13:58:27.203Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2l2c8AKpzS8CgOGXellFSA.jpeg" /></figure><ul><li>Fast</li><li>No Domain Knowledge Requiring</li><li>High Performing</li></ul><p>This blog will mainly focus on a not very widely known application area of deep learning, structured data.</p><p>In machine learning/deep learning or any kind of predictive modeling task data comes before the algorithm/methodology. This is the main reason why machine learning requires a lot of feature engineering before certain tasks such as image classification, NLP, and many other “unusual” data that can’t be directly fed into a logistic regression or a random forest model. In contrary, these type of tasks are done significantly well by deep learning without any requirement of nasty and time consuming feature engineering. Most of the time these features require domain knowledge, creativity and a lot of trial and error. Of course domain expertise and clever feature engineering are still very valuable but the techniques I will be mentioning throughout this post will be enough for you to aim for top 3 in a Kaggle competition (<a href="http://blog.kaggle.com/2016/01/22/rossmann-store-sales-winners-interview-3rd-place-cheng-gui/">http://blog.kaggle.com/2016/01/22/rossmann-store-sales-winners-interview-3rd-place-cheng-gui/</a>) without any prior knowledge on the subject.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/284/1*PNfywqNs7q2jMRrgf1zm2g.jpeg" /><figcaption>Fig 1. a cute dog and an angry cat (labeled by me)</figcaption></figure><p>Due to complex nature and ability of feature generation (e.g. convolutional layers of CNNs), deep learning is abundantly applied to various kinds of image, textual and audio data problems. These are without a doubt very important problems for advancement of AI, and there must be a very good reason why every year top researchers in the field compete to classify cats, dogs, and ships better than the previous year. But these are rarely the cases we see in industry. Companies work with databases involving structured datasets, and these are the domains that shape every day lives.</p><p>Let’s define structured data to be more clear for the rest of this post. Here you can think of rows as collected individual data points or observations and columns as fields that represents a single attribute of each observation. For example data from an online retail store might have rows as sales made by customers and columns as item bought, quantity, price, time stamp, and so on ….</p><p>Below we have online seller data, rows as each unique sales and columns describing that particular sales.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RfE1wmdUHFlyiyAmsPHc1A.png" /><figcaption>Fig 2. pandas dataframe an example to structured data</figcaption></figure><p>Let’s talk about how we can leverage neural networks for structured data task. Actually at a theoretical level it’s very trivial to create a fully connected network with any desired architecture, then using “rows” as inputs. Given the loss function after a couple of dot products and backpropagation we would end up having a trained network which then can be used to make predictions.</p><p>Even though as it seems very straightforward there are major reasons why people prefer tree based methods over neural networks when it comes to structured data. This can be understand from the algorithm’s point of view, by exploring how actually algorithms see and treat our data.</p><p>The main separation in terms of having structured and unstructured data is that with unstructured data even though it’s “unusual”, we often deal with single entities with single units like pixels, voxels, audio frequencies, radar back-scatters, sensor measurements… so on so forth. In contrary, with structured data we often need to deal with many different type of data under two main groups; numerical and categorical. Categorical data requires processing prior to training, because most of the algorithms as well as neural networks can’t handle them directly yet.</p><p>There are various options available for encoding variables such as label/numerical encoding and one-hot encoding. But these techniques are problematic in terms of memory and real representation of categorical levels. The former property is probably more obvious and can be illustrated with an example.</p><p>Let’s say we have day of week information as a column. If we one-hot encode or arbitrarily label encode this variable we would assume equal and arbitrary distance/difference among levels respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZsYkXEa1qrKeGiplNnFHyQ.jpeg" /><figcaption>Fig 3. one-hot encoding and label encoding</figcaption></figure><p>But both of these two approaches assume that the difference between each pair of days are equal but in reality we easily know that this is not true, so should our algorithm!</p><blockquote>“The continuous nature of neural networks limits their applicability<br>to categorical variables. Therefore, naively applying<br>neural networks on structured data with integer<br>representation for category variables does not work well” [1]</blockquote><p>Tree algorithms do not require any assumptions on continuity of categorical variables since they can find states by splitting as needed, but with neural networks this is not the case. Here comes the entity embedding to help. Entity embedding is used to map discrete values into multi-dimensional space where values with similar function outputs are closer to each other. You may think of if we were embedding states in a country for a sales problem, similar states in terms of sales would be closer to each in this projected space.</p><p>Since we don’t want to make arbitrary assumptions between the levels of our categorical variables, we will learn a better representation of each in Euclidean space. This representation will be equal to nothing but the dot product of one-hot encoded data and learn-able weights.</p><p>Embeddings are very widely used in NLP, as each word is represented as a vector. Two famous embedding examples are <a href="https://nlp.stanford.edu/projects/glove/">Glove</a> and <a href="https://en.wikipedia.org/wiki/Word2vec">word2vec</a>. We can see how powerful embeddings are from the figure 4 [2]. These vectors are readily available for you to download them and use them as it fits your goal, which is actually very cool thinking of the information they hold.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sXNXYfAqfLUeiDXPCo130w.png" /><figcaption>Fig 4. Word2Vec from tensorflow tutorials</figcaption></figure><p>Even though embeddings can be applied to different context either with supervised or unsupervised manner, our main objective is to understand how to do these projections for categorical variables.</p><h3>Entity Embeddings</h3><p>Even though entity embeddings have a different name, they are not much different then the use case we have seen in word embeddings. After all, the only thing we care is to have higher dimensional vector representation of our for our grouped data; this may be words, day of weeks, countries and many others you can imagine. This transaction from word embeddings to meta(categorical in our case) data embeddings enabled Yoshua Bengio et al. to win a Kaggle competition(<a href="https://www.kaggle.com/c/pkdd-15-predict-taxi-service-trajectory-i">https://www.kaggle.com/c/pkdd-15-predict-taxi-service-trajectory-i</a>) back in 2015 with a single simple automated approach, which is not the usual case for winning solutions.</p><blockquote>“To deal with the discrete meta-data, consisting of client ID, taxi ID, date and time information, we learn embeddings jointly with the model for each of these information. This is inspired by neural language modeling approaches [2], in which each word is mapped to a vector space of fixed size (the vector is called the embedding of the word).”[3]</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*WFydi5x5kkFgjc1wSRIEYQ.png" /><figcaption>Fig 5. Taxi Metadata embeddings visualization with t-SNE 2D projections</figcaption></figure><p>Step by step we will explore how to learn these features within a neural network. Define a fully connected neural network, separate numerical and categorical variables.</p><p>For each categorical variable:</p><ol><li>Initialize a random embedding matrix as m x D.</li></ol><p><strong><em>m: </em></strong><em>number of unique levels of categorical variable (Monday, Tuesday, …)</em></p><p><strong><em>D: </em></strong><em>desired dimension for representation, a hyperparameter which can be between 1 and m-1 (if 1 then it will be label encoding, if m it will be one-hot encoding)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/307/1*wf-YxfeOhwWI7ekorRVKog.png" /><figcaption>Fig 6. Embedding Matrix</figcaption></figure><p>2. Then for each forward pass through neural network we do a lookup for the given level (e.g Monday for “dow”) from the embedding matrix, which will give us a vector as 1 x D.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/341/1*JSX1eeJrfS-k9TNuarqrnQ.png" /><figcaption>Fig 7, Embedding Vector after lookup</figcaption></figure><p>3. Append this 1 x D vector to our input vector (numerical vector). Think this process as augmenting a matrix, where we add an embedding vector for each category that been embedded by doing lookup for each particular row.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/719/1*Qa5Oh3-BXfJHsxED-aFIEA.png" /><figcaption>Fig 8. After adding embedding vectors</figcaption></figure><p>4. While doing backpropagation we are also updating these embedding vectors in the way of gradient which minimizes our loss function.</p><p><em>Usually inputs are not updated but for embedding matrices we have this special case where we allow our gradient to flow all the way back to these mapped features and hence optimize them.</em></p><p>We can think this as a process, which allows our categorical embeddings to be better represented at every iteration.</p><p><strong>Note:</strong> Rule of thumb is to keep categories that don’t have very high cardinality. As in if a variable has unique levels for 90% of the observations then it wouldn’t be a very predictive variable and we may very well get rid of it.</p><h3><strong>Good News</strong></h3><p>We can very well implement above mentioned architecture in our favorite framework (preferably a dynamic one) by doing lookup and allowing requires_grad=True in our embedding vectors and learning them. But all these steps and more is already done in Fast.ai. Beyond making structural deep learning easy this library also provides many state-of-art features like differential learning rates, SGDR, cyclical learning rate, learning rate finder and so on. These are all the things we would like to take advantage of. You can read further about these topics from these very cool other blog posts:</p><p><a href="https://medium.com/@bushaev/improving-the-way-we-work-with-learning-rate-5e99554f163b">https://medium.com/@bushaev/improving-the-way-we-work-with-learning-rate-5e99554f163b</a></p><p><a href="https://medium.com/@surmenok/estimating-optimal-learning-rate-for-a-deep-neural-network-ce32f2556ce0">https://medium.com/@surmenok/estimating-optimal-learning-rate-for-a-deep-neural-network-ce32f2556ce0</a></p><p><a href="https://medium.com/@markkhoffmann/exploring-stochastic-gradient-descent-with-restarts-sgdr-fa206c38a74e">https://medium.com/@markkhoffmann/exploring-stochastic-gradient-descent-with-restarts-sgdr-fa206c38a74e</a></p><p><a href="http://teleported.in/posts/cyclic-learning-rate/">Teleported.in</a></p><h3><strong>Walk through With Fast.ai</strong></h3><p>In this part we will take a look at how we can bypass all these steps mentioned and build a neural network that is more effective with structured data.</p><p>For this purpose we will take a look at an active Kaggle competition <a href="https://www.kaggle.com/c/mercari-price-suggestion-challenge/l">https://www.kaggle.com/c/mercari-price-suggestion-challenge/</a>. In this challenge we are trying to predict the price of an item sold by an online seller. This is a very suitable example to entity embeddings since data is mostly categorical with relatively high cardinality (not too much) and there isn’t much else.</p><p><strong>Data:</strong></p><p>~1.4 M rows</p><ul><li><strong>item_condition_id</strong>: condition of the item (cardinality: 5)</li><li><strong>category_name: </strong>category name (cardinality: 1287)</li><li><strong>brand_name</strong>: name of the brand (cardinality: 4809)</li><li><strong>shipping:</strong> whether shipping is included in price (cardinality 2)</li></ul><p><strong>Important Note: </strong>I won’t be having a validation set in this example since I’ve already find my best model parameters but you should always do hyperparameter tuning with a validation set.</p><p><strong>Step 1:</strong></p><p>Fill missing values as a level, since missing-ness itself is an important information.</p><pre>train.category_name = train.category_name.fillna(&#39;missing&#39;).astype(&#39;category&#39;)<br>train.brand_name = train.brand_name.fillna(&#39;missing&#39;).astype(&#39;category&#39;)<br>train.item_condition_id = train.item_condition_id.astype(&#39;category&#39;)</pre><pre>test.category_name = test.category_name.fillna(&#39;missing&#39;).astype(&#39;category&#39;)<br>test.brand_name = test.brand_name.fillna(&#39;missing&#39;).astype(&#39;category&#39;)<br>test.item_condition_id = test.item_condition_id.astype(&#39;category&#39;)</pre><p><strong>Step 2:</strong></p><p>Preprocess data, scale for numerical columns since neural networks like normalized data or in other words equally scaled data. If you don’t scale your data one feature may be over emphasized by the network since it’s all about dot products and gradients. It would be better if we were scaling both training and test by training statistics, but this shouldn’t matter much. Think of dividing each pixel by 255, same logic.</p><p>I’ve combined training and test data since we want same levels to have same encodings.</p><pre>combined_x, combined_y, nas, _ = proc_df(combined, &#39;price&#39;, do_scale=True)</pre><p><strong>Step 3:</strong></p><p>Create model data object. Path is where Fast.ai stores models and activations.</p><pre>path = &#39;../data/&#39;<br>md = ColumnarModelData.from_data_frame(path, test_idx, combined_x, combined_y, cat_flds=cats, bs= 128</pre><p><strong>Step 4:</strong></p><p>Decide D (dimension of embedding). cat_sz is a list of tuples (col_name, cardinality + 1) for each categorical columns.</p><pre># We said that D (dimension of embedding) is an hyperparameter<br># But here is Jeremy Howard&#39;s rule of thumb<br>emb_szs = [(c, min(50, (c+1)//2)) for _,c in cat_sz]</pre><pre># [(6, 3), (1312, 50), (5291, 50), (3, 2)]</pre><p><strong>Step 5:</strong></p><p>Create a learner, this is the core object of Fast.ai library.</p><pre># params: embedding sizes, number of numerical cols, embedding dropout, output, layer sizes, layer dropouts<br>m = md.get_learner(emb_szs, len(combined_x.columns)-len(cats),<br>                   0.04, 1, [1000,500], [0.001,0.01], y_range=y_range)</pre><p><strong>Step 6:</strong></p><p>This part is explained in much more detail in other posts that I’ve mentioned before.</p><p>Take advantage of full Fast.ai joy.</p><p>We are picking our learning rate to be from a point before where loss starts increasing…</p><pre># find best lr<br>m.lr_find()</pre><pre># find best lr<br>m.sched.plot()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*iJHmLEDgX1spH-VLspkJ3g.png" /><figcaption>Fig 9. learning rate loss plot</figcaption></figure><p>Fit</p><pre>we can see that with just 3 epochs we have<br>lr = 0.0001<br>m.fit(lr, 3, metrics=[lrmse])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/369/1*bEMEtx2ZqVIUb-26jVL76A.png" /></figure><p>Fit more</p><pre>m.fit(lr, 3, metrics=[lrmse], cycle_len=1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/378/1*f4QSCXJoq5y3CEjA_NDJXw.png" /></figure><p>And some more…</p><pre>m.fit(lr, 2, metrics=[lrmse], cycle_len=1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/384/1*X2EupuRhAh9oND53kJNYjQ.png" /></figure><p>So, these brief but effective steps can take you to top ~10% without any further need within minutes. If you really want to aim high, I would suggest you to exploit item_description column and make it multiple categorical variables. Then leave the job to entity embeddings, of course don’t forget to stack and ensemble :)</p><p>This was my very first blog post I hope you enjoyed it! I must admit this thing is kind of addictive, so I might come back very soon…</p><p>Appreciate all the claps you have for me :)</p><p><strong>Bio:</strong> I am a current graduate student in USF, Master’s of Analytics program. I’ve been applying machine learning for 3 years now and currently practicing deep learning with Fast.ai.</p><p><strong>Linkedin:</strong> <a href="https://www.linkedin.com/in/kerem-turgutlu-12906b65/en">https://www.linkedin.com/in/kerem-turgutlu-12906b65/en</a></p><p><strong>References:</strong></p><p>[1] Cheng Guo, Felix Berkhahn (2016, April, 22) <em>Entity Embeddings of Categorical Variables. Retrieved from </em><a href="https://arxiv.org/abs/1604.06737">https://arxiv.org/abs/1604.06737</a>.</p><p>[2] TensorFlow Tutorials: <a href="https://www.tensorflow.org/tutorials/word2vec">https://www.tensorflow.org/tutorials/word2vec</a></p><p>[3] Yoshua Bengio, et al.<em> Artificial Neural Networks Applied to Taxi Destination Prediction. Retrieved from </em><a href="https://arxiv.org/pdf/1508.00021.pdf">https://arxiv.org/pdf/1508.00021.pdf</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b8ca4138b848" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/structured-deep-learning-b8ca4138b848">Structured Deep Learning</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>