<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Udacity PyTorch Challengers - Medium]]></title>
        <description><![CDATA[Articles and tutorials written by and for PyTorch students with a beginner’s perspective. Together we learn. - Medium]]></description>
        <link>https://medium.com/udacity-pytorch-challengers?source=rss----c6e264a51486---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Udacity PyTorch Challengers - Medium</title>
            <link>https://medium.com/udacity-pytorch-challengers?source=rss----c6e264a51486---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 12:31:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/udacity-pytorch-challengers" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Trials, errors and trade-offs in my deep learning model]]></title>
            <link>https://medium.com/udacity-pytorch-challengers/trials-errors-and-trade-offs-in-my-deep-learning-model-d2e0c3d51794?source=rss----c6e264a51486---4</link>
            <guid isPermaLink="false">https://medium.com/p/d2e0c3d51794</guid>
            <category><![CDATA[pytorchudacityscholar]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Hong Min]]></dc:creator>
            <pubDate>Fri, 18 Jan 2019 16:22:53 GMT</pubDate>
            <atom:updated>2019-01-18T16:22:53.249Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>In this post,<br>I will explain ‘Bias-Variance Tradeoff’, ‘Regularization’, and ‘Learning rate decay’ in short. and tell you about my <strong>trial and errors for better performance of my deep learning model,</strong> including the reason of each ones and codes written by pytorch. I trained Classifier with Oxford flower data for final lab challenge of<strong> Pytorch Scholaship Challenge from facebook</strong>.</blockquote><blockquote><em>I wish that it would be helpful when you start to train your first model or struggle with your model to improve it!</em></blockquote><h3><a href="https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff">Bias-Variance Tradeoff</a></h3><p>Let’s start with the meaning of bias and variance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*JZbxrdzabrT33Yl-LrmShw.png" /><figcaption>high bias — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — high variance</figcaption></figure><p>The <strong>bias</strong> is an error from wrong assumptions. High bias can cause the model to miss relevant relations between features and target. Shortly, high bias can cause when the model is <strong>underfitted</strong>.</p><p>The <strong>variance</strong> is an error from sensitivity to small fluctuations in the training set. The high variance is related to <strong>overfitting</strong>. overfitting happens with the highly flexible model that fitted on random noise in training dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*7G9MBbN7g0J9Tq5u.png" /><figcaption><a href="https://en.wikipedia.org/wiki/Overfitting">overfitting</a> after yellow sign, blue line = training loss &amp; red line = validation loss</figcaption></figure><p>What we want to get is low bias and low variance.</p><p>But we have <strong>bias-variance tradeoff</strong>, it is whereby models with lower bias have a higher variance across samples, and vice versa. <br>take a look graphic that represents overfitting. The model almost perfectly fitted on the training data (higher variance), but not in the validation data. So the model is not generalized well (lower bias).</p><p><strong>Variance getting lower</strong> with <br>bigger training data (data augmentation is one of way to increase number of data), dimension reduction and feature selection (you can do regularization)</p><p><strong>Bias getting lower</strong> as features are added.</p><h3><a href="https://en.wikipedia.org/wiki/Regularization_(mathematics)">Regularization methods</a></h3><p>regularization is the process of adding information from weights in order to prevent overfitting, improving generalization.</p><blockquote><strong>Early stopping</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/453/0*p-T-SNCubiv6It3c.png" /><figcaption>where to stop?</figcaption></figure><p><strong>: </strong>A training procedure like gradient descent will tend to learn more and more complex functions as the number of iterations increases. By regularizing on time, the complexity of the model can be controlled. In practice, just terminate the training when the error repeatedly increases based on N number of time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/0*LjNKwcGSR33yb7kh.png" /><figcaption>L1 regularization — — — L2 regularization</figcaption></figure><blockquote><strong>L1 regularization</strong></blockquote><p><strong>: </strong>Penalize loss function with L1 norm(sum of absolute values of weights).<br> It reduces unimportant feature first. So, it has an effect on variable selection.</p><pre>&#39;&#39;&#39;Creates a criterion that measures the mean absolute error (MAE) with L1 regularization.&#39;&#39;&#39;</pre><pre>The loss can be described as:<br>criterion = <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.L1Loss">torch.nn.L1Loss()</a></pre><blockquote><strong>L2 regularization</strong></blockquote><p><strong>:</strong> Penalize loss function with square of L2 norm(sum of square of weights).<br> It reduces outliers first. Improve the generalization ability of the linear model.</p><pre>&#39;&#39;&#39; Creates a criterion that measures the mean squared error (squared L2 norm)&#39;&#39;&#39;<br>criterion = <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.MSELoss">torch.nn.MSELoss()</a></pre><blockquote><a href="http://jmlr.org/papers/volume15/srivastava14a.old/srivastava14a.pdf"><strong>Drop Out</strong></a></blockquote><figure><img alt="Dropoutì ëí ì´ë¯¸ì§ ê²ìê²°ê³¼" src="https://cdn-images-1.medium.com/proxy/1*iWQzxhVlvadk6VAJjsgXgg.png" /></figure><p>: Deactivate random units during training.<br>In practice, add it after layer that you want to regularize.</p><pre>dropout1 = torch.nn.Dropout(<em>p=0.5</em>) <br># p is a probability of an element to be zeroed.</pre><blockquote><a href="https://arxiv.org/abs/1502.03167"><strong>Batch normalization</strong></a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*rXY5zJrDdHv6EdKhJvKqcA.png" /></figure><p>: Normalize output of activation at every epoch.<br>In practice, add it after layer that you want to regularize.</p><pre>#  Applies Batch Normalization over a 2D or 3D input<br>bn_1d = <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.BatchNorm1d">nn.BatchNorm1d(num_features)</a></pre><pre># Applies Batch Normalization over a 4D input<br>bn_2d = <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.BatchNorm2d">nn.BatchNorm2d(num_features)</a></pre><pre>#  Applies Batch Normalization over a 5D input<br>bn_3d = <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.BatchNorm3d">nn.BatchNorm3d(num_features)</a></pre><h3><a href="https://ko.coursera.org/lecture/deep-neural-network/learning-rate-decay-hjgIA">Learning rate decay</a></h3><p>If the learning rate is small, It is too slow to converge even it couldn’t converge. <br>If the learning rate is too big, the optimizer overshoots and loss getting worse.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*EP8stDFdu_OxZFGimCZRtQ.jpeg" /></figure><p><strong>Slowly reduce learning rate over time</strong> is called learning rate decay. It is helpful when your algorithm tends to towards optimal minimum, but end up wandering around the optimal and never exactly converge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/384/0*81W1UffAu_2dPFNX.png" /><figcaption>learning rate decay functions for <a href="https://www.tensorflow.org/api_docs/python/tf/train/linear_cosine_decay">linear cosine decay</a>, stepwise decay, and <a href="https://arxiv.org/abs/1608.03983">cosine decay</a>. <a href="https://ai.googleblog.com/2018/03/using-machine-learning-to-discover.html">(from google blog)</a></figcaption></figure><h3>Trial and Errors</h3><ul><li>I worked on <a href="http://www.robots.ox.ac.uk/~vgg/data/flowers/">flower dataset from Oxford visual geometry group</a> with 102 different species. (means 102 different targets)</li><li>There are 6552 number of training data and 818 number of validation data.</li><li>I checked training loss and validation loss during training model.</li><li>You can get my whole code for training <a href="https://github.com/minh364/Note_for_MachineLearning/blob/master/DeepLearning/Train_image_classifier_with_pretrained_model.ipynb">here</a>.</li></ul><p>1. To build an image classifier, we usually use pretrained models. You can load a pretrained network <a href="http://pytorch.org/docs/master/torchvision/models.html">here</a>. I started training with pretrained densenet by image net dataset.</p><pre>from torchvision import models<br>model = models.densenet121(pretrained= True)</pre><p>2. I froze parameters from the feature extraction part(structured by convolution layers) of the model.</p><pre>for param in model.parameters():<br>    param.requires_grad = False</pre><p>And <strong>newly define an architecture of classifier</strong> part and trained it.<br>To determined architecture, I trained around 10 epochs with some architecture as experience. <br>Just I expected, the result was quite bad when classifier structured by only one fully connected layer. it is because it cannot get any non-linearity without activation. so connected 2 fully connected layers with activation function.</p><pre>classifier = nn.Sequential(OrderedDict([<br>                          (&#39;fc1&#39;, nn.Linear(1024, 512)),<br>                          (&#39;Relu&#39;, nn.ReLU()),<br>                          (&#39;dropout&#39;, nn.Dropout(0.3)),<br>                          (&#39;fc2&#39;, nn.Linear(512,101))<br>                          ]))<br>    <br>model.classifier = classifier</pre><p>3. Training loss and validation loss repeatedly decreased, but it is slow. I thought it is close to <strong>underfitting</strong>.</p><p>3–1 Adjust rate of dropout and even remove the dropout layer. so I tried things below. performance has improved marginally.<br>3–2 Maintain Dropout layer and try <strong>learning rate decay</strong>.<br>I applied step function that multiplies gamma 0.1 at every 10 epoch. then I set the learning rate higher than before. I expected to converge quickly to the optimal minimum with large steps on the beginning and small steps on the last. the result, it was a little faster than before on the very beginning of the training, but I faced where the loss no longer falls soon.</p><pre>optimizer = optim.Adam(model.classifier.parameters(), lr=0.01)</pre><pre>scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=10)</pre><pre>```don’t forget you need to use <strong>scheduler.step()</strong> on your training code, not an <strong>optimizer.step()</strong>.```</pre><p>4. the learning rate I tried on 3–2 wasn’t overshooting like yellow line, but it was still high like green one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/459/0*a84BMgTmoBCCg3MJ.jpg" /><figcaption>when your learning rate is …</figcaption></figure><p>So I reduce the learning rate and use an <a href="https://openreview.net/pdf?id=ryQu7f-RZ"><strong>amsgrad</strong></a><strong>,</strong> advanced method from Adam optimizer. It uses long-term memory of past gradients when optimizing.</p><pre>optimizer = optim.Adam(model.classifier.parameters(), lr=0.0001, amsgrad=True)</pre><p>Then, I got above 90% accuracy at 11 epochs. After 11 epochs it tends to keep drop, but the decreasing rate of loss was reduced and vibration was increased.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*eoIhOV1atdnu-XJll37ogw.png" /><figcaption>My model was saved on epoch 26, with its best validation accuracy 94.99%.</figcaption></figure><p>4. Finally, I unfroze feature extractor part!</p><pre>for param in model.parameters():<br>    param.requires_grad = True</pre><p>after unfreezing, 1 epoch of training takes time far more than before. It is because parameters for training are increased.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/745/1*veYfyBlBiMKP2Gaz6hUecw.png" /><figcaption>around 95% at epoch 41.</figcaption></figure><p>Anyway, loss started to decrease again. But the model was not trained well after epoch 41. So I stopped the training.</p><blockquote>But, It is true. Still, there are way to improve performance of model.<br>There are some resource for further.</blockquote><p><strong>General</strong></p><ul><li><a href="http://lamda.nju.edu.cn/weixs/project/CNNTricks/CNNTricks.html">Must Know Tips/Tricks in Deep Neural Networks</a> (by <a href="http://lamda.nju.edu.cn/weixs/">Xiu-Shen Wei</a>)</li><li><a href="/service/ftp://ftp.sas.com/pub/neural/FAQ.html">Neural Network FAQ</a></li></ul><p><strong>Hyperparameter optimization</strong></p><ul><li><a href="https://towardsdatascience.com/understanding-hyperparameters-and-its-optimisation-techniques-f0debba07568">Understanding Hyperparameters and its Optimisation techniques</a></li><li><a href="https://www.datacamp.com/community/tutorials/parameter-optimization-machine-learning-models">Hyperparameter Optimization in Machine Learning Models (article</a>)</li><li><a href="https://medium.com/@mikkokotila/a-comprehensive-list-of-hyperparameter-optimization-tuning-solutions-88e067f19d9">A Comprehensive List of Hyperparameter Optimization &amp; Tuning Solutions</a> (tools)</li></ul><blockquote>How about start to train your own model.</blockquote><blockquote>I wish it was helpful for you.<br>Good luck with your Training, and your Study!</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d2e0c3d51794" width="1" height="1" alt=""><hr><p><a href="https://medium.com/udacity-pytorch-challengers/trials-errors-and-trade-offs-in-my-deep-learning-model-d2e0c3d51794">Trials, errors and trade-offs in my deep learning model</a> was originally published in <a href="https://medium.com/udacity-pytorch-challengers">Udacity PyTorch Challengers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Style Transfer using Deep Neural Network and PyTorch]]></title>
            <link>https://medium.com/udacity-pytorch-challengers/style-transfer-using-deep-nural-network-and-pytorch-3fae1c2dd73e?source=rss----c6e264a51486---4</link>
            <guid isPermaLink="false">https://medium.com/p/3fae1c2dd73e</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Ritul]]></dc:creator>
            <pubDate>Wed, 16 Jan 2019 13:46:28 GMT</pubDate>
            <atom:updated>2019-01-16T13:46:28.325Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HFD6SjpXIamKzw4O4HmuMQ.jpeg" /></figure><h3>Introduction</h3><p>Nowadays everyone is excited about doing projects using machine learning or deep learning. Through this blog, I will give you a chance to be ‘ Picasso ‘ of deep learning as we are going to explore the method of style transfer using Deep Convolutional Neural Networks. we will use pre-trained network VGG19 for that.</p><p>Don’t worry, it just sounds tough but actually way easy. You just need to be familiar with Python, PyTorch and some Deep Learning stuff such as CNN and how to use pre-trained networks (as we will be using a pre-trained CNN for our style transfer). I will brush up your concepts about CNN.</p><h3><strong>What is CNN( Convolutional Neural Network)</strong></h3><p>A convolutional layer + activation function, followed by a pooling layer, and a linear layer (to create the desired output size) make up the basic layers of a CNN.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/517/1*UCfn71ycVf4-ISUGKtMing.png" /></figure><ol><li><strong>Convolutional layer</strong>: It is obtained by applying various filters on input image for feature extraction. The number of filters used defines the depth of the layer. Activation function is applied to output of the convolutional layer for scaling.</li><li><strong>Pooling layer</strong>: A max pooling layer reduces the x-y size of input and only keeps the most active pixel values. Below is an example of a 2x2 pooling kernel, with a stride of 2.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/559/1*pynt5an155xWRSGLlf-uPA.png" /></figure><h3>Why we are using VGG19 for Style Transfer</h3><p>We can use either of VGG16 and VGG19 for feature extraction as they are performing very well as compared to others in case of style transfer. For example, here I have used VGG19.</p><p>Other than VGG, you can use SqueezeNet, it is faster but results are worst and in case of Inception, it performs well but you have to change striding/kernels, max pooling to average pooling, search over various layer combos. So VGG is best at the moment.</p><p>You can read more <a href="https://www.reddit.com/r/MachineLearning/comments/7rrrk3/d_eat_your_vggtables_or_why_does_neural_style/">here</a></p><h3>How to perform Style Transfer</h3><p>Style transfer uses the features found in the 19-layer VGG Network, which is comprised of a series of convolutional and pooling layers, and a few fully-connected layers. Convolutional layers are named by the stack and their order in the stack. As first convolutional layer is named as conv1_1 and the deepest convolutional layer is conv5_4.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*UKfg9SZfPu7i5X3a8XpBOQ.png" /></figure><p>Style transfer relies on <strong>separating content and style of an image</strong>. Our target is to create a new image containing style of style image and content of content image( base image).</p><ol><li>Content( objects and their arrangement) from the given content image.</li><li>Style( colour and texture) from given style image.</li></ol><h4><strong><em>Load in VGG19 (features):</em></strong></h4><p>VGG19 is consists of two parts:</p><ol><li>vgg19.features (convolutional and pooling layer)</li><li>vgg19.classifier ( last three layers for output)</li></ol><p>For style transfer we need only features portion ,so we will load in that and freeze the weights.</p><pre>import matplotlib.pyplot as plt<br>import numpy as np</pre><pre>import torch<br>import torch.optim as optim<br>from torchvision import transforms, models</pre><pre>vgg = models.vgg19(pretrained=True).features</pre><pre># freeze all VGG parameters since we’re only optimizing the target image<br>for param in vgg.parameters():<br> param.requires_grad_(False)</pre><h4>Load style and content image:</h4><pre># define load_image() function which deals with images size<br># load in content and style image <br>content = load_image(&#39;images/x.jpg&#39;)<br># Resize style to match content, makes code easier<br>style = load_image(&#39;images/y.jpg&#39;,shape=content.shape[-2:])<br>#covert this images to tensor and apply transforms</pre><h4>Content and style feature:</h4><ol><li>For content representation of target image we pass the content image through model and take the output of conv4_2 layer, as it is considered to be containing most accurate content features.</li><li>For Style representation of target image, we consider the outputs of conv1_1, conv2_1,conv3_1,conv4_1, and conv5_1 layers, again this for the same reason containing accurate style features.</li></ol><p>We will then calculate the gram matrix for output of each convolutional layer used for style feature extraction, to find a co-relation between any spatial information.</p><p><strong>Gram matrix</strong> is calculated by multiplying a matrix by its transpose.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*X6upHYkV8Z3ugK36fQc_sg.png" /></figure><h3>Creating our canvas and putting all together:</h3><pre># define get_feature() and get content and style features only once before forming the target image<br>content_features = get_features(content, vgg)<br>style_features = get_features(style, vgg)</pre><pre># calculate the gram matrices for each layer of our style representation<br>style_grams = {layer: gram_matrix(style_features[layer]) for layer in style_features}</pre><pre># create a third &quot;target&quot; image and prep it for change<br># it is a good idea to start of with the target as a copy of our *content* image<br># then iteratively change its style<br>target = content.clone().requires_grad_(True)</pre><h3>Loss and Weight :</h3><p><strong>Individual layer style weight</strong></p><p>We assign weights to the outputs of each layer to control their style effect on our final image.If u want larger style artifacts than you should give higher weights to initial layers conv1_1, conv2_1 and vice versa. Weights are in the range of 0–1.</p><p><strong>Content and Style weight</strong></p><p>We define an alpha (content_weight) and a beta (style_weight). This ratio will affect how <em>stylized</em> your final image is. It’s recommended to keep content_weight as 1 and change style_weight.</p><pre># weights for each style layer <br># weighting earlier layers more will result in *larger* style artifacts<br># notice we are excluding `conv4_2` our content representation<br>style_weights = {&#39;conv1_1&#39;: 1.,<br>                 &#39;conv2_1&#39;: 0.75,<br>                 &#39;conv3_1&#39;: 0.2,<br>                 &#39;conv4_1&#39;: 0.2,<br>                 &#39;conv5_1&#39;: 0.2}</pre><pre>content_weight = 1  # alpha<br>style_weight = 1e6  # beta</pre><p><strong>Content loss</strong></p><p>It is mean squared difference between target and content features at layer conv4_2.</p><pre>content_loss = torch.mean((target_features[&#39;conv4_2&#39;] - content_features[&#39;conv4_2&#39;])**2)</pre><p><strong>Style loss</strong></p><p>For style loss we will calculate gram matrix of target image and than will compare it with the gram matrix of style image at layers used for style feature extraction as conv1_1 , conv2_1 etc. Again it is mean squared difference.</p><p><strong>Total Loss</strong></p><p>It will be calculated by adding style and content loss after weighting them with alpha and beta.</p><pre>total_loss = content_weight * content_loss + style_weight * style_loss</pre><p>Our aim here is to minimize the total loss by iterating and updating the values.</p><pre># for displaying the target image, intermittently<br>show_every = 400</pre><pre># iteration hyper parameters<br>optimizer = optim.Adam([target], lr =0.003)<br>steps = 2000  # decide how many iterations to update your image (5000)</pre><pre>for ii in range(1, steps+1):<br>    <br>    # get the features from your target image<br>    target_features = get_features(target, vgg)<br>    <br>    # the content loss<br>    content_loss = torch.mean((target_features[&#39;conv4_2&#39;] - content_features[&#39;conv4_2&#39;])**2)<br>    <br>    # the style loss<br>    # initialize the style loss to 0<br>    style_loss = 0<br>    # then add to it for each layer&#39;s gram matrix loss<br>    for layer in style_weights:<br>        # get the &quot;target&quot; style representation for the layer<br>        target_feature = target_features[layer]<br>        target_gram = gram_matrix(target_feature)<br>        _, d, h, w = target_feature.shape<br>        # get the &quot;style&quot; style representation<br>        style_gram = style_grams[layer]<br>        # the style loss for one layer, weighted appropriately<br>        layer_style_loss = style_weights[layer] * torch.mean((target_gram - style_gram)**2)<br>        # add to the style loss<br>        style_loss += layer_style_loss / (d * h * w)<br>        <br>    # calculate the *total* loss<br>    total_loss = content_weight * content_loss + style_weight * style_loss<br>    <br>    # update your target image<br>    optimizer.zero_grad()<br>    total_loss.backward()<br>    optimizer.step()<br>   </pre><p>You can find complete code for style transfer <a href="https://github.com/p-rit/pytorch-Scholarship-challenge/blob/master/style%20transfer/Style%20transfer.ipynb">here</a></p><h3>Conclusion</h3><p>By reading this blog, you will get an overview about how style transfer happens and picture editing apps like <em>Prisma</em> works. Here we used gram matrix calculation but you can also improve your style transfer by using various other approaches such as encoder and decoder networks etc.</p><p>Moreover, the major drawback of this technique is we are paying in terms of time for better results, you can also search for real-time style transfer as an update on the existing one.</p><h4>References:</h4><ol><li><a href="https://in.udacity.com/course/deep-learning-pytorch--ud188">PyTorch nano degree</a></li><li><a href="https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Gatys_Image_Style_Transfer_CVPR_2016_paper.pdf">https://www.cvfoundation.org/openaccess/content_cvpr_2016/papers/Gatys_Image_Style_Transfer_CVPR_2016_paper.pdf</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3fae1c2dd73e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/udacity-pytorch-challengers/style-transfer-using-deep-nural-network-and-pytorch-3fae1c2dd73e">Style Transfer using Deep Neural Network and PyTorch</a> was originally published in <a href="https://medium.com/udacity-pytorch-challengers">Udacity PyTorch Challengers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Neural Network]]></title>
            <link>https://medium.com/udacity-pytorch-challengers/understanding-neural-network-c68f456838e?source=rss----c6e264a51486---4</link>
            <guid isPermaLink="false">https://medium.com/p/c68f456838e</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Akash Singh]]></dc:creator>
            <pubDate>Tue, 08 Jan 2019 03:31:18 GMT</pubDate>
            <atom:updated>2019-01-08T03:31:18.059Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<h3>Understanding the Neural Network using PyTorch</h3><p>An artificial <strong>neural network</strong> is an interconnected group of nodes, similar to the vast <strong>network</strong> of neurons in a brain. It is the building block of deep learning models it has mainly 3 layers Input Layer, a hidden layer and output layer every node has its own activation function weights and biases attached to it according to which its output is calculated and fed into the next layer except the input layer which only has input.</p><p>Now we will make a simple Neural Network with Input Layer containing 2 Nodes a single Hidden Layer with 2 Nodes and an output Layer with 1 Node</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*x2BIAmIy-8O1PClE3Uw0nA.png" /><figcaption>This is how the neural network look</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*isfgx0pT59uU_gx1IVzQ4g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*JL2oz95eWgkPXtKlMfYauA.png" /><figcaption>The model is initialized with random weights and biases</figcaption></figure><p>Now we will do a forward pass in your neural network we will pass the random weights and biases through our network and calculate the output and use it to get the error and then we will do a backpropagation to reduce the error.</p><p>Now let us give the input value as 1 and 2 and use it to calculate the value of all the nodes in the network</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*SLh5KrYXxOjOkoSsYc6w1w.png" /><figcaption>We have given input 1 and 2 and then calculated the value of h1,h2, and output this whole step can be done by just using model(inputs). I have done this to show you how exactly forward pass takes place</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*7_XpQlBshga2-OUqrmx9aA.png" /><figcaption>This is how our model looks now</figcaption></figure><p>We want the target to be1 but we get the output as 0.59 so now we calculate the error the same way as for mean squared error.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*WJws3szd2SjykIgyv2NMZg.png" /></figure><p>Now comes backpropagation we have got the error we will use it to calculate the new biases and weights</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*0pVYPv52CzaheeEQw2ku1A.png" /><figcaption>We will add the negative of this in W11_2</figcaption></figure><p>All this maths seems pretty daunting lucky for us and PyTorch has a simple function to calculate this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*ySUFXiXOkenMu69i7L7jmw.png" /></figure><p>The above step will calculate the partial derivative and update the weights and biases accordingly</p><p>A fully connected neural network is the heart of deep learning in my next post I would talk about what is convolution neural network and how by combing convolution neural network and fully connected neural network we can classify images.</p><p>If you want to know more about Neural Network you can check this out <a href="https://www.youtube.com/watch?v=aircAruvnKk">https://www.youtube.com/watch?v=aircAruvnKk</a> <a href="https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html">https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html</a></p><p><em>Originally published at </em><a href="https://medium.com/@akash.singh7337/understanding-neural-network-ff81a19c92e6"><em>medium.com</em></a><em> on January 5, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c68f456838e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/udacity-pytorch-challengers/understanding-neural-network-c68f456838e">Understanding Neural Network</a> was originally published in <a href="https://medium.com/udacity-pytorch-challengers">Udacity PyTorch Challengers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Loss Function and Error in Neural Network]]></title>
            <link>https://medium.com/udacity-pytorch-challengers/understanding-loss-function-and-error-in-neural-network-676a81cc776c?source=rss----c6e264a51486---4</link>
            <guid isPermaLink="false">https://medium.com/p/676a81cc776c</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[loss-function]]></category>
            <category><![CDATA[neural-networks]]></category>
            <dc:creator><![CDATA[Shashi Gharti]]></dc:creator>
            <pubDate>Mon, 07 Jan 2019 15:51:15 GMT</pubDate>
            <atom:updated>2019-01-07T16:07:04.188Z</atom:updated>
            <content:encoded><![CDATA[<p>Loss function helps us to quantify how good/bad our current model is in predicting some value which it is trained to predict. This article aims you to explain the role of loss function in neural network.</p><p>I assume that you have basic understanding of Neural Networks and Pytorch library. Throughout this tutorial, I will use google <a href="https://colab.research.google.com/">colab</a> to write and test the code. I will be using Dog vs Cat dataset from kaggle. You can download it from <a href="https://www.kaggle.com/c/dogs-vs-cats">here</a>.</p><h3>Cat Images from kaggle dataset</h3><p>The dataset that I downloaded looks like this. I will be using cat images only.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/629/1*7BZikVANdBxF7-npBxA-qw.png" /></figure><h3><strong>Let’s Get Started!</strong></h3><p>I will start my explanation with an example of a simple neural network as shown in Figure 1 where x1 and x2 are inputs to the function f(x). The output y_hat is the weighted sum of inputs passed through the activation function. I have omitted bias for the sake of simplicity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/294/1*VU4cY78JrRTaoz-NTTYlSA.jpeg" /><figcaption>Single Layer Neural Network</figcaption></figure><p><strong>What is a loss function?</strong></p><p>Let’s prepare a training dataset of cats along with true labels for each input image and pass it to our model.</p><pre>train_transforms =transforms.Compose([<br>                         transforms.RandomResizedCrop(224),<br>                         transforms.ToTensor(),<br>                         transforms.Normalize([0.485, 0.456, .406],<br>                                              [0.229, 0.224,.225])])<br>train_data = datasets.ImageFolder(data_dir + &#39;/train&#39;, transform=train_transforms)<br>train_loader = torch.utils.data.DataLoader(train_data, batch_size=64, shuffle=True)</pre><p>Following is Densenet, a pretrained model that we will use for this tutorial.</p><pre>model = models.densenet121(pretrained=True)<br>classifier = nn.Sequential(nn.Linear(2048, 512),   <br>                           nn.ReLU(),<br>                           nn.Dropout(p=0.2),<br>                           nn.Linear(512,2),<br>                           nn.LogSoftmax(dim=1)<br>)</pre><p>All the training images will be passed to this model in batches:</p><pre>for epoch in range(epochs):<br>  for inputs, labels in train_loader:<br>    logps = model.forward(inputs)</pre><p>For each batch, we compare the output from the model (predicted values) with the actual labels. To explain in simple words, we check if the image predicted as cat by the model is actually a cat or not. If all the predictions are correct, then our model is good. But if model identifies dog as a cat, then we have an error. This is what the loss function calculates.</p><p>Loss function takes in the predicted score coming out from our model together with true target or labels, compares it and gives us some quantitative value of how good or bad those predictions are for the training images.</p><pre>criterion = nn.NLLLoss()</pre><pre>logps = model.forward(inputs)<br>batch_loss = criterion(logps, labels)</pre><p>There are different types of <a href="https://isaacchanghau.github.io/post/loss_functions/">loss functions</a> that we can use. Choosing the optimal loss function depends on various factors and the type of problem we are trying to solve such as regression or classification. In this tutorial, we are using <a href="https://pytorch.org/docs/stable/nn.html#nllloss">NLLLoss</a> function.</p><h4>How do we minimize this error?</h4><p>Once we calculate the loss, our next step is to decide whether to continue training or not . The decision is based on the loss value. If the loss value is very high, then we need to update the weight parameters of the model and repeat the same process. If the loss is low and satisfies our need then we stop the training process.</p><p>Based on the calculated error, the optimization functions are used to optimize the weights using <a href="https://en.wikipedia.org/wiki/Gradient_descent">gradient descent</a> and <a href="https://en.wikipedia.org/wiki/Backpropagation">back propagation</a>.</p><pre>#compute the  gradients<br>loss.backward()</pre><pre>#update the parameters<br>optimizer.step()</pre><p>In this way, we modify the weight parameters of the model and repeat the same process until the loss is reduced and it is minimal.</p><h3>Output</h3><p>Yay! We have completed the training part for our model. The calculated loss per batch looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/607/1*6ouYVyCoNQ9y1d8DTpDs3A.png" /><figcaption>Loss calculation per batch</figcaption></figure><p>The working code for this tutorial can be downloaded from <a href="https://colab.research.google.com/drive/1I1J0Uti2Xo1ubDXg1nldvkB_BWol1hRh#scrollTo=nvd3WI4s6n3a">here</a>.</p><h3>Summary</h3><p>The main objective of this tutorial is to give detailed explanation on error calculation and loss function rather than how to train a model. So we examined and observed the error calculation and optimization step in detail, while training our model. However, this tutorial doesn’t explain all the <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html">steps</a> involved in training process of neural network. I hope this tutorial adds some value to help you understand the importance and role of loss function in neural network.</p><p><strong>References</strong></p><p><a href="https://isaacchanghau.github.io/post/loss_functions/">https://isaacchanghau.github.io/post/loss_functions/</a><br><a href="https://seba-1511.github.io/tutorials/beginner/blitz/neural_networks_tutorial.html">https://seba-1511.github.io/tutorials/beginner/blitz/neural_networks_tutorial.html</a><br><a href="https://heartbeat.fritz.ai/5-regression-loss-functions-all-machine-learners-should-know-4fb140e9d4b0">https://heartbeat.fritz.ai/5-regression-loss-functions-all-machine-learners-should-know-4fb140e9d4b0</a><br><a href="https://www.youtube.com/watch?v=h7iBpEHGVNc">https://www.youtube.com/watch?v=h7iBpEHGVNc</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=676a81cc776c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/udacity-pytorch-challengers/understanding-loss-function-and-error-in-neural-network-676a81cc776c">Understanding Loss Function and Error in Neural Network</a> was originally published in <a href="https://medium.com/udacity-pytorch-challengers">Udacity PyTorch Challengers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Differentiating PyTorch from all other Deep Learning frameworks]]></title>
            <link>https://medium.com/udacity-pytorch-challengers/differentiating-pytorch-from-all-other-deep-learning-frameworks-86a7f89754b5?source=rss----c6e264a51486---4</link>
            <guid isPermaLink="false">https://medium.com/p/86a7f89754b5</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Robin Familara]]></dc:creator>
            <pubDate>Mon, 07 Jan 2019 04:11:12 GMT</pubDate>
            <atom:updated>2019-01-07T04:11:12.029Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>“PyTorch are dynamic Neural Networks which are like tensors on steroids running in Python with strong GPU acceleration”</blockquote><p>How does PyTorch fare among all other Deep Learning frameworks? Is it a cut above the rest right on its early staging or is it still flowing its way towards the top of the chart in terms of features and performance?</p><p>Deep Learning systems and applications are gaining popularity in numerous fields, such as image classifications, object detection, translations, and Natural Language Processing. We witness emerging trends of a growing number of DL software frameworks in both open source and commercial markets which are represented by those top of the line such as Tensor-Flow, Caffe, Torch, Keras, Theano, and the one described as the “youngest” but would most likely be the fastest framework called PyTorch.</p><p>Most DL frameworks utilize a similar layered software architecture to provide APIs to enable users to configure the Deep Neural Network model and the training methods which includes the optimizer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/965/1*J1XSrLJpkMytj8YXJymCSw.jpeg" /><figcaption>Deep Learning Framework and optimizer</figcaption></figure><p><strong>Before comparing further, let’s explore the special features of PyTorch:</strong></p><p><a href="http://pytorch.org/">PyTorch</a> has seen a high level of adoption within the deep learning framework community. It shines for a rapid prototyping with Dynamic Neural Networks, auto-differentiation, deep Python integration, and strong support for GPUs. It also enables fast, flexible experimentation and efficient production through a hybrid front-end, distributed training, and ecosystem of tools and libraries.</p><p>PyTorch runs on Python which means that anyone with a basic understanding of Python can immediately start to build their own first deep learning models. Given PyTorch framework’s architectural style, an entire Deep Learning model process is far simpler and transparent as compared to all other Deep Learning frameworks. It was developed by Facebook and is used by Twitter, Salesforce, the University of Oxford, and many others.</p><p>PyTorch is mainly used to train deep learning models quickly and effectively, so it is the framework of choice for a large number of researchers, engineers, and developers</p><p><strong>Advantages of PyTorch:</strong></p><p>1.)It is Based On Python (Python is the simplest programming language)</p><p>2.)Faster Deep Learning training (More parameters, faster training time)</p><p>3.)Dynamic approach via GPU (Each level of computation can be viewed)</p><p>4.)Increase in Developer productivity (It is user-friendly and simple to use)</p><p>5.)Easier to learn (The PyTorch documentation is helpful for beginners)</p><p>6.)A community of focused developers (Small group who solve big issues)</p><p>7.)Transparent to the user(Dynamic Graph outputs viewed faster)</p><p>8.)Easy to debug (Uses PyCharm for define-by-run mode during runtime)</p><p>9.)Data parallelism (Allows “<strong>torch.nn.DataParallel”</strong> to wrap any module)</p><p><strong>Disadvantages of PyTorch:</strong></p><p>1.)It lacks model serving in production (Although it will change in the future)</p><p>2.)It’s new and not yet widely known (Has less costumers/users at this point)</p><p>3.) It lacks interfaces for monitoring and visualization such as Tensorboard (As a workaround, you can connect externally to Tensorboard)</p><p><strong>Comparative Analysis of DL Frameworks speed:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*njX03ohZquP_Gvv4VZR6zA.jpeg" /></figure><p>The Diagram represents grouping of the frameworks by models to see which models were fastest using which framework. In case of inception models, only Tensor Flow can be compared to Keras and in both cases Tensorflow is faster. The ResNet50 achieves lowest training time when Tensorflow is used. VGG models stand in opposition to that, because both are trained quickest through PyTorch.</p><p>PyTorch has various advantages and counting. It may have disadvantages but will eventually be addressed in the near future. It is definitely one of the top Deep Learning frameworks nowadays due to its high performance, fast training of different models, and outstanding features which trends it as researchers’ favorite and engineers’ choice recently.</p><p><strong>Related reference links for more reads:</strong></p><ol><li><a href="https://pytorch.org/">https://pytorch.org</a></li><li><a href="https://www.fast.ai/">https://www.fast.ai/</a></li></ol><p>I never want to end explaining this interesting topic but I’ve realized the need to allow you to explore furthermore on the important facts in Deep Learning through PyTorch. I also highly encourage you to read more about PyTorch and its pros and cons updates in comparison to other Deep Learning Frameworks. Regularly practice and test to see it yourself as to why we would love to utilize it onwards. Also, I wish all of you the best on your deep learning ventures in the years to come!</p><p>Up Next — The further emergence of PyTorch in 2019 and onwards</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=86a7f89754b5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/udacity-pytorch-challengers/differentiating-pytorch-from-all-other-deep-learning-frameworks-86a7f89754b5">Differentiating PyTorch from all other Deep Learning frameworks</a> was originally published in <a href="https://medium.com/udacity-pytorch-challengers">Udacity PyTorch Challengers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Brief Overview of Loss Functions in Pytorch]]></title>
            <link>https://medium.com/udacity-pytorch-challengers/a-brief-overview-of-loss-functions-in-pytorch-c0ddb78068f7?source=rss----c6e264a51486---4</link>
            <guid isPermaLink="false">https://medium.com/p/c0ddb78068f7</guid>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-neural-network]]></category>
            <dc:creator><![CDATA[Pratyaksha Jha]]></dc:creator>
            <pubDate>Sun, 06 Jan 2019 14:52:46 GMT</pubDate>
            <atom:updated>2019-01-06T14:52:46.329Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rIPbsKLYGg9HSt9L" /><figcaption>Photo by <a href="https://unsplash.com/@element5digital?utm_source=medium&amp;utm_medium=referral">Element5 Digital</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>What are loss functions?</h3><p>Training the neural network is similar to how humans learn. We give data to the model, it predicts something and we tell it whether the prediction is correct or not. The model then corrects its mistakes. The model does this repeatedly until it reaches a certain level of accuracy, decided by us. Telling the model that the prediction was wrong is crucial for it to learn well. This where the loss function comes in. It tells the model how far off its estimation was from the actual value. While communicating with a human is easier, to tell so to a machine we need a medium(pun intended). This communication needs a <strong><em>how </em></strong>and a <strong><em>what</em></strong>. The <strong><em>How </em></strong>is the programming language Python and the <strong><em>What </em></strong>is the Mathematics. In this post, I’ll go through some <em>Hows</em>, <em>Whats </em>and the intuition behind them.</p><p>For a quick recap of how neural networks train, have a look at <a href="https://towardsdatascience.com/how-do-we-train-neural-networks-edd985562b73">this</a> amazing post. My post is meant for people who are familiar with Deep Learning. For nitty-gritty details refer Pytorch <a href="https://pytorch.org/docs/stable/nn.html#loss-functions">Docs</a>.</p><h3>Mean Absolute Error</h3><p>torch.nn.L1Loss</p><p>Measures the mean absolute error.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/252/1*TEjDaYANLY-V_xpLTO4vew.png" /></figure><p>where <em>x </em>is the actual value and <em>y </em>is the predicted value.</p><p><strong>What does it mean?<br></strong>It measures the numerical distance between the estimated and actual value. It is the simplest form of error metric. The absolute value of the error is taken because if we don’t then negatives will cancel out the positives. This isn’t useful to us, rather it makes it more unreliable. <br>The lower the value of MAE, better is the model. We can not expect its value to be zero, because it might not be practically useful. This leads to wastage of resources. For example, if our model’s loss is within 5% then it is alright in practice, and making it more precise may not really be useful.</p><p><strong>When to use it?<br></strong>+ Regression problems<br>+ Simplistic model<br>+ As neural networks are usually used for complex problems, this function is rarely used.</p><h3>Mean Square Error Loss</h3><p>torch.nn.MSELoss</p><p>It measures the mean squared error (squared L2 norm).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/276/1*R-hZsru4TeNsSTPSxb779w.png" /></figure><p>where <em>x </em>is the actual value and <em>y </em>is the predicted value.</p><p><strong>What does it mean?</strong><br>The squaring of the difference of prediction and actual value means that we’re amplifying large losses. If the classifier is off by 200, the error is 40000 and if the classifier is off by 0.1, the error is 0.01. This penalizes the model when it makes large mistakes and incentivizes small errors.</p><p><strong>When to use it?</strong><br>+ Regression problems.<br>+ The numerical value features are not large.<br>+ Problem is not very high dimensional.</p><h3>Smooth L1 Loss</h3><p>torch.nn.SmoothL1Loss</p><p>Also known as Huber loss, it is given by —</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/491/1*iMPKyAwlzWmC4BmtT7XkdQ.png" /></figure><p><strong>What does it mean?<br></strong>It uses a squared term if the absolute error falls below 1 and an absolute term otherwise. It is less sensitive to outliers than the mean square error loss and in some cases prevents exploding gradients. In mean square error loss, we square the difference which results in a number which is much larger than the original number. These high values result in exploding gradients. This is avoided here as for numbers greater than 1, the numbers are not squared.</p><p><strong>When to use it?<br></strong>+ Regression.<br>+ When the<strong> </strong>features have large values.<br>+ Well suited for most problems.</p><h3>Negative Log-Likelihood Loss</h3><p>torch.nn.NLLLoss</p><p>The negative log-likelihood loss:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/258/1*Ve5pDZkRFKDaFnfSJJHncw.png" /></figure><p><strong>What does it mean?<br></strong>It maximizes the overall probability of the data. It penalizes the model when it predicts the correct class with smaller probabilities and incentivizes when the prediction is made with higher probability. The logrithm does the penalizing part here. Smaller the probabilities, higher will be its logrithm. The negative sign is used here because the probabilities lie in the range [0, 1] and the logrithms of values in this range is negative. So it makes the loss value to be positive.</p><p><strong>When to use it?<br></strong>+<strong> </strong>Classification.<br>+ Smaller quicker training.<br>+ Simple tasks.</p><h3>Cross-Entropy Loss</h3><p>torch.nn.CrossEntropyLoss</p><p>Measures the cross-entropy between the predicted and the actual value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/329/1*rsaojcasXHxwCc-_LEywpw.png" /></figure><p>where <em>x </em>is the probability of true label and <em>y </em>is the probability of predicted label.</p><p><strong>What does it mean?</strong><br>Cross-entropy as a loss function is used to learn the probability distribution of the data. While other loss functions like squared loss penalize wrong predictions, cross entropy gives a greater penalty when incorrect predictions are predicted with high confidence. What differentiates it with negative log loss is that cross entropy also penalizes wrong but confident predictions and correct but less confident predictions, while negative log loss does not penalize according to the confidence of predictions.</p><p><strong>When to use it?</strong><br>+ Classification tasks<br>+ For making confident model i.e. model will not only predict accurately, but it will also do so with higher probability. <br>+ For higher precision/recall values.</p><h3>Kullback-Leibler divergence</h3><p>torch.nn.KLDivLoss</p><p>KL divergence gives a measure of how two probability distributions are different from each other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/339/1*c2ZVnnG0n6msFdyrujEN6A.png" /></figure><p>where <em>x </em>is the probability of true label and <em>y </em>is the probability of predicted label.</p><p><strong>What does it mean?<br></strong>It is quite similar to cross entropy loss. The distinction is the difference between predicted and actual probability. This adds data about information loss in the model training. The farther away the predicted probability distribution is from the true probability distribution, greater is the loss. It does not penalize the model based on the confidence of prediction, as in cross entropy loss, but how different is the prediction from ground truth. It usually outperforms mean square error, especially when data is not normally distributed. The reason why cross entropy is more widely used is that it can be broken down as a function of cross entropy. Minimizing the cross-entropy is the same as minimizing KL divergence.<br><em>KL = — xlog(y/x) = xlog(x) — xlog(y) = Entropy — Cross-entropy</em></p><p><strong>When to use it?<br></strong>+<strong> </strong>Classification<br>+ Same can be achieved with cross entropy with lesser computation, so avoid it.</p><h3>Margin Ranking Loss</h3><p>torch.nn.MarginRankingLoss</p><p>It measures the loss given inputs <em>x1, x2</em>, and a label tensor <em>y</em> with values (1 or -1). If y == 1 then it assumed the first input should be ranked higher than the second input, and vice-versa for y == -1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*_w45nk6GZ5CWK16nuHQxLQ.png" /></figure><p><strong>What does it mean?<br></strong>The prediction y of the classifier is based on the ranking of the inputs <em>x1 </em>and x2. Assuming margin to have the default value of 0, if <em>y </em>and <em>(x1-x2)</em> are of the same sign, then the loss will be zero. This means that <em>x1/x2 </em>was ranked higher(for <em>y=1/-1</em>), as expected by the data. If <em>y </em>and <em>(x1-x2)</em> are of the opposite sign, then the loss will be the non-zero value given by <em>y * (x1-x2)</em>. This means that either <em>x2 </em>was ranked higher when <em>x1</em> should have been ranked higher or vice versa. Although its usage in Pytorch in unclear as much open source implementations and examples are not available as compared to other loss functions.</p><p><strong>When to use it?</strong><br>+ GANs.<br>+ Ranking tasks.</p><h3>Hinge Embedding Loss</h3><p>torch.nn.HingeEmbeddingLoss</p><p>Measures the loss given an input tensor<em> x</em> and a labels tensor <em>y</em> containing values (1 or -1). It is used for measuring whether two inputs are similar or dissimilar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/559/1*0kTCumW2UvsY1vei3bIIlg.png" /></figure><p><strong>What does it mean?<br></strong>The prediction y of the classifier is based on the value of the input <em>x</em>. Assuming margin to have the default value of 1, if <em>y=-1</em>, then the loss will be maximum of 0 and <em>(1 — x)</em>. If <em>x</em> &gt; 0 loss will be <em>x</em> itself (higher value), if 0&lt;<em>x</em>&lt;1 loss will be <em>1 — x </em>(smaller value) and if <em>x</em> &lt; 0 loss will be 0 (minimum value). For <em>y</em> =1, the loss is as high as the value of <em>x</em>.</p><p><strong>When to use it?</strong><br>+ Learning nonlinear embeddings<br>+ Semi-supervised learning<br>+ Where similarity or dissimilar of two inputs is to be measured.</p><h3>Cosine Embedding Loss</h3><p>torch.nn.CosineEmbeddingLoss</p><p>It measures the loss given inputs <em>x1, x2</em>, and a label tensor <em>y</em> containing values (1 or -1). It is used for measuring whether two inputs are similar or dissimilar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/1*lGz---Oq1wFfidZhT9E6BA.png" /></figure><p><strong>What does it mean?<br></strong>The prediction y of the classifier is based on the cosine distance of the inputs <em>x1 </em>and x2. Cosine distance refers to the angle between two points. It can be easily found out by using dot products as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/345/1*1aCg6om9VImdKf9-X2me1g.png" /></figure><p>As cosine lies between - 1 and + 1, loss values are smaller. This aids in computation. Assuming margin to have the default value of 0, if <em>y</em> =1, the loss is (1 - cos(<em>x1, x2))</em>. For <em>y=-1</em>, then the loss will be maximum of 0 and cos<em>(x1, x2)</em>. If cos<em>(x1, x2)</em> &gt; 0 loss will be cos<em>(x1, x2)</em> itself (higher value), and if cos<em>(x1, x2)</em> &lt; 0 loss will be 0 (minimum value).</p><p><strong>When to use it?</strong><br>+ Learning nonlinear embeddings<br>+ Semi-supervised learning<br>+ Where similarity or dissimilar of two inputs is to be measured.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*gQXcj6foEwD3VHvri-79VA.jpeg" /></figure><p>Check out <a href="https://medium.com/@zhang_yang/pytorch-loss-funtions-in-plain-python-b79c05f8b53f">this post</a> for plain python implementation of loss functions in Pytorch.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c0ddb78068f7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/udacity-pytorch-challengers/a-brief-overview-of-loss-functions-in-pytorch-c0ddb78068f7">A Brief Overview of Loss Functions in Pytorch</a> was originally published in <a href="https://medium.com/udacity-pytorch-challengers">Udacity PyTorch Challengers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multi-Layer Perceptrons (MLPs)]]></title>
            <link>https://medium.com/udacity-pytorch-challengers/multi-layer-perceptrons-mlps-b42166dc0067?source=rss----c6e264a51486---4</link>
            <guid isPermaLink="false">https://medium.com/p/b42166dc0067</guid>
            <category><![CDATA[perceptron]]></category>
            <category><![CDATA[udacity]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[beginner]]></category>
            <dc:creator><![CDATA[nahid]]></dc:creator>
            <pubDate>Sun, 06 Jan 2019 14:48:46 GMT</pubDate>
            <atom:updated>2019-01-06T14:48:46.405Z</atom:updated>
            <content:encoded><![CDATA[<p>As with any new subject or topic, it is important to lay down the basics to set the tone, line of thought and context. The basics may or may not be essential to the bigger picture BUT it is a necessary starting point. And the starting point to this topic we are delving into was my previous article — <a href="https://medium.com/udacity-pytorch-challengers/what-is-a-perceptron-a2d73996545f">What is a Perceptron?</a></p><p>As simple and straightforward a typical or linear perceptron is, in actual fact perceptrons are much more versatile than that. Versatility is almost a necessity as most AI case studies do not have only one solution, there’s always a better and more efficient way to approach problems.</p><h4>The problem with linear perceptrons</h4><p>Perceptrons do really well when the data or the situation we’re dealing with is linearly separable ie when the data can be easily classified or separated. Previously, we dealt with the cancer problem and while that was a great example to start with, any cancer expert will tell you that the data that determines the result of a cancer test is a lot more complicated than that.</p><p>Let’s take an even simpler example — University Admissions. The simplest way to approach this would be to select high school students with the best academic results. So, in doing this we can list out all the University applicants’ results. Arrange them in decreasing order and accept the top 100 students into the University. Which can easily be implemented by a linear perceptron.<br>But, there are many more factors that determine a student’s contribution to and performance at the University level for example — co-curricular activities, sports, financial capacity etc. And if we were to factor in a few of these more factors, our problem will get more and more complicated and less and less linear. Suddenly, linear perceptrons aren’t looking so great.</p><p>Basically, linear perceptrons will struggle at solving problems that aren’t so black and white ie non-linear data. Another great example that depicts this is how easily a perceptron can describe the typical boolean gates — AND &amp; OR but fails at describing the XOR gate. More on this in the resources provided below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*IkA-zQbYFKC14XffsFnSxg.jpeg" /><figcaption>Linear data (above), Non-linear data (below)</figcaption></figure><h4>Multi-Layer Perceptrons (MLPs)</h4><p>In order to solve more complex and non-linear problems we can begin by stacking linear perceptrons over each other. This network of stacked perceptrons or <em>deep, artificial neural networks </em>are known as Multi-Layer Perceptrons. “Deep” due to the presence of multiple layers. And “network” due to the this layered arrangement working together towards a common goal.</p><p>An MLP consists of an input layer, output layer and a fixed number of layers in between known as <em>hidden layers</em>. The number of these hidden layers vary depending on the application. Also note that just as in a singular or linear perceptron each node in the input and hidden layer have an associated weight attached to it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/1*jfEsZoTqJ6W_6tE1wMCdhQ.jpeg" /><figcaption>Source: <a href="https://www.oreilly.com/library/view/getting-started-with/9781786468574/ch04s04.html">https://www.oreilly.com/library/view/getting-started-with/9781786468574/ch04s04.html</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*NZP5ut5f-dziqxZgS7zwxw.jpeg" /><figcaption>Source: <a href="https://www.kdnuggets.com/2015/04/preventing-overfitting-neural-networks.html">https://www.kdnuggets.com/2015/04/preventing-overfitting-neural-networks.html</a></figcaption></figure><p>The above image is an illustration of different <em>approaches</em> one can take with an MLP to classify your data (more on these approaches in future articles!). Here I’d like to highlight the non-linearity in the data points (green &amp; red dots). As you can clearly see in these images, it is impossible for a single layer perceptron or a linear perceptron to classify these data points by plotting a straight line because there is NO SUCH straight line that will separate them. Hence the use of an MLP, while not ideal it does an extremely good job at classifying most of these data points correctly.</p><h4>MLPs in action!</h4><p>Just like the linear perceptron, MLPs are often applied to supervised learning problems where the network is able to learn from the data given. Due the presence of multiple layers in the network, more precise relationships and connections can be drawn and deduced from that data that it’s given. This process of making sense of the data is called <em>training</em>. And this training is achieved by adjusting variables such as the weights and biases with the goal of minimizing an error function.</p><p>As you can imagine, there are many ways one can train their <em>Artificial Neural Network (ANN) </em>and one approach that we will discuss here is the <strong>Backpropagation Algorithm</strong>.<br>Remember the first time you rode a bicycle? That probably came about after at least falling over 10 times. So essentially, each time you fell over your brain kept figuring out what your mistakes were and every time you kept trying your slowly fixed those mistakes until eventually you were able to ride mistake-free. <br>Your brain’s ability to learn is essentially what the backpropagation algorithm does. It corrects the mistakes made by the ANN.</p><p>Let’s say you feed an ANN with some data at the input, this flows through your network, random weights and biases are assigned to each node and an output is calculated. This output is compared with the data initially present, given that this is a supervise learning problem. This comparison will naturally give rise to an error between the network’s result and the expected result in the data. The backpropagation algorithm now “propagates” or sends this error back into the network and based on this, the weights are adjusted in order to minimize this error. This process continues until it reaches a stopping condition which can either be a specific error threshold or a specific number of training rounds. <br>Consider your network, <em>trained</em>! Now it is capable to dealing with new or unfamiliar inputs and will likely give results/predictions that are in keeping with the data it trained on.</p><h4>Further reading:</h4><p>For more details on the math and coding -</p><ul><li><a href="https://ujjwalkarn.me/2016/08/09/quick-intro-neural-networks/">A quick introduction to Neural Networks</a>.</li><li><a href="https://www.youtube.com/watch?v=qWK7yW8oS0I">The coding train on YouTube</a> (feel free to check out his other videos on Neural Networks too!).</li><li><a href="https://skymind.ai/wiki/multilayer-perceptron">A beginner’s guide to MLPs</a>.</li><li><a href="https://medium.com/@raulchandna/artificial-neuron-introduction-using-cricket-a-unicorn-and-logic-gates-9ae6a680b8d3">Logic gates explained in..Excel!</a></li><li><a href="https://towardsdatascience.com/neural-representation-of-logic-gates-df044ec922bc">Neural representation of AND and OR logic gates in Python</a>.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b42166dc0067" width="1" height="1" alt=""><hr><p><a href="https://medium.com/udacity-pytorch-challengers/multi-layer-perceptrons-mlps-b42166dc0067">Multi-Layer Perceptrons (MLPs)</a> was originally published in <a href="https://medium.com/udacity-pytorch-challengers">Udacity PyTorch Challengers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is overfitting?]]></title>
            <link>https://medium.com/udacity-pytorch-challengers/what-is-overfitting-f0e13f20b8fc?source=rss----c6e264a51486---4</link>
            <guid isPermaLink="false">https://medium.com/p/f0e13f20b8fc</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[pytorchudacityscholar]]></category>
            <category><![CDATA[overfitting]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[udacity]]></category>
            <dc:creator><![CDATA[Avinash]]></dc:creator>
            <pubDate>Sat, 05 Jan 2019 17:27:48 GMT</pubDate>
            <atom:updated>2019-01-05T17:27:48.161Z</atom:updated>
            <content:encoded><![CDATA[<p>If you have just started learning Machine Learning, you will often hear this term called overfitting, but what exactly it is?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NTa-SZdmO7th-_TpN_lMHA.png" /><figcaption>Image from Lesson 2, Overfitting and Underfitting from Udacity’s <a href="https://in.udacity.com/course/deep-learning-pytorch--ud188">Intro to Deep Learning with PyTorch</a></figcaption></figure><p>Before diving into overfitting, let’s first review what really is Machine Learning. In ML, we define a mathematical model, which goes through a bunch of our data, say like hundreds of pictures of flowers and comes up with a way to predict them. Now, whenever, we show a picture of a flower, it will guess which category it belonged to.</p><p>But what if it sees an entirely new flower image, which it had never seen before? Would it predict that too?</p><p>A good ML model can make near accurate predictions on the new data. This is called generalisation. The entire goal is that, it sees a bunch of images, generalises it so well that it can predict the new ones.</p><p>Now, when you are training your model with training data, you might end up training it too well. But what does that mean? The ML recognises by seeing the features from the image. In case of flowers, it might see curves, lines, colour patterns to recognise a flower. Now, when you overtrain it, the ML model gets used to the same patterns. And it cannot guess anything outside of it. Imagine when you are playing with a kid, you are teaching the kid about circles. Small ones, big ones, red ones etc. But if you show a square, the kid won’t figure out. Because it is overtrained on the circles.</p><p>The overfitting is the exactly same. When our model gets trained too much, it cannot generalise well. It cannot make good predictions, outside of the training data.</p><p>The following video from Udacity’s <a href="https://in.udacity.com/course/deep-learning-pytorch--ud188">Intro to Deep Learning with PyTorch</a> does a great job at explaining what are underfitting and overfitting:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fxj4PlXMsN-Y%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dxj4PlXMsN-Y&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fxj4PlXMsN-Y%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/454acc33d39eb510da4883da47c17990/href">https://medium.com/media/454acc33d39eb510da4883da47c17990/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f0e13f20b8fc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/udacity-pytorch-challengers/what-is-overfitting-f0e13f20b8fc">What is overfitting?</a> was originally published in <a href="https://medium.com/udacity-pytorch-challengers">Udacity PyTorch Challengers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Transfer Learning]]></title>
            <link>https://medium.com/udacity-pytorch-challengers/introduction-to-transfer-learning-22d2ebeb7dee?source=rss----c6e264a51486---4</link>
            <guid isPermaLink="false">https://medium.com/p/22d2ebeb7dee</guid>
            <category><![CDATA[image-recognition]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[pytorch]]></category>
            <dc:creator><![CDATA[Alan Choon]]></dc:creator>
            <pubDate>Sat, 05 Jan 2019 05:22:11 GMT</pubDate>
            <atom:updated>2019-01-05T05:22:11.611Z</atom:updated>
            <content:encoded><![CDATA[<p>This article is an introduction to transfer learning (TL) using <a href="http://pytorch.org/">PyTorch</a>. I will illustrate the concept in simple terms and present the tools used to perform TL, applied to an image recognition problem. <strong>The prerequisite for following this article is to have a basic understanding of Neural Networks (how is forward and back propagation performed, what is a loss function and how an optimizer updates the weights of a neural network.)</strong> By the end of this article, I hope to show how to effectively apply TL to image recognition problems.</p><h3>But first, what is Transfer Learning?</h3><p>Imagine if you had a model that has been trained for a certain problem, for example, we have been approached by a novice florist, to classify images of preordered flowers and tag their names so the orders can be fulfilled. We have trained the model and it has given us excellent results, we are happy now.</p><p>Now, a different problem arises. What if we want to classify images of cats and dogs now? Can we apply our previously trained model our new problem? The answer is, we sure can! It is analogous to learning by humans. The knowledge we gain from 1 domain can sometimes be applicable to another (think calculus to problems in physics).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/0*LeRwBCRAB6e1komC.png" /><figcaption><em>image from: Ellagrin/Shutterstock.com</em></figcaption></figure><p>However, the model cannot be applied verbatim just like that, it needs to learn even further to tweak the solution to solve the new problem. This is exactly the intuition behind Transfer Learning, we essentially reuse a pre-trained model and let it learn and optimize further to solve a new problem.</p><p>Now that we have an intuition of what TL is, let’s apply our knowledge to an existing image classification problem. The problem that we will be working on is classifying 102 species of flowers, the data can be obtained <a href="https://www.kaggle.com/wassimseifeddine/102flowersdataset">here</a>.</p><h3>Notebook Set-up</h3><p>Training a deep learning model to recognize images will take a long time in CPU. We can utilize a GPU instead for superior processing power.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/0*9ZbEb62syxvVqlkm.jpg" /></figure><p>How can we use a GPU? Introducing <a href="https://colab.research.google.com/notebooks/welcome.ipynb">Google Colab</a>. Google Colab is essentially Jupyter Notebook hosted on a cloud environment. This amazing tool provides free Tesla K80 GPU. You can save your Google Colab notebooks on a Google Drive folder, and sync your data and model files on Drive and Colab, saving space on your hard drive.</p><p>To get started, you can refer to this <a href="https://colab.research.google.com/notebooks/welcome.ipynb#scrollTo=1fr51oVCHRZU">link</a> for examples of code in Google Colab Notebook. Note that you would need a Google Account to gain access. Once you are ready, create a new Python 3 notebook. Click on Runtime in the Notebook and change Runtime Type to GPU for hardware accelerator.</p><p>Next, let’s mount our Google Drive, to access the images saved in Drive folder (Note that you have to save the images you want to classify in a Google Drive Folder).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XtgOyiOz0ho9jd54iR37iA.png" /></figure><p>You should be able to see your Google Drive folders on the left panel:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/263/1*0rPZS6jLLpvS0DgW5JVd9g.png" /></figure><p>Change working directory to folder containing images using the command:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OE6TqMvnmlPhTxTTRhNrtQ.png" /><figcaption>Here, pytorch_challenge is the folder containing my images.</figcaption></figure><h3>Importing required libraries</h3><p>The code below imports required libraries and checks if GPU is available. (pip install torch before importing).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*60nWW4gkCTHoyRxgvFX7ag.png" /></figure><p>torch refers to Python’s PyTorch library while PIL is Python’s Imaging Library which is used to represent an image.</p><p>We need to perform transformations to our data, namely adding noise to our train data and resizing both train and validation/test data to suit our model.</p><h3>Data Preprocessing</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CgOdP2dLU-ee1wYirVIxkQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kCiYjUWY2njUt3sDVoHu9Q.png" /></figure><p>We first define the transformations to the images we want to implement. train_transforms are performed on images in our training set while valid_transforms are performed on images in our validation set.</p><p>Essentially, we are randomly rotating, resizing and flipping our train images. All images are cropped to 224x224 to fit our model (to be trained).</p><p>datasets.ImageFolder looks into the folders in our train/valid/test directories and performs the transformations on the images in the folders, along with generating labels based on the subdirectories in the folders.</p><p>datasets.DataLoader takes the ImageFolder Objects, along with parameters such as batch_size and shuffle and returns a generator which can yield a tuple containing a torch tensor representing our images in each batch and their labels.</p><h3>Preparing our pre-trained model</h3><p>Next, we load our pre-trained model using models from torchvision</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z0xaLJ2wlLi5nGMdOZPrAw.png" /></figure><p>Our pre-trained model is Resnet152. (You can read more about Resnet <a href="https://medium.com/@14prakash/understanding-and-implementing-architectures-of-resnet-and-resnext-for-state-of-the-art-image-cf51669e1624">here</a>).</p><p>This is not the only choice of pre-trained models. For a full list of pre-trained models and their comparisons, refer to this <a href="https://pytorch.org/docs/stable/torchvision/models.html">link</a>.</p><p>To view the layers in the pre-trained model, simply run:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bYJCrKReGd19wDKTvNp1bQ.png" /><figcaption>Snapshot of Resnet-152 Layers</figcaption></figure><p>We freeze all parameters except for Layer4, The 2nd last layer (average pooling layer) and the fully connected layer. Freezing here refers to telling our model not to update weights in the layers during the optimization step when we train our model later on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6pjxc0JtYPjzQAeQdPNOHw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CkTD4NE6Ocx-ojFh9OaGpw.png" /></figure><p>The rationale for freezing the first few layers of our model is to reduce the number of parameters that need to be updated, for faster computing speed and to be more memory efficient.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LnCURSwzprB4fVcLUhPXMQ.png" /><figcaption>Without freezing, total updatable params: ~58M. With freezing, it is reduced to ~15M.</figcaption></figure><p>We need to replace the fully connected layer in the pre-trained model with a new layer, ensuring our outputs match the number of labels that we want (in this case 102).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AeRvy2IKs7_GviZ3lR8q9A.png" /></figure><h3>Retraining our Pre-trained model</h3><p>Define our loss and optimizer functions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T0jHhT2q6xdZyfe57sM-_A.png" /></figure><p>nn.CrossEntropyLoss criterion combines nn.LogSoftmax() and nn.NLLLoss().</p><p>It takes the outputs from LogSoftmax of our last layer and calculates the loss by taking their negative log likelihood.</p><p>The criterion can also be set to NLLLoss, however, we would have to set our last layer to nn.LogSoftmax in this case which is why we are sticking to this criterion for simplicity.</p><p>Here, Adadelta is selected as the optimizer as it can adapt the learning rate over time.</p><p>A learning rate scheduler is also used to decrease our learning rate by half every 10 epochs, to ensure that the model can learn smoothly without too many oscillations around the same loss.</p><p>The next step will be to go ahead and train the model. As the code to do this is quite lengthy, I will not be sharing the code here. You can find it on my github <a href="https://github.com/alanchn31/Udacity_Pytorch_Course/blob/master/Final%20Project%20(Transfer%20Learning)/Image_Classifier_Project_(Resnet_152).ipynb">repo</a>.</p><p>The main steps for training are: 1) for each epoch, forward propagate to obtain loss. 2) Using loss, perform backpropagation. 3) Update parameters using optimizer 4) At the end of each epoch, keep track of loss and compare with minimum loss. If loss decreases, save new loss and our model.</p><p>And that’s essentially how Transfer Learning is done! We freeze initial layers of a pre-trained model and unfreeze the last few layers to retrain the parameters. This can also be applied to other image recognition problems.</p><h3>Visualizing our Results:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/881/1*VvTSn8LV6xFh-lQ0coAuvA.png" /></figure><p>Within 34 epochs, our model is able to obtain a validation accuracy of 97%. Not bad at all!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/667/1*GdF6cqAj6AhMKxJCSuaKxQ.png" /><figcaption>Model is able to correctly identify a pink primrose</figcaption></figure><p>Alright, that’s the end of this article. Please leave a comment if there are any concepts or tips that I may have missed! Special thanks to Udacity and Facebook for providing the resources in assistance of the completion of this project.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=22d2ebeb7dee" width="1" height="1" alt=""><hr><p><a href="https://medium.com/udacity-pytorch-challengers/introduction-to-transfer-learning-22d2ebeb7dee">Introduction to Transfer Learning</a> was originally published in <a href="https://medium.com/udacity-pytorch-challengers">Udacity PyTorch Challengers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Set up Colab for Lab Challenge]]></title>
            <link>https://medium.com/udacity-pytorch-challengers/set-up-colab-for-lab-challenge-5ec171ac10b3?source=rss----c6e264a51486---4</link>
            <guid isPermaLink="false">https://medium.com/p/5ec171ac10b3</guid>
            <dc:creator><![CDATA[Wenjing Liu]]></dc:creator>
            <pubDate>Sat, 05 Jan 2019 00:18:41 GMT</pubDate>
            <atom:updated>2019-06-27T09:27:02.760Z</atom:updated>
            <content:encoded><![CDATA[<h3>Set up Colab for PyTorch Projects</h3><h3><strong>What is Colab?</strong></h3><p>Colab (short for Google Colaboratory) is a research tool for machine learning education and research. It is a Jupyter notebook environment that requires no setup to use. For more information, click <a href="https://research.google.com/colaboratory/faq.html">here</a>.</p><h3><strong>How to access Colab?</strong></h3><p>Go to your Google Drive, click on the “New” button in the upper left corner, then select “Colaboratory” to create a new file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PgfFmSR3rUZkEoUgOKDB1A.jpeg" /></figure><h3><strong>How to install PyTorch in Colab?</strong></h3><p>If you click on the arrow sign on the left side, and type “pytorch”, Colab will provide a code snippet to install <strong>PyTorch 0.4.1</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*25eIm-zS-HyvaS_XYU4V_A.jpeg" /></figure><p>Or you might want to install some other version, e.g. <strong>PyTroch 1.0.0</strong>, then you need to copy and paste the following code manually.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t--hGfYuQiNtYMB2VicDkw.jpeg" /></figure><h3><strong>How to set up Colab for Udacity Lab project?</strong></h3><p>If you are setting up Colab to train models for the final Lab project of Udacity PyTorch Challenge (Nov 2018-Jan 2019), you would need <strong>PyTorch 0.4.0</strong>. First run the following code block to install PyTorch 0.4.0 and PIL (pillow).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fuWXspb-wP0who2hhodyRQ.jpeg" /></figure><p>Restart the runtime by selecting the option “Runtime -&gt; Restart runtime” in the menu. Then run the following code block to check the versions. The default PIL version in Colab so far is 4.0.0, which could cause error “AttributeError: module ‘PIL.Image’ has no attribute ‘register_extensions’”. And if you don’t restart the runtime, the version won’t get updated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TxY_nKv8N9_LIlCmUUhjwA.jpeg" /></figure><p>After that, run the following code block to mount Google Drive. You will be asked to go to a URL to get your authorization code, which allows the Jupyter notebook to access your Google Drive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vkEt1wF-v_DI4TSPTzflRA.jpeg" /></figure><p>You could name it “drive” or something else. If it was “drive”, you’d be able to access any file in your Google Drive with path “drive/My Drive/&lt;filename&gt;”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MuQ2OVtdF4PwnFe9qdT85w.jpeg" /></figure><p>Now you have set up Colab for the Lab project.</p><h3><strong>How to check Colab GPU profile?</strong></h3><p>Run the following code to check the Colab GPU profile. If you want to check runtime information, select the option “Runtime -&gt; Manage sessions” in the menu.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yuq35VqZhMqytYt2cvuXsg.jpeg" /></figure><h3><strong>Why PyTorch 0.4.0?</strong></h3><p>Colab provides code snippet to install <strong>PyTorch 0.4.1</strong> (as mentioned before).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RhBBaydBkUEwvXRJ7Y9jWQ.jpeg" /></figure><p>However Udacity Lab uses <strong>PyTorch 0.4.0</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x2DXgN4R5zdI2L0sCYArnQ.jpeg" /></figure><p>If two PyTorch versions don’t match, you might encounter errors when uploading your trained model (model’s <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html">state_dict</a>) to Udacity Lab. Some suggested setting “strict=False” for method “load_state_dict()” to avoid the error. But I found <strong>the model weights would change</strong> if I did so, e.g. in <a href="https://github.com/Nov05/Udacity-PyTorch_Challenge/blob/master/Playground/2018-12-06_State_Dict_Comparison.ipynb">this case</a> half of the weights changed after the upload. Also some reported that her/his accuracy dropped dramatically from 90% in local machine to below 20% in Udacity Lab (it didn’t pass the test). Hence I would suggest setting the right PyTorch version for the Udacity Lab project. Good luck!</p><p><strong>P.S. To get all the code in this post, you can visit </strong><a href="https://colab.research.google.com/drive/1dq9Hkf0qFuNxz8BHU5aAJZqwQLjs7yS3"><strong>here</strong></a><strong>.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ec171ac10b3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/udacity-pytorch-challengers/set-up-colab-for-lab-challenge-5ec171ac10b3">Set up Colab for Lab Challenge</a> was originally published in <a href="https://medium.com/udacity-pytorch-challengers">Udacity PyTorch Challengers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>