<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Deep Learning Demystified - Medium]]></title>
        <description><![CDATA[Simple intuitive explanations for  everything Deep Learning. From basic concepts to cutting edge advances. - Medium]]></description>
        <link>https://medium.com/deep-learning-demystified?source=rss----ce61a9254205---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Deep Learning Demystified - Medium</title>
            <link>https://medium.com/deep-learning-demystified?source=rss----ce61a9254205---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 05:36:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/deep-learning-demystified" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Deep CV: Advanced Convolutional Layers]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/deep-learning-demystified/deep-cv-advanced-convolutional-layers-f358a7e4699f?source=rss----ce61a9254205---4"><img src="https://cdn-images-1.medium.com/max/1200/1*_7qBwPgLGkOFVDR48oxkbg.gif" width="1200"></a></p><p class="medium-feed-snippet">Visual explanation of Advanced Convolutional Layers. Explains Depthwise Separable, Deconvolutional, Dilated and Grouped Convolutional&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/deep-learning-demystified/deep-cv-advanced-convolutional-layers-f358a7e4699f?source=rss----ce61a9254205---4">Continue reading on Deep Learning Demystified »</a></p></div>]]></description>
            <link>https://medium.com/deep-learning-demystified/deep-cv-advanced-convolutional-layers-f358a7e4699f?source=rss----ce61a9254205---4</link>
            <guid isPermaLink="false">https://medium.com/p/f358a7e4699f</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Harsha Bommana]]></dc:creator>
            <pubDate>Tue, 07 Jul 2020 07:12:09 GMT</pubDate>
            <atom:updated>2020-07-07T07:12:09.796Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[TensorFlow 2: Model Building with tf.keras]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/deep-learning-demystified/tensorflow-2-model-building-with-tf-keras-1a9b050373ed?source=rss----ce61a9254205---4"><img src="https://cdn-images-1.medium.com/max/2250/1*mFX-wJ6zw0ttT5i1nvvTcw.jpeg" width="2250"></a></p><p class="medium-feed-snippet">Getting started with using TensorFlow 2&#x2019;s tf.keras API to build Deep Learning models.</p><p class="medium-feed-link"><a href="https://medium.com/deep-learning-demystified/tensorflow-2-model-building-with-tf-keras-1a9b050373ed?source=rss----ce61a9254205---4">Continue reading on Deep Learning Demystified »</a></p></div>]]></description>
            <link>https://medium.com/deep-learning-demystified/tensorflow-2-model-building-with-tf-keras-1a9b050373ed?source=rss----ce61a9254205---4</link>
            <guid isPermaLink="false">https://medium.com/p/1a9b050373ed</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Harsha Bommana]]></dc:creator>
            <pubDate>Sun, 21 Jun 2020 06:09:25 GMT</pubDate>
            <atom:updated>2020-06-21T06:09:24.941Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Deep Learning Demystified | Official Website Launch]]></title>
            <link>https://medium.com/deep-learning-demystified/deep-learning-demystified-official-website-launch-76e00696c299?source=rss----ce61a9254205---4</link>
            <guid isPermaLink="false">https://medium.com/p/76e00696c299</guid>
            <dc:creator><![CDATA[Harsha Bommana]]></dc:creator>
            <pubDate>Sat, 25 Apr 2020 16:04:38 GMT</pubDate>
            <atom:updated>2020-04-25T16:04:38.893Z</atom:updated>
            <content:encoded><![CDATA[<p>Greetings from Deep Learning Demystified,</p><p>I hope all are safe and sound amidst this COVID-19 situation. While we are staying safe indoors, let’s continue upskilling ourselves in Deep Learning.</p><p>We are proud to announce the official launch of our website: <a href="https://deeplearningdemystified.com/">https://deeplearningdemystified.com</a>. You can find all our articles there.</p><p>We would also like to announce that we will be coming up with some very detailed Tensorflow 2.0 articles very soon so do stay tuned for those!</p><p>We have also launched an Instagram page where we put up deep learning related content, you can find it here: <a href="https://www.instagram.com/deeplearningdemystified/">https://www.instagram.com/deeplearningdemystified/</a>.</p><p>Thank you all for your support!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=76e00696c299" width="1" height="1" alt=""><hr><p><a href="https://medium.com/deep-learning-demystified/deep-learning-demystified-official-website-launch-76e00696c299">Deep Learning Demystified | Official Website Launch</a> was originally published in <a href="https://medium.com/deep-learning-demystified">Deep Learning Demystified</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deep CV: How do CNNs work?]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/deep-learning-demystified/deep-cv-how-do-cnns-work-47ad18507811?source=rss----ce61a9254205---4"><img src="https://cdn-images-1.medium.com/max/806/1*WdfxfOVtq4fKFVyxfk_xFQ.png" width="806"></a></p><p class="medium-feed-snippet">Intuitive and visual explanation of Convolutional Neural Networks.</p><p class="medium-feed-link"><a href="https://medium.com/deep-learning-demystified/deep-cv-how-do-cnns-work-47ad18507811?source=rss----ce61a9254205---4">Continue reading on Deep Learning Demystified »</a></p></div>]]></description>
            <link>https://medium.com/deep-learning-demystified/deep-cv-how-do-cnns-work-47ad18507811?source=rss----ce61a9254205---4</link>
            <guid isPermaLink="false">https://medium.com/p/47ad18507811</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Harsha Bommana]]></dc:creator>
            <pubDate>Thu, 26 Mar 2020 20:57:44 GMT</pubDate>
            <atom:updated>2020-04-25T00:46:46.394Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Deep NLP: Sequential Models with RNNs]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/deep-learning-demystified/deep-nlp-sequential-models-with-rnns-94c89e4c7db0?source=rss----ce61a9254205---4"><img src="https://cdn-images-1.medium.com/max/2600/1*faeoOZ62VRPYeD9W2etWmw.png" width="2667"></a></p><p class="medium-feed-snippet">RNNs are the entry point to using Deep Learning for Natural Language Processing. Learn exactly how they work.</p><p class="medium-feed-link"><a href="https://medium.com/deep-learning-demystified/deep-nlp-sequential-models-with-rnns-94c89e4c7db0?source=rss----ce61a9254205---4">Continue reading on Deep Learning Demystified »</a></p></div>]]></description>
            <link>https://medium.com/deep-learning-demystified/deep-nlp-sequential-models-with-rnns-94c89e4c7db0?source=rss----ce61a9254205---4</link>
            <guid isPermaLink="false">https://medium.com/p/94c89e4c7db0</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Harsha Bommana]]></dc:creator>
            <pubDate>Thu, 27 Feb 2020 16:58:14 GMT</pubDate>
            <atom:updated>2020-04-25T00:47:10.349Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Deep NLP: Word Vectors with Word2Vec]]></title>
            <link>https://medium.com/deep-learning-demystified/deep-nlp-word-vectors-with-word2vec-d62cb29b40b3?source=rss----ce61a9254205---4</link>
            <guid isPermaLink="false">https://medium.com/p/d62cb29b40b3</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Harsha Bommana]]></dc:creator>
            <pubDate>Thu, 21 Nov 2019 07:11:28 GMT</pubDate>
            <atom:updated>2020-04-25T00:48:50.318Z</atom:updated>
            <content:encoded><![CDATA[<p>Using deep learning for natural language processing has some amazing applications which have been proven to be performing very well. The core concept is to feed human readable sentences into neural networks so that the models can extract some sort of information from them.</p><p>This can be used for many tasks, some of the popular ones are <strong>sentiment analysis </strong>and <strong>natural entity recognition. </strong>Sentiment analysis is where we try to predict whether a given text has a negative or positive emotion overall. We feed the text into a neural network and it will output a value between 0 to 1 to indicate the overall emotion. In natural entity recognition, we try to extract the key entities in a given text. Some examples of types of entities are names, addresses, organizations, etc. Given some text, the neural network will have to extract all the entities available. For example if we give the sentence: “Josh works in Google”, it should tell us that “Josh” is a name and “Google” is an organization.</p><p>There are many such tasks in the world of NLP. But before we dive into implementing these tasks using deep learning, we first need to understand how to <strong>preprocess </strong>textual data for our neural networks. Neural networks can take <strong>numbers </strong>as an input, but not raw text. Hence, we need to figure out a way to convert these words into a numerical format. The traditional method we used to follow is <strong>one hot encoding</strong>. But nowadays using <strong>word vectors </strong>is a much better way to represent textual data for many reasons. We will discuss both these techniques in depth and compare them. First let’s discuss one hot encoding.</p><h3>One Hot Encoding</h3><p>In one hot encoding, we essentially represent each word as a <strong>vector of length V</strong>, where V is the total number of unique words available in the entire textual data. V is also called the <strong>vocabulary count</strong>. Each word’s one hot encoded vector is essentially a binary vector with the value 1 being in a <em>unique index</em> for each word and the value 0 being in every other index of the vector. Let’s visualize this.</p><p>Suppose our data comprises of the following 2 sentences:</p><ul><li>Deep learning is hard</li><li>Deep learning is fun</li></ul><p>Here, the value of V is 5, because there are 5 unique words: [‘Deep’, ‘learning’, ‘is’, ‘hard’, ‘fun’]. Now to represent each word, we will use a vector of length 5.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*tTUfHXVrq7hFC4j3herRQw.png" /><figcaption>One hot encoding example</figcaption></figure><p>This is how we represent words as numbers using one hot encoding. Each and every word in the dataset has a corresponding one hot encoded vector which is unique. This is a simple and straightforward approach to convert all the words in a set of data into numbers and is one of the first methods implemented for this purpose, however this method has many issues.</p><p>Firstly, the size of each word’s one hot encoded vector will always be V, which is the size of the entire vocabulary. In the example it was 5 but usually the value of V can reach 10k or even 100k. This means we need a <strong>huge vector</strong> just to represent a <strong>single word</strong>, which can lead to excessive memory usage while representing text as vectors.</p><p>Secondly, the index which is assigned to each word does not hold any <strong>semantic meaning</strong>, it is merely an arbitrary value assigned to it. When we consider the vectors for two words, we would ideally want the vectors of <strong>similar meaning</strong> to have <strong>similar vectors</strong>. For example, it would be good if the vector for “Dog” and “Cat” are close to each other since the words share certain amount of similarity. However, with one hot encoded vectors, the vectors for “Dog” and “Cat” are just as close to each other as “Dog” and “Computer”. Hence the neural network has to work really hard to understand each word since they are being treated as completely <strong>isolated entities</strong>. The usage of word vectors aims to resolve both of these issues.</p><blockquote>One hot encoded vectors take up excessive memory and they do not hold any semantic information of the word. Word vectors aim to resolve both of these problems.</blockquote><h3>Word Vectors</h3><p>Word vectors are essentially a much more efficient method of representing words. Word vectors take up much lesser space than one hot encoded vectors and they also hold semantic information regarding the word. These are the key features of word vectors. Before diving into the method of finding word vectors, we need to understand the one core principle behind the concept of word vectors:</p><blockquote>Similar words occur more frequently together than dissimilar words.</blockquote><p>Just think about this for a while, does it make sense? When a word occurs within the vicinity of another word, it doesn’t always mean it has a similar meaning, but when we consider the <strong>frequency of words</strong> which are found close together, we find that words of <strong>similar meaning</strong> are often <strong>found together</strong>.</p><p>For example, the word “Dog” will be found within the vicinity of the word “Cat” a lot more frequently than it will be found within the vicinity of the word “Computer”, this is because “Dog” shares certain <strong>semantic similarity</strong> with “Cat” and there will hence be many sentences which have both “Dog” and “Cat”. This is the key factor which deep learning researchers have exploited to come up with word vectors. We will now discuss one of the first implementations of word vectors: Word2Vec by Google.</p><h3>Word2Vec</h3><p>In Word2Vec, there are two different architectures: CBOW and Skip Gram. We will first go through Skip Gram architecture, after which, understanding CBOW will be much easier.</p><p>The first thing we do for Word2Vec is to collect word co occurrence data. Basically we need a set of data telling us which words are occurring close to a certain word. We use something called as a <strong>context window </strong>for doing this.</p><p>Let’s consider the sentence: “Deep Learning is very hard and fun”. We need to set something knows as <strong>window size. </strong>Let’s assume it’s 2 for this case. What we do is iterate over all the words in the given data, which in this case is just 1 sentence, and then consider a window of words which surround it. Here since our window size is 2, we will consider 2 words behind the word and 2 words after the word. Hence for each word, we will get 4 words associated with it. We will do this for each and every word in the data and collect the word pairs. Let’s visualize this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/555/1*wRETdklL34qSk8B6CGMUOA.png" /><figcaption>Context Window example</figcaption></figure><blockquote>As you can see for the words near the edges, we have lesser words to consider and hence number of context words will be lesser than 4</blockquote><p>As we are passing the context window through the text data, we find all word pairs of target and context words to form a dataset in the format of (target word, context word). For the sentence above, it will look like this:</p><p><strong>1st Window pairs:</strong> (Deep, Learning), (Deep, is)</p><p><strong>2nd Window pairs: </strong>(Learning, Deep), (Learning, is), (Learning, very)</p><p><strong>3rd Window pairs:</strong> (is, Deep), (is, Learning), (is, very), (is, hard)</p><p>And so on. At the end our target word vs context word data set is going to look like this:</p><blockquote>(Deep, Learning), (Deep, is), (Learning, Deep), (Learning, is), (Learning, very), (is, Deep), (is, Learning), (is, very), (is, hard), (very, learning), (very, is), (very, hard), (very, and), (hard, is), (hard, very), (hard, and), (hard, fun), (and, very), (and, hard), (and, fun), (fun, hard), (fun, and)</blockquote><p>This can be considered as our <em>“training data”</em> for word2vec.</p><p>In the Skip Gram model, we try to predict each <strong>context word</strong> <em>given</em> a <strong>target word</strong>. We use a neural network for the prediction task. The input to the neural network will be a one hot encoded version of the target word and the output is the one hot encoded version of the context word. Hence the size of input and output layers is V (vocabulary count). This neural network has only 1 hidden layer in the middle, the <strong>size of this hidden layer</strong><em> </em>determines the <strong>size of word vectors</strong> we wish to have at the end. We will consider this as 300 for now.</p><p>Let’s say V is 10k and one of the target, context word pairs in our dataset is (Deep, Learning). The word2vec training will look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*-88xENJ8uSCLbwUO-AK0IA.png" /><figcaption>Skip Gram architecture in Word2Vec</figcaption></figure><p>Since this neural network has a total of 3 layers, there will be only 2 weight matrices for the network, W1 and W2. W1 will have dimensions of 10000*300 and W2 will have dimensions of 300*10000. These two weight matrices will play an important role in calculating word vectors and we will discuss them more later.</p><p>For our entire (target word, context word) dataset which we have collected from the original textual data, we will pass each pair into the neural network and train it. Essentially the task which the neural network is trying to do here is to guess which <strong>context words</strong> can appear <em>given</em> a <strong>target word</strong>. After training the neural network, if we input any target word into the neural network, it will give a vector output which represents the words which have a high probability of appearing near the given word.</p><p>For CBOW the only difference is that we try to predict the target word given the context words, essentially we just invert the skip gram model to get the CBOW model. It looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*DhLPK1UIqjyOFEPMNVYWbA.png" /><figcaption>CBOW Architecture in Word2Vec</figcaption></figure><p>Here when we give a vector representation of a <strong>group of context words</strong>, we will get the most appropriate target word which will be within the vicinity of those words. For example, if we give the sentence: Deep _____ is very hard, where [“Deep”, “is”, “very”, “hard”] represents the context words, the neural network should hopefully give “Learning” as the <strong>output target word</strong>. This is the core task the neural network tries to train for in the case of CBOW.</p><p>Now we know the task of the neural network in word2vec, but this still doesn’t help us get the word vectors we require. Now we have to understand that the neural network is doing this task so that it <strong>distills and captures semantic information</strong> about words by training on their relationship with other similar words. As the neural network trains on all the target and context word pairs in the dataset, it will eventually start understanding which words are <strong>semantically similar</strong> to which other words. This happens because of the phenomenon we discussed earlier where similar words occur together more frequently than dissimilar words. Hence the neural network will be exposed to (Dog, Cat) way more times than it sees (Dog, Computer), hence this semantic similarity is captured by it.</p><p>The neural network is capturing a lot of information regarding semantics of a given word, but how do we extract this information? Whenever a neural network is training, we can say that the weight matrices in between the layers store the <em>“information”</em> or <em>“patterns”</em> which the neural network has learnt. The values in the weight matrices don’t necessarily make any sense to us as humans but whatever a neural network learns is represented in the weights.</p><p>To extract the word vectors from the weights of the neural network, we generally consider W1, which is also called the <em>“Embedding Layer”</em>. There are some implementations which consider the weights of W2 as well but for now let’s look at how to get the vectors from the first weight matrix.</p><p>When we are training the neural network, the first thing which happens is the multiplication of the input layer with the first weight matrix. This is a visual of that operation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*LRULsBkpd7gL6erJzS3u7w.png" /><figcaption>Neural network first layer multiplication operation (Input Layer * W1)</figcaption></figure><p>If we represent the weights in a matrix form, we get a matrix with 10k rows and 300 columns. Since the input is a one hot encoded vector ([0,0,…,1,…,0]) which has one 1 in the vector, when we multiply the weight matrix with a one hot encoded vector, <strong>one row</strong> in the matrix will <strong>retain it’s values</strong> and all other rows will <strong>become 0</strong>. Hence the values in the hidden layer will simply be the values of this 1 row.</p><p>Hence, when we are using a particular word, let’s say the nth word of the vocabulary, we will <em>only</em> be <strong>affecting the nth row in the weight matrix</strong> because all other rows <strong>becomes 0</strong>. Therefore we can say that the nth row in the weight matrix contains all the <strong>trained information</strong> regarding the nth word and this row will be the<strong> trained word vector</strong> for the nth word. This is how we extract word vectors from a trained word2vec neural network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*1O5gLhOg25HviI8bwZxV4g.png" /></figure><p>That covers the fundamentals of word vectors. Word2Vec is just one of the several methods out there today used for generating word vectors for NLP and we will discuss more methods in the future. Here we have discussed how to preprocess text into a format compatible with neural networks, and in the next few articles on Deep NLP we will look at how we use these word vectors to perform several different tasks in NLP. Thanks for reading!</p><p>Read more Deep Learning articles at <a href="https://deeplearningdemystified.com/">https://deeplearningdemystified.com</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d62cb29b40b3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/deep-learning-demystified/deep-nlp-word-vectors-with-word2vec-d62cb29b40b3">Deep NLP: Word Vectors with Word2Vec</a> was originally published in <a href="https://medium.com/deep-learning-demystified">Deep Learning Demystified</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Generalization in Neural Networks]]></title>
            <link>https://medium.com/deep-learning-demystified/generalization-in-neural-networks-7765ee42ac23?source=rss----ce61a9254205---4</link>
            <guid isPermaLink="false">https://medium.com/p/7765ee42ac23</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Harsha Bommana]]></dc:creator>
            <pubDate>Thu, 17 Oct 2019 07:24:56 GMT</pubDate>
            <atom:updated>2020-04-25T00:49:06.764Z</atom:updated>
            <content:encoded><![CDATA[<p>Whenever we train our own Neural Networks, we need to take care of something called the <strong>generalization </strong>of the Neural Network. This essentially means how good our model is at learning from the given data and applying the learnt information elsewhere.</p><p>When training a neural network, there’s going to be some data which the Neural Network <strong>trains on, </strong>and there’s going to be some data reserved for checking the performance of the Neural Network. If the Neural Network performs well on the data which it <strong>has not </strong>trained on, we can say it has <strong>generalized well</strong> on the given data. Let’s understand this with an example.</p><p>Suppose we are training a neural network which should tell us if a given image has a dog or not. Let’s assume we have several pictures of dogs, each dog belonging to a certain breed, and there are 12 total breeds within those pictures. I’m going to keep all the images of 10 breeds of dogs for training and the remaining images of the 2 breeds will be kept aside for now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*pUl4Nrxxk4sGkpsrFGK5PQ.png" /><figcaption>Dogs training testing data split</figcaption></figure><p>Now before going to the Deep Learning side of things, let’s look at this from a human perspective. Let’s consider a human being who has never seen a dog in their entire life (just for the sake of an example). Now we will show this human the 10 breeds of dogs and tell them that these are dogs. After this, if we show them the other 2 breeds, will they be able to tell that they are also dogs? Well hopefully they should, 10 breeds should be enough to understand and identify the unique features of a dog. This concept of learning from some data and <strong>correctly </strong>applying the gained knowledge on other data is called <strong>generalization.</strong></p><p>Coming back to deep learning, our aim is to make the neural network learn as effectively from the given data as possible. If we successfully make the neural network understand that the <strong>other 2 breeds</strong> are <strong>also </strong>dogs, then we have trained a very <strong>general</strong> neural network and it will perform really well in the real world.</p><p>This is actually easier said than done and training a general neural network is one of the most frustrating tasks of a deep learning practitioner. This is because of a phenomenon in neural networks called <strong>overfitting.</strong> If the neural network trains on the 10 breeds of dogs and refuses to classify the other 2 breeds of dogs as dogs, then this neural network has <strong>overfit </strong>on the training data. What this means is that the neural network has <strong>memorized</strong> those 10 breeds of dogs and considers only them to be dogs. Due to this, it fails to form a <em>general understanding</em> of what dogs look like. Combating this issue while training Neural Networks is what we are going to be looking at in this article.</p><blockquote>Now we don’t actually have the liberty to divide all our data on a basis like breed, instead we will simply split all the data. One part of the data, usually the bigger part (around 80–90%), will be used for training the model, and the rest will be used to test it. Our objective is to make sure that the performance on the testing data is around the same as the performance on the training data. We use metrics like loss and accuracy to measure this performance.</blockquote><p>There are certain aspects of neural networks which we can control in order to prevent overfitting. Let’s go through them one by one. The first thing is <strong>number of parameters.</strong></p><h4>Number of Parameters</h4><p>In a neural network, the number of parameters essentially means the number of weights. This is going to be directly proportional to the number of <strong>layers </strong>and the number of <strong>neurons </strong>in each layer. The relationship between number of parameters and overfitting is as follows: the <strong>more </strong>the parameters, the <strong>more </strong>the chance of overfitting. I’ll explain why.</p><p>We need to define our problem in terms of <strong>complexity. </strong>A very complex dataset would require a very complex function to successfully understand and represent it. Mathematically speaking, we can somewhat associate complexity with <strong>non linearity. </strong>Let’s recall the neural network formula.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/607/1*2_yYFsppKW8R2Ey_LHiKMA.png" /></figure><p>Over here, W1, W2 and W3 are the weight matrices of this neural network. Now what we need to pay attention to is the <strong>activation functions </strong>in the equation which is applied at every layer. Because of these activation functions, each layer is <strong>nonlinearly connected </strong>with the next layer.</p><p>The output of the first layer is f(W_1*X) <em>(Let it be L1)</em>, the output of the second layer is f(W_2*L1). As you can see here, because of the activation function (f), the output of the second layer has a <strong>nonlinear</strong> relationship with the first layer. So at the end of the neural network, the final value Y will have a certain degree of nonlinearity with respect to the input X depending on the <strong>number of layers </strong>in the neural network.</p><blockquote>The <strong>more the number of layers</strong>, the <strong>more the number of activation functions</strong> disrupting the linearity between the layers, and hence the <strong>more the nonlinearity</strong>.</blockquote><p>Because of this relationship, we can say that our neural network becomes <strong>more complex</strong> if it has <strong>more layers</strong> and <strong>more nodes</strong> in each layer. Hence we need to adjust our parameters based on the complexity of our data. There is no definite way of doing this except for repeated experimentation and comparing of results.</p><p>In a given experiment, if the test score is much <strong>lower </strong>than the training score, then the <strong>model has overfit</strong> and that means the neural network has <strong>too many parameters</strong> for the given data. This basically means that the neural network is <strong>too complex </strong>for the given data and needs to be simplified. If the test score is around the same as the training score, then the model has <strong>generalized</strong>, but this does not mean that we have reached the maximum potential of neural networks. If we increase the parameters the performance will increase, but it also might overfit. So we need to keep experimenting to optimize the number of parameters by balancing <strong>performance with generalization.</strong></p><blockquote>We need to match the <strong>neural network’s complexity with our data complexity. If the neural network is too complex, it will start memorizing the training data instead of having a general understanding of the data, hence causing overfitting.</strong></blockquote><p>Usually how deep learning practitioners go about this is to first train a neural network with a sufficiently high number of parameters such that the model will overfit. So initially we try to get a model which fits extremely well on the training data. Next we try and reduce the number of parameters iteratively till the model <strong>stops overfitting</strong>, this can be considered as an <strong>optimal neural network</strong>. Another technique which we can use to prevent overfitting is using <strong>dropout neurons.</strong></p><h4>Dropout Neurons</h4><p>In Neural Networks, adding dropout neurons is one of the most popular and effective ways to reduce overfitting in neural networks. What happens in dropout is that essentially each neuron in the network has a certain probability of completely <strong>dropping out</strong> from the network. This means that at a particular instant, there will be certain neurons which will <strong>not be connected</strong> to any <strong>other neuron</strong> in the network. Here’s a visual example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/1*3eb9xQnRVk1ToWayMq_W2w.png" /></figure><p>At every instant during training, a different set of neurons will be dropped out in a <strong>random fashion</strong>. Hence we can say that at each instant we are effectively training a certain <strong>subset neural network </strong>which has fewer neurons than the original neural network. This subset neural network will change each and every time because of the random nature of the dropout neurons.</p><p>What essentially happens here is that while we train a neural network with dropout neurons, we are basically training <strong>many smaller</strong> subset neural networks and since the weights are a part of the original neural network, the final weights of the neural network can be considered as an <strong>average </strong>of all the <em>corresponding</em> subset neural network weights. Here’s a basic visualization of what’s going on:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LzJWhtBF67XtyHEOJTV3_Q.png" /></figure><p>This is how dropout neurons work in a neural network, but <em>why </em>does dropout prevent overfitting? There are two main reasons to this:</p><p>The first reason is that dropout neurons promote <strong>neuron independence. </strong>Because of the fact that the neurons surrounding a particular neuron may or may not exist during a certain instant, that neuron cannot rely on those neurons which surround it. Hence it will be <em>forced</em> to be <strong>more independent</strong> while training.</p><p>The second reason is that because of dropout, we are essentially training multiple smaller neural networks at once. In general, if we train multiple models and average their weights, the performance usually increases because of the <strong>accumulation of independent learnings</strong> of each neural network, however this is an expensive process since we need to define multiple neural networks and train them individually. However in the case of dropout, this does the <em>same thing</em> while we need <strong>only one</strong> neural network from which we are training <strong>multiple possible configurations</strong> of subset neural networks.</p><blockquote>Training multiple neural networks and aggregating their learnings is called “ensembling” and usually boosts performance. Using dropout essentially does this with while having only 1 neural network.</blockquote><p>The next technique for reducing overfitting is <strong>weight regularization.</strong></p><h4>Weight Regularization</h4><p>While training neural networks, there is a possibility that the value of certain weights can become very large. This happens because these weights are <em>focusing</em> on certain <strong>features</strong> in the training data which is causing them to increase in value <strong>continuously</strong> throughout the training process. Because of this, the network overfits on the training data.</p><p>We don’t need the weight to continuously increase to capture a certain pattern, instead it’s fine if they have a value which is higher than the other weights on a <strong>relative basis</strong>. But during the training process while a neural network is trained on the data over multiple iterations, the weights have a possibility of <strong>constantly increasing</strong> in value till they become huge, which is unnecessary.</p><p>One of the other reasons why huge weights are bad for a neural network is because of the increased <strong>input output variance. </strong>Basically when there is a huge weight in the network, it is very <em>susceptible </em>to <strong>small changes</strong> in the input, but the neural network should essentially output the <em>same thing</em> for <em>similar inputs</em>. When we have huge weights, even when we keep two separate data inputs which are very similar, there is a chance that the outputs vastly differ. This causes many <strong>incorrect predictions</strong> to occur on the testing data, hence <strong>decreasing</strong> the generalization of the neural network.</p><blockquote>The general rule of weights in neural networks is that the higher the weights in the neural network, the more complex the neural network. Because of this, neural networks having higher weights generally tend to overfit.</blockquote><p>So basically we need to limit the growth of the weights so that they don’t grow too much, but how exactly do we go about this? Neural networks try to minimize the loss while training, so we can try to include a part of the weights in that loss function, so that weights are <strong>also minimized </strong>while training, but of course decreasing the loss in the first priority.</p><p>There are two methods of doing this, they are called the L1 and L2 regularization. In L1 we take a small part of the sum of all the <strong>absolute values </strong>of the weights in the network. In L2, we take a small part of the sum of all the <strong>squared values </strong>of the weights in the network. We just <strong>add</strong> this expression to the overall loss function of the neural network. The equations are as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/1*UvG7GVQ8rVU6ChkToSTqpg.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/221/1*XsWYgNW6G__ja7Fuuz8t9Q.gif" /></figure><p>Here lambda is a value which allows us to alter the <strong>extent </strong>of weight change. We basically just add the L1 or L2 terms to the loss function of the neural net so that the network will <strong>also try to minimize </strong>these terms. By adding L1 or L2 regularization, the network will limit the growth of it’s weights since the magnitude of the weights are a part of the loss function and the network always tries to minimize the loss function. Let’s highlight some of the differences between L1 and L2.</p><p>With L1 regularization, while a weight is decreasing due to regularization, L1 tries to push it down completely to <strong>zero. </strong>Hence unimportant weights which aren’t contributing much to the neural network will eventually become zero. However in the case of L2, since the square function becomes inversely proportional for values below 1, the weights aren’t pushed to zero but they are pushed to <strong>small values. </strong>Hence the unimportant weights have much lower values than the rest.</p><p>That covers the important methods of preventing overfitting. In deep learning we usually use a mix of these methods to improve the performance of our neural networks and improving the generalization of our models. Thanks for reading!</p><p>Read more Deep Learning articles at <a href="https://deeplearningdemystified.com/">https://deeplearningdemystified.com</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7765ee42ac23" width="1" height="1" alt=""><hr><p><a href="https://medium.com/deep-learning-demystified/generalization-in-neural-networks-7765ee42ac23">Generalization in Neural Networks</a> was originally published in <a href="https://medium.com/deep-learning-demystified">Deep Learning Demystified</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Optimizers]]></title>
            <link>https://medium.com/deep-learning-demystified/https-medium-com-deep-learning-demystified-understanding-optimizers-313b787a69fe?source=rss----ce61a9254205---4</link>
            <guid isPermaLink="false">https://medium.com/p/313b787a69fe</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Harsha Bommana]]></dc:creator>
            <pubDate>Mon, 07 Oct 2019 05:16:16 GMT</pubDate>
            <atom:updated>2020-04-25T00:49:40.187Z</atom:updated>
            <content:encoded><![CDATA[<p>In deep learning we have the concept of loss, which tells us how poorly the model is performing at that current instant. Now we need to use this loss to <strong>train </strong>our network such that it performs better. Essentially what we need to do is to take the loss and try to <strong>minimize </strong>it, because a lower loss means our model is going to perform better. The process of minimizing (or maximizing) any mathematical expression is called <strong>optimization</strong> and we need now see how we can use these optimization methods for neural networks.</p><p>In a neural network, we have many <strong>weights </strong>in between each layer. We have to understand that <strong>each and every</strong> weight in the network will affect the output of the network in some way because they are all directly or indirectly connected to the output.</p><p>Hence we can say that if we change any particular weight in the neural network, the output of the network will also change.</p><p>As you can see in the illustration below, we are taking 3 different scenarios. In each scenario, we are selecting some random weight and we are altering it. We are also seeing what parts of the neural network get affected and which don’t get affected <strong>after </strong>we change the selected weight. In all 3 scenarios, there is at least one affected node in the final layer of the network. Since all nodes of the last layer are also connected to the output node, it is safe to say that as long as some part of the last layer gets affected, the output node will also be affected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*6Rrf0TC87tl7yJV775Lk8Q.jpeg" /><figcaption>Visualizing which parts of a network altering particular weights will affect</figcaption></figure><p>From the diagram we can also observe that the weights <strong>further away</strong> from the output node (closer to the start of the network), affect more nodes in between. Hence we can say that they affect the output very <strong>indirectly </strong>since there are a lot of weights between them and the output. The weights which are closer to the output affect fewer nodes in between, hence they have a more <strong>direct </strong>impact on the output node.</p><p>Now that we understand how to change the output of the network by changing the weights, let’s go ahead to understand how we can minimize the loss. Changing the weights <strong>changes the output. </strong>Changing the output <strong>changes the loss</strong>, since loss is a function of the predicted value (Y_pred), which is basically the output of the network. Hence we can say that changing the weights will ultimately <strong>change the loss.</strong></p><p>We have established the relationship between the weights and the final loss, but so far we have only talked about <strong>changing. </strong>Changing can mean increase or decrease, but we need to <strong>decrease </strong>the loss. So now we need to see how to <strong>change </strong>the weights in such a way that the loss <strong>decreases. </strong>This process is called optimization.</p><p>Looking at it from a mathematical perspective, we can do this by using <strong>partial derivatives. </strong>A partial derivative will allow us to understand how two mathematical expressions <strong>affect each other. </strong>Let us take X and Y, which are connected by some arbitrary mathematical relationship. If we find the partial derivative of X <strong>with respect to</strong> Y, we can understand how changing X will affect Y. If the partial derivative is <strong>positive, </strong>that means <strong>increasing X</strong> will also <strong>increase Y</strong>. If it’s <strong>negative </strong>that means <strong>increasing X</strong> will <strong>decrease Y</strong>.</p><p>Hence we need to find the partial derivatives of each and every <strong>weight </strong>in the neural network <strong>with respect to </strong>the loss. At a particular instant, if the weight’s partial derivative is <strong>positive, </strong>then we will <strong>decrease </strong>that weight in order to decrease the loss. If the partial derivative is <strong>negative, </strong>then we will <strong>increase </strong>that weight in order to decrease the loss. Our ultimate goal is to decrease the loss afterall.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/1*mBNRcmy48FdxEnhJ7bFzwQ.png" /><figcaption>Partial Derivative for optimizing loss visual</figcaption></figure><p>This algorithm is called <strong>Gradient Descent</strong>, also called Stochastic Gradient Descent (SGD). And this is the most basic method of optimizing neural networks. This happens as an iterative process and hence we will update the value of each weight <strong>multiple </strong>times before the Loss converges at a suitable value. Let’s look at the mathematical way of representing a single update:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*1P6nV2QDB09eZ0q_tGtm7w.png" /></figure><p>Here the alpha symbol is the <strong>learning rate. </strong>This will affect the <strong>speed </strong>of optimization of our neural network. If we have a large learning rate, we will reach the minima for Loss faster because we are taking big steps, however as a result we may not reach a very good minimum since we are taking big steps and hence we might <strong>overshoot </strong>it. A smaller learning rate will solve this issue, but it will take a lot of steps for the neural network’s loss to decrease to a good value. Hence we need to keep a learning rate at an optimal value. Usually keeping alpha = 0.01 is a safe value.</p><p>There is, however, one big problem with just gradient descent. We are not pushing the loss towards the <strong>global minima</strong>, we are merely pushing it towards the closest <strong>local minima. </strong>Let’s see and illustration of this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/575/1*E0CZejU6bInnCr9iYBOcVA.png" /><figcaption>SGD Local Minima Problem</figcaption></figure><p>Here, we are starting from the labelled green dot. Every subsequent green dot represents the loss and new weight value after a <strong>single update</strong> has occurred. The gradient descent will only happen till the local minima since the <strong>partial derivative (gradient) </strong>near the local minima is <strong>near zero. </strong>Hence it will stay near there after reaching the local minima and will not try to reach the global minima.</p><p>This is a rather simple graph and in reality the graph will be much more complicated than this with <strong>many </strong>local minimas present, hence if we use just gradient descent we are not guaranteed to reach a good loss. We can combat this problem by using the concept of <strong>momentum</strong>.</p><h4>Momentum</h4><p>In momentum, what we are going to do is essentially try to capture some information regarding the <strong>previous updates</strong> a weight has gone through before performing the current update. Essentially, if a weight is constantly moving in a particular direction (increasing or decreasing), it will slowly accumulate some <em>“momentum” </em>in that direction. Hence when it faces some resistance and actually has to go the opposite way, it will still continue going in the original direction for a while because of the accumulated momentum.</p><p>This is comparable to the actual momentum in physics. Let’s consider a small valley and a ball. If we release the ball at one side of the valley, it will continuously roll down and in that process it is gaining momentum towards that direction. Finally when the ball reaches the bottom, it doesn’t stop there. Instead it starts rolling up on the opposite side for a while since it has gained some momentum even though gravity is telling it to come down.</p><p>We are essentially trying to recreate this scenario mathematically so that the gradient descent algorithm can try to bypass local minima to try and reach the global minima. Let’s look at the formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a_oS-GXKJ9toY636qg6_YQ.png" /><figcaption>Momentum Update Formula</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*m53YeFIk4PjCtsHGO3b0OQ.png" /><figcaption>Weight Update Formula (Momentum)</figcaption></figure><p>Here, V is the <strong>momentum factor. </strong>As you can see, in each update it essentially adds the current derivative with a part of the <strong>previous</strong> momentum factor. And then we just add this to the weight to get the updated weight. n here is the <strong>coefficient of momentum</strong> and it decides how much momentum gets <strong>carried forward</strong> each time.</p><p>As the weight gets updated, it will essentially store a <em>portion</em> of all the <strong>previous gradients</strong> in the momentum factor. Hence once it faces a change in the opposite direction, like a local minima, it will continue moving in the same direction until the magnitude of the momentum factor <strong>gradually decreases</strong> and starts pointing in the <strong>opposite direction</strong>. In most cases, the momentum factor usually is enough to make the weights overcome the local minima.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*X1vlfB5ZC8d6kIa8Xe0oiQ.png" /><figcaption>Momentum visual example</figcaption></figure><p>Another added advantage of momentum is that because of the accumulation of gradients, the weight will converge at an appropriate loss much faster. Let’s now look at another technique in optimization which allows gradient descent to happen in a smarter way.</p><h4>Nesterov Accelerated Gradients (NAG)</h4><p>In NAG, what we try to do is instead of calculating the gradients at the <strong>current position</strong> we try to calculate it from an approximate <strong>future position</strong>. This is because we want to try to calculate our gradients in a smarter way. Just before reaching a minima, the momentum will start reducing before it reaches it because we are using gradients from a future point. This results in improved stability and lesser oscillations while converging, furthermore, in practice it performs better than pure momentum.</p><p>Let’s look at how exactly we use NAG in optimizing a weight in a neural network.</p><p>If we look at the whole momentum equation expanded:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nEAhYFj-m4Uzfvv2x8HEVA.png" /><figcaption>Momentum full formula expanded</figcaption></figure><p>Here, we see that the next W value is essentially adding <strong>n*v_old</strong> with <strong>alpha*current gradient</strong> and the <strong>current weight</strong>. Considering the fact that alpha*current gradient is going to be a <strong>very small</strong> value, we can approximate the next W value by just adding n*v_old with the current W value. This will give an approximate <strong>future W </strong>value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*7vLbBq1pCbKafxjGWHJaag.png" /><figcaption>Calculating approximate future weight value</figcaption></figure><p>Now what we want to do is instead of calculating gradients with respect to the current W value, we will calculate them with respect to the future W value. This allows the momentum factor to start adapting to sharp gradient changes before they even occur, leading to increased stability while training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nru77xL87aLJWDlBRwll4A.png" /><figcaption>Nesterov Momentum Formula</figcaption></figure><p>This is the new momentum equation with NAG. As you can see, we are taking gradients from an approximate future value of W instead of the current value of W.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*5kZ3a5wqhX3uMpN6nvaCqw.png" /><figcaption>Nesterov VIsual Example</figcaption></figure><p>Here is an example of using NAG. At the current W value, we add n*v_old to approximate the future W value. Then we calculate the gradient at that point and use that instead of current gradient value while calculating the v_new value. As you can see in this example, even though the momentum is technically <strong>supposed to increase</strong> in this scenario, it will start <strong>decreasing at this point</strong> itself because the <strong>future W</strong> value has a gradient pointing in the <strong>opposite direction</strong>. Now let’s look at some adaptive optimization methods.</p><h3>Adaptive Optimization</h3><p>In these methods, the parameters like learning rate (alpha) and momentum coefficient (n) will not stay constant throughout the training process. Instead, these values will constantly adapt for each and every weight in the network and hence will also change along with the weights. These kind of optimization algorithms fall under the category of <em>adaptive optimization</em>.</p><p>The first algorithm we are going to look into is <strong>Adagrad.</strong></p><h4>Adagrad</h4><p>Adagrad is short for <em>adaptive gradients. </em>In this we try to change the learning rate (alpha) for each update. The learning rate changes during each update in such a way that it will decrease if a weight is being updated too much in a short amount of time and it will increase if a weight is not being updated much.</p><p>First, each weight has its own <strong>cache </strong>value, which collects the squares of the gradients till the current point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jp795S14USet83p1dn5MRQ.png" /><figcaption>Cache updation for Adagrad</figcaption></figure><p>The cache will continue to increase in value as the training progresses. Now the new update formula is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W2GEvRREEtXFFRqIZScKMg.png" /><figcaption>Adagrad Update Formula</figcaption></figure><p>This is of the same form as the original gradient descent formula except for the fact that the learning rate (alpha) constantly changes throughout the training. The E in the denominator is a very small value to ensure division by zero does not occur.</p><p>Essentially what’s happening here is that if a weight has been having <strong>very huge updates</strong>, it’s cache value is also going to <strong>increase</strong>. As a result, the learning rate will <strong>become lesser</strong> and that weight’s update magnitudes will decrease over time. On the other hand, if a weight has not been having any significant update, it’s cache value is going to be <strong>very less</strong>, and hence its learning rate will <strong>increase</strong>, forcing it to take <strong>bigger updates</strong>. This is the basic principle of the Adagrad optimizer.</p><p>However the disadvantage of this algorithm is that regardless of a weight’s past gradients, the cache will <strong>always increase</strong> by some amount because square cannot be negative. Hence the learning rate of every weight will eventually decrease to a <strong>very low value</strong> till training does not happen significantly anymore.</p><p>The next adaptive optimizer, <strong>RMSProp </strong>effectively solves this problem.</p><h4>RMSProp</h4><p>In RMSProp the only difference lies in the cache updating strategy. In the new formula, we introduce a new parameter, the decay rate (gamma).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pGeOYC4r4Oe5H1u3FkL_Xg.png" /><figcaption>RMSProp Cache Update Formula</figcaption></figure><p>Here the gamma value is usually around 0.9 or 0.99. Hence for each update, the square of gradients get added at a very slow rate compared to adagrad. This ensures that the learning rate is changing constantly based on the way the weight is being updated, just like adagrad, but at the same time the learning rate does not decay too quickly, hence allowing training to continue for much longer.</p><p>Next we will look into Adam optimizer, which is widely regarded as one of the best go to optimizers for deep learning in general.</p><h4>Adam</h4><p>Adam is a little like combining RMSProp with Momentum. First we calculate our m value, which will represent the momentum at the current point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VGq0Wfne44qBsy2Adq9qow.png" /><figcaption>Adam Momentum Update Formula</figcaption></figure><p>The only difference between this equation and the momentum equation is that instead of the learning rate we keep (1-Beta_1) to be multiplied with the current gradient.</p><p>Next we will calculate the accumulated cache, which is exactly the same as it is in RMSProp:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AKw3_35YAIb3KkHwIcHIMA.png" /></figure><p>Now we can get the final update formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pSfwLd33GnntqSmgpVy-Uw.png" /><figcaption>Adam weight updation formula</figcaption></figure><p>As we can observe here, we are performing accumulating the gradients by calculating momentum and also we are constantly changing the learning rate by using the cache. Due to these two features, Adam usually performs better than any other optimizer out there and is usually preferred while training neural networks. In the paper for Adam, the recommended parameters are 0.9 for beta_1, 0.99 for beta_2 and 1e-08 for epsilon.</p><p>That concludes this article in which I have covered most of the important optimizers which most deep learning practitioners use on a regular basis. I hope you have learnt something new! Thanks for reading!</p><p>Read more Deep Learning articles at <a href="https://deeplearningdemystified.com/">https://deeplearningdemystified.com</a></p><pre>Note: I am using notation d(Loss)/d(Wi) which actually represents dLoss/dW at the value of Wi.</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=313b787a69fe" width="1" height="1" alt=""><hr><p><a href="https://medium.com/deep-learning-demystified/https-medium-com-deep-learning-demystified-understanding-optimizers-313b787a69fe">Understanding Optimizers</a> was originally published in <a href="https://medium.com/deep-learning-demystified">Deep Learning Demystified</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Loss Functions Explained]]></title>
            <link>https://medium.com/deep-learning-demystified/loss-functions-explained-3098e8ff2b27?source=rss----ce61a9254205---4</link>
            <guid isPermaLink="false">https://medium.com/p/3098e8ff2b27</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Harsha Bommana]]></dc:creator>
            <pubDate>Mon, 30 Sep 2019 06:55:29 GMT</pubDate>
            <atom:updated>2020-04-25T00:49:59.771Z</atom:updated>
            <content:encoded><![CDATA[<p>In any deep learning project, configuring the loss function is one of the most important steps to ensure the model will work in the intended manner. The loss function can give a lot of practical flexibility to your neural networks and it will define how exactly the output of the network is connected with the rest of the network.</p><p>There are several tasks neural networks can perform, from predicting <strong>continuous values</strong> like monthly expenditure to <strong>classifying discrete</strong> classes like cats and dogs. Each different task would require a different type of loss since the output format will be different. For very specialized tasks, it’s up to us how we want to define the loss.</p><p>From a very simplified perspective, the loss function (J) can be defined as a function which takes in two parameters:</p><ol><li>Predicted Output</li><li>True Output</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*N1PyOYeog-vyytRbwEwQCQ.png" /><figcaption>Neural Network Loss Visualization</figcaption></figure><p>This function will essentially calculate how <strong>poorly</strong> our model is performing by comparing what the model is predicting with the actual value it is supposed to output. If Y_pred is very far off from Y, the Loss value will be <strong>very high</strong>. However if both values are almost similar, the Loss value will be <strong>very low. </strong>Hence we need to keep a loss function which can <strong>penalize </strong>a model effectively while it is training on a dataset.</p><p>If the loss is very high, this huge value will propagate through the network while it’s training and the weights will be changed a little more than usual. If it’s small then the weights won’t change that much since the network is already doing a good job.</p><p>This scenario is somewhat analogous to studying for exams. If one does poorly in an exam, we can say the loss is very high, and that person will have to change a lot of things within themselves in order to get a better grade next time. However if the exam went well, then they wouldn’t do anything very different from what they are already doing for the next exam.</p><p>Now let’s look at <strong>classification</strong> as a task and understand how the loss functions work in this case.</p><h3>Classification Losses</h3><p>When a neural network is trying to predict a <strong>discrete </strong>value, we can consider it to be a classification model. This could be a network trying to predict what kind of animal is present in an image, or whether an email is spam or not. First let’s look at how the output is represented for a classification neural network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/345/1*rFl5a2klSb0879Elf__Iqg.png" /><figcaption>Classification Neural Network Output Format</figcaption></figure><p>The number of nodes of the output layer will depend on the number of classes present in the data. Each node will represent a single class. The value of each output node essentially represents the <strong>probability </strong>of that class being the correct class.</p><pre>Pr(Class 1) = Probability of Class 1 being the correct class</pre><p>Once we get the probabilities of all the different classes, we will consider the class having the <strong>highest </strong>probability to be the predicted class for that instance. First let’s explore how <strong>binary classification </strong>is done.</p><h4>Binary Classification</h4><p>In binary classification, there will be only one node in the output layer even though we will be predicting between two classes. In order to get the output in a probability format, we need to apply an activation function. Since probability requires a value in between 0 and 1 we will use the <strong>sigmoid function </strong>which can <em>squish </em>any real value to a value between 0 and 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/1*5mGgjZYOd6Xtu1cNPE8k6Q.png" /><figcaption>Sigmoid Function Graph Visualization</figcaption></figure><p>As the input to the sigmoid becomes larger and tends to plus infinity, the output of the sigmoid will tend to 1. And as the input becomes smaller and tends to negative infinity, the output will tend to 0. Now we are guaranteed to always get a value between 0 and 1, which is exactly how we need it to be since we require probabilities.</p><p>If the output is above 0.5 (50% Probability), we will consider it to be falling under the <strong>positive class</strong> and if it is below 0.5 we will consider it to be falling under the <strong>negative class. </strong>For example if we are training a network to classify between cats and dogs, we can assign dogs the positive class and the output value in the dataset for dogs will be <strong>1</strong>, similarly cats will be assigned the negative class and the output value for cats will be <strong>0.</strong></p><p>The loss function we use for binary classification is called <strong>binary cross entropy (BCE)</strong>. This function effectively penalizes the neural network for binary classification task. Let’s look at how this function looks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/771/1*7nqhvFpW5-D55z4ejalnjQ.png" /><figcaption>Binary Cross Entropy Loss Graphs</figcaption></figure><p>As you can see, there are two separate functions, one for each value of Y. When we need to predict the <strong>positive class </strong>(Y = 1), we will use</p><pre>Loss = -log(Y_pred)</pre><p>And when we need to predict the <strong>negative class</strong> (Y = 0), we will use</p><pre>Loss = -log(1-Y_pred)</pre><p>As you can see in the graphs. For the first function, when Y_pred is equal to 1, the <strong>Loss is equal to 0</strong>, which makes sense because Y_pred is <strong>exactly the same</strong> as Y. As Y_pred value becomes closer to 0, we can observe the Loss value increasing at a <strong>very high rate</strong> and when Y_pred becomes 0 it tends to infinity. This is because, from a classification perspective, 0 and 1 have to be polar opposites due to the fact that they each represent completely different classes. So when Y_pred is 0 when Y is 1, the loss will have to be very high in order for the network to learn it’s mistakes more effectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/516/1*-bEoqRQKmBye7iqv7PUJBA.png" /><figcaption>Binary Classification Loss Comparisons</figcaption></figure><p>We can mathematically represent the entire loss function into one equation as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EJPT0utTkQ2qrHfjDID5RA.png" /><figcaption>Binary Cross Entropy Full Equation</figcaption></figure><p>This loss function is also called as <strong>Log Loss. </strong>This is how the loss function is designed for a binary classification neural network. Now let’s move on to see how the loss is defined for a multiclass classification network.</p><h4>Multiclass Classification</h4><p>Multiclass classification is appropriate when we need our model to predict <strong>one </strong>possible class output every time. Now since we are still dealing with probabilities it might make sense to just apply sigmoid to all the output nodes so that we get values between 0–1 for all the outputs, but there is an issue with this. When we are considering probabilities for multiple classes, we need to ensure that the <strong>sum</strong> of all the individual probabilities is <strong>equal to one</strong>, since that is how probability is defined. Applying sigmoid does not ensure that the sum is always equal to one, hence we need to use another activation function.</p><p>The activation function we use in this case is <strong>softmax. </strong>This function ensures that all the output nodes have values between 0–1 and the <strong>sum of all</strong> output node values <strong>equals to 1 always</strong>. The formula for softmax is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*T6ZFOuQgnaXg914MjQpA1g.png" /><figcaption>Softmax Formula</figcaption></figure><p>Let’s visualize this with an example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/417/1*17kG7Wk8sX4K514lhk7MTQ.png" /><figcaption>Softmax Example Visualization</figcaption></figure><p>So as you can see, we are simply passing all the values into a exponential function. After that, to make sure they are all in the range of 0–1 and to make sure the sum of all the output values equals to 1, we are just dividing each exponential with the sum of all exponentials.</p><p>So why do we have to pass each value through an exponential before normalizing them? Why can’t we just normalize the values themselves? This is because the goal of softmax is to make sure <strong>one value is very high</strong> (close to 1) and all <strong>other values are very low</strong> (close to 0). Softmax uses exponential to make sure this happens. And then we are normalizing because we need probabilities.</p><p>Now that our outputs are in a proper format, let’s go ahead to look at how we configure the loss function for this. The good thing is that the loss function is essentially the same as that of binary classification. We will just apply <strong>log loss</strong> on each output node with respect to its respective target value and then we will find the sum of this across all output nodes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/1*g5VvhALGnJJYhSkoYccDxA.png" /><figcaption>Categorical Cross Entropy Visualization</figcaption></figure><p>This loss is called as <strong>Categorical Cross Entropy. </strong>Now let’s move onto a special case of classification called multilabel classification.</p><h4>Multilabel Classification</h4><p>Multilabel classification is done when your model needs to predict <strong>multiple classes </strong>as the output. For example, let’s say you are training a neural network to predict the ingredients present in a picture of some food. There will be multiple ingredients we need to predict so there will be <strong>multiple 1’s </strong>in Y.</p><p>For this we can’t use softmax because softmax will always force only one class to become 1 and other classes to become 0. So instead we can simply keep sigmoid on all the output node values since we are trying to predict each class’s <strong>individual probability</strong>.</p><p>As for the loss we can directly use log loss on each node and sum it, similar to what we did in multiclass classification.</p><p>Now that we have covered classification, let’s now move on to <strong>regression</strong>.</p><h3>Regression Loss</h3><p>In regression, our model is trying to predict a <strong>continuous value. </strong>Some examples of regression models are:</p><ul><li>House price prediction</li><li>Person Age prediction</li></ul><p>In regression models, our neural network will have <strong>one output node</strong> for every continuous value we are trying to predict. Regression losses are calculated by performing direct comparisons between the output value and the true value.</p><p>The most popular loss function we use for regression models is the <strong>mean squared error </strong>loss function. In this we simply calculate the square of the difference between Y and Y_pred and average this over all the data. Suppose there are n data points:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Da7hfvG2APKDXW4mAwgtjQ.png" /><figcaption>Mean Squared Error Loss Function</figcaption></figure><p>Here Y_i and Y_pred_i refer to the i’th Y value in the dataset and the corresponding Y_pred from the neural network for the same data.</p><p>That concludes this article. Hopefully now you have a deeper understanding of how loss functions are configured for various tasks in deep learning. Thank you for reading!</p><p>Read more Deep Learning articles at <a href="https://deeplearningdemystified.com/">https://deeplearningdemystified.com</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3098e8ff2b27" width="1" height="1" alt=""><hr><p><a href="https://medium.com/deep-learning-demystified/loss-functions-explained-3098e8ff2b27">Loss Functions Explained</a> was originally published in <a href="https://medium.com/deep-learning-demystified">Deep Learning Demystified</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Neural Networks — Part 2]]></title>
            <link>https://medium.com/deep-learning-demystified/introduction-to-neural-networks-part-2-c261a99f4138?source=rss----ce61a9254205---4</link>
            <guid isPermaLink="false">https://medium.com/p/c261a99f4138</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[explanation]]></category>
            <category><![CDATA[learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Harsha Bommana]]></dc:creator>
            <pubDate>Sun, 26 May 2019 13:36:44 GMT</pubDate>
            <atom:updated>2020-04-25T00:50:19.545Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction to Neural Networks — Part 2</h3><h3>Backpropagation</h3><p>Now that we have seen how a neural network is represented, we can go on to see how exactly it works. Since there are many layers having many neurons, there exists a complex set of weights to get an output from some input variables. Each weight in this network can be changed and hence there are countless configurations a neural network can have. A <strong>trained </strong>neural network has some weights configuration which accurately predicts correct outputs from some input data and that is what we hope to achieve. We will now go through how exactly a neural network trains itself to get this desirable weight configuration.</p><p>Backpropagation is the name of the algorithm a neural network uses to train itself. This revolutionary algorithm is a mixture of the chain rule in derivation and gradient descent, which is a common optimization algorithm which is used in linear and logistic regression.</p><p>To understand how backpropagation works, first we have to understand the relationship between the output and the weights in between. It is clear that every weight in the neural network will affect the output in some way due to the way the neural network is connected. Due to this fact, we can say that if I <strong>change</strong> a particular weight, the output will <strong>change</strong> in some way. We can also find the exact mathematical equation defining the relationship between each weight and the output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/1*sSYs6mpCKyQPh7BC-ahEbw.png" /></figure><p>Let’s say we are given a dataset with three input features, (X1, X2 and X3) and we need to find the relationship between the input features and the output using a neural network. We will now see what exactly the neural network does.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*PjNfqsc68flOK4dFVCY_CQ.png" /></figure><p>First step is to feed in the data and getting the output from the neural network. We shall call the output, Y_pred. Next we should compare the predicted value with the actual value. This value will be called the <strong>error. </strong>It is essentially how bad or how far off the model is from predicting the values correctly. Our goal is to now <strong>minimize the error.</strong></p><p>Since Y_pred is a function of all the weights in the model and Error is a function of Y_pred, we can say that the Error will also depend on the weights. This means that we need to adjust our weights in such a way that the error is minimized. We do that using <strong>partial derivatives.</strong></p><p>Let’s take a simple example of an equation, Y = W1*X1 + W2*X2. If we find the partial derivative of W1 or W2 with respect to Y, we can find out <strong>how </strong>W1 or W2 can affect Y.</p><p>If the partial derivative of W1 with respect to Y is <strong>POSITIVE, </strong>that means <strong>DECREASING </strong>the weight will <strong>DECREASE</strong> Y.</p><p>If the partial derivative of W1 with respect to Y is <strong>NEGATIVE, </strong>that means <strong>INCREASING</strong> the weight will <strong>DECREASE</strong> Y.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/1*iVKOY8bTDHv2Q-H6__MLGQ.png" /></figure><p>These two rules are all we need to optimize the weights in the neural network. We need to find the partial derivative of <strong>every </strong>weight with respect to the <strong>error. </strong>If that partial derivative is <strong>positive, </strong>then we <strong>decrease </strong>the value of that weight so that error gets decreased. If that partial derivative is <strong>negative, </strong>then we <strong>increase </strong>that weight so that the error gets decreased. This is the basic underlying concept of how weights are updated after we calculate the error.</p><p>Since the last layer is the closest to the error, we will first <strong>derive </strong>the last layer with respect to the error and update those weights. Then we will move to the second last layer to do the same and so on and so forth. We repeat this process till we reach the first layer and all the weights are updated. This entire process is called <strong>backpropagation.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*mTTmfdMcFlPtyu8__vRHOQ.gif" /></figure><p>We perform backpropagation for a single row of data and update the weights. We then repeat for all the data available in the training data set, this entire cycle is called <strong>one epoch. </strong>Usually neural networks can take several epochs to train and it is up to us to decide how many epochs it will train for.</p><h3>Error Calculation Methods</h3><p>Error is a very important part of a neural network because it allows us to estimate how poorly the model is performing and accordingly we can update our weights to improve performance. Now let’s go through how exactly error can be calculated in neural networks. The type of error calculation method we choose will depend upon the type of task we are trying to do. There are mainly two types of tasks a typical neural network can do:</p><ol><li>Regression</li><li>Classification</li></ol><p>Regression is when our output variable is <strong>continuous </strong>in nature. When Y is a numerical variable which we have to predict, the task is called regression. Examples include trying to predict house prices or trying to predict how many marks someone will score in an exam.</p><p>Classification is when our output variable is <strong>discrete </strong>in nature. When Y is trying to represent a certain class out of a defined number of classes, the task is called classification. Examples include trying to predict between a cat and a dog, or trying to predict whether an email is spam or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*aCQOAmB9Aro4nq0y94e7Wg.png" /></figure><p>Here, we have to understand that in the regression graph the output variable is a continuous value just like X. In the classification graph, there are 2 input variables (X1 and X2) and the output is represented by the <strong>color </strong>of the points. Eg: Y = 0 for RED and Y = 1 for GREEN.</p><p>Let’s first cover regression.</p><p>For regression, the error function we use is <strong>Mean Squared Error.</strong> We are simply trying to find the difference between the actual value and predicted value, and square it. The formula for the same is given below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*zIMyjb1zFWUm1PJKPV9Miw.png" /></figure><p>Here, E is the error of the model. Y pred is the output of the model for the data and Y is the actual output which we are supposed to get. We square this so that negative and positive differences both become positive in the end. N represents the number of data rows we input into the model. We are essentially calculating error of all points and averaging them. As we train the model we hope to achieve a very <strong>low MSE</strong> so that we know that our neural network is predicting values which are close to the actual values.</p><p>Moving on to classification.</p><p>In classification the way we represent data is completely different. Because we are dealing with multiple classes which may not be in a numerical format, we have to convert them into a numerical format so that the neural network can process them. We typically convert them using the method called <strong>one hot encoding.</strong></p><p>Let’s take the example of having three cities as classes and we have to predict between them. San Francisco, New York and Boston. We will then have an array of size <strong>3 </strong>representing Y for each row of data. It is done in the following way:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/499/1*8B0MO2Klygj5M2fEipXWbw.png" /></figure><p>The array size will indicate the total number of classes and the position of <strong>1 </strong>in the array will represent the class.</p><p>Before we calculate the error in this scenario. We also have to ensure that the last layer of the neural network has a number of neurons equal to the number of classes. The last layer of the neural network should also be having the activation function of <strong>softmax.</strong> This activation function essentially converts the last layer into a <strong>probability distribution. </strong>Hence the sum of all the values of the nodes in the last layer will be <strong>equal to 1</strong>.</p><p><em>(Note: If there are only two classes we can use just one node and use Sigmoid instead of Softmax)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/417/1*17kG7Wk8sX4K514lhk7MTQ.png" /></figure><p>Now that it’s a probability distribution and all the values are between 1 and 0, we can proceed to calculate the error. The error we use here is called <strong>Log Loss, </strong>or <strong>Categorical Cross Entropy. </strong>What this function basically does is to compare each element of Y pred and Y and see how far apart they are. Then we average this value across all elements.</p><p>From the classification perspective, when we predict 0 at a particular element but the actual value is 1, or when we predict 1 and the actual value is 0, that means our model is performing extremely bad and we have to have a huge error in this case. This is the main concept of this error function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/537/1*1KFiJacjut4nFetMcNLT1g.png" /></figure><p>Having this kind of a error function forces the neural network to understand where it is going wrong much quicker and hence allows it to learn from the data much faster. I would advise you to go through the mathematical formulas of Softmax and Log Loss for better understanding of what’s going on.</p><p>This concludes this two part tutorial on Neural Networks. I will be going much deeper into some of the concepts we have covered today in future articles and I will be going through several technical tutorials as well. Thank you for reading!</p><p>Read more Deep Learning articles at <a href="https://deeplearningdemystified.com/">https://deeplearningdemystified.com</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c261a99f4138" width="1" height="1" alt=""><hr><p><a href="https://medium.com/deep-learning-demystified/introduction-to-neural-networks-part-2-c261a99f4138">Introduction to Neural Networks — Part 2</a> was originally published in <a href="https://medium.com/deep-learning-demystified">Deep Learning Demystified</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>