<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Inside AI on Medium]]></title>
        <description><![CDATA[Stories by Inside AI on Medium]]></description>
        <link>https://medium.com/@inside_ai?source=rss-5fccc8e98750------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*hoxTeiK4VDGMiaKmocObbg.png</url>
            <title>Stories by Inside AI on Medium</title>
            <link>https://medium.com/@inside_ai?source=rss-5fccc8e98750------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 09:41:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@inside_ai/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[14. Introduction to Deep Learning with Computer Vision — Learning Rates & Mathematics — Part 2…]]></title>
            <link>https://medium.com/hitchhikers-guide-to-deep-learning/14-introduction-to-deep-learning-with-computer-vision-learning-rates-mathematics-part-2-c5290b492f0b?source=rss-5fccc8e98750------2</link>
            <guid isPermaLink="false">https://medium.com/p/c5290b492f0b</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Inside AI]]></dc:creator>
            <pubDate>Sun, 20 Sep 2020 18:18:45 GMT</pubDate>
            <atom:updated>2022-06-22T05:14:43.827Z</atom:updated>
            <content:encoded><![CDATA[<h3>14. Introduction to Deep Learning with Computer Vision — Learning Rates &amp; Mathematics — Part 2 (SuperConvergence)</h3><p>Written by: <a href="https://medium.com/u/f1a8d777cc44">Praveen Kumar</a> &amp; <a href="https://medium.com/u/4ce50580cfab">Nilesh Singh</a>!</p><p><a href="https://medium.com/hitchhikers-guide-to-deep-learning/13-introduction-to-deep-learning-with-computer-vision-learning-rates-mathematics-part-1-4973aacea801">Learning Rates &amp; Mathematics: Part 1</a></p><p>In the previous blog, we gained an understanding of “<a href="https://medium.com/hitchhikers-guide-to-deep-learning/13-introduction-to-deep-learning-with-computer-vision-learning-rates-mathematics-part-1-4973aacea801">Learnings rates, Local and Global Minima, &amp; Saddle points.</a>” In this blog, we will look into how learning rates can vary to avoid certain pitfalls that a deep learning model can fall under and decrease the overall efficiency of learning.</p><h3>Finding Optimal starting Learning Rate</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7hbhT9zi6YuqJ5Vz.png" /><figcaption>Learning rates [<a href="https://www.google.com/url?sa=i&amp;url=https%3A%2F%2Fwww.jeremyjordan.me%2Fnn-learning-rate%2F&amp;psig=AOvVaw3hFXbgJdFZqHxGsM5okjAM&amp;ust=1599569083742000&amp;source=images&amp;cd=vfe&amp;ved=0CA0QjhxqFwoTCLiLp6yJ1-sCFQAAAAAdAAAAABAq">Source</a>]</figcaption></figure><p>We discussed in the last blog that choosing a learning rate is a tricky part of tuning your model’s hyper-parameter. In the above image, we see choosing a learning rate changes the model’s approach to learn features.</p><p>When we design a model, we have the option to choose any of the following types of learning rate:</p><ol><li><strong>Constant learning rate</strong>: Constant learning rate refers to setting the learning rate an initial value and uses the same value over the training phase without changing it over time.</li><li><strong>Dynamically changing the learning rate</strong>: This type of learning rate varies with time and number of epochs and can adapt to high or low values based on the learning process. This may just sound right but certainly tricky to implement in reality. We shall see it in greater depth as we move ahead in this article.</li></ol><p>About the image shown above, look at the middle plot and you can see that initially, the steps (shown in red arrows) are larger and slowly decreases as we move towards global minima.</p><h3>A systematic approach to choose a learning rate?</h3><p>There is no universal optimal learning rate and method (we know its sad to hear 😒). Ideally, we want to start with a learning rate that yields significant decreases in the loss function. A systematic approach in finding such a learning rate is by “<em>observing the magnitudes of loss change with different learning rates”</em>.</p><p>First, we need to gradually increase the learning rate either linearly (suggested by Leslie Smith) or exponentially (suggested by Jeremy Howard) as shown below,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/0*ORVHaxx3YP2w7ZR9.PNG" /><figcaption>Learning rate over iterations [<a href="https://www.google.com/url?sa=i&amp;url=http%3A%2F%2Fmiguel-data-sc.github.io%2F2017-11-05-first%2F&amp;psig=AOvVaw01F8mTa44evhOFCLgddGKv&amp;ust=1599570002003000&amp;source=images&amp;cd=vfe&amp;ved=0CA0QjhxqFwoTCOCfy-KM1-sCFQAAAAAdAAAAABAD">Source</a>]</figcaption></figure><p>and after each mini-batch, record the loss at each increment as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/947/0*HrVglF7eRACaJQj7.png" /><figcaption>Learning rate Vs Loss [<a href="https://www.google.com/url?sa=i&amp;url=https%3A%2F%2Fwww.jeremyjordan.me%2Fnn-learning-rate%2F&amp;psig=AOvVaw1dpqB7AZeDk3XeljKCKbJG&amp;ust=1599570445276000&amp;source=images&amp;cd=vfe&amp;ved=0CA0QjhxqFwoTCKiawLSO1-sCFQAAAAAdAAAAABAD">Source</a>]</figcaption></figure><p>The learning rate should be set within the range where the occurrence of loss decreases drastically. In the above image, this lies somewhere close to 10e-1 and 10e-2.</p><p>This technique was proposed by Leslie Smith in <a href="https://arxiv.org/abs/1506.01186">Cyclical Learning Rates for Training Neural Networks</a> and evangelized by Jeremy Howard in <a href="http://course.fast.ai/index.html">fast.ai’s course</a>.</p><p>If you wish to implement for your model: <a href="https://github.com/surmenok/keras_lr_finder">Reference</a></p><h3>Learning Rate Annealing, another systematic approach</h3><p>Selecting a good starting learning rate is merely the first step. To efficiently train a robust model, we will need to gradually decrease the learning rate during training. If the learning rate remains unchanged during training, it might be too large to converge and cause the loss function to fluctuate around the local minimum. The approach is to use a higher learning rate to quickly reach the regions of (local) minima during the initial training stage and set a smaller learning rate as training progresses to explore “deeper and more thoroughly” in the region to find the minimum.</p><p>The most popular form of learning rate annealing is a <em>step decay</em> where the learning rate is reduced by some percentage after a set number of training epochs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/509/0*QGMOKzaXIucokkzO.png" /></figure><p>There is an array of methods for learning rate annealing: step-wise annealing, exponential decay, cosine annealing(strongly suggest by Jeremy Howard), etc. More details on an annealing learning rate at <a href="http://cs231n.github.io/neural-networks-3/#anneal">Stanford’s CS231</a>.</p><p><em>More generally, we can establish that it is useful to define a </em><strong><em>learning rate schedule</em></strong><em> in which the learning rate is updating during training according to some specified rule.</em></p><h3>Cyclic Learning Rate (CLR)</h3><blockquote>1. Intuition behind CLR</blockquote><p>In previous approaches, we discussed that decreasing the learning rate over time to time would give us better learning performance. We always assumed that the gradient would be in a curved bowl shape. However, in real life, this is never(almost) the case. Learning rate annealing methods previously discussed will not work if the gradient curve has more minima and maxima.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*e7E8boUx7uNrxWLg" /></figure><p>In the above image, the left gradient is perfectly shaped and annealing methods will be a great choice. However, if you consider the gradient curve to the right, will annealing methods still be useful? Give it a thought! 😉</p><p>Consider the following 3 scenarios which show gradient curves in 2D:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/0*og9-ynvlgRnd8Gt_.png" /></figure><p>Assume you are standing at the red line in the above image. In all the 3 cases, you need a push (learning rate should be increased) that could help you get out of the local minima and continue to search for global minima. In case you are stuck at some point and the learning rate is slowly decreased (like annealing methods), you might as well never come out of any of those 3 situations. In that case, the model’s accuracy would jitter around a very small range and neither decrease nor increase. So you require learning rates that not only decrease with time (like when you are near to your global minima) but also increase in case you get stuck in the path (like the above image). Surely one can never know where we might stand at any instance of time, but one can wisely monitor the loss and accuracy curves and can try out different strategies. Ultimately, our goal is to avoid plateaus or valleys on the path to perform better and find global minima.</p><blockquote>2. Working of CLR</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*s2BQbf0rpMX8eChh" /></figure><p>The cyclic learning rate, proposed by L. Smith, makes use of varying learning rates over time. The above image shows the behavior of the triangular policy in CLR (simply because the patterns are triangular). We have other policies that show different behavior. These behavior patterns are mathematically constructed. You can even construct your function and behavior pattern for the learning rate if you a good mathematical background.</p><blockquote>One of the key points in understanding CLR functions is the difference between iterations and epochs. People often confuse between these two keywords. WE ARE HERE FOR YOU. DON’T WORRY. 👐</blockquote><blockquote>1 Epoch: One epoch refers to one complete pass of your complete training data through the model (Both forward &amp; backward). For example, If you have 1000 images in your training set, then 1 epoch refers to allow all these 1000 images to pass through your model, and backpropagate all the weights, no matter what batch size you choose.</blockquote><blockquote>1 Iteration: This refers to the batch of data to complete one forward and backward pass. This depends on the batch size. For example, if your batch size is 10, then to complete 1000 images, we need 100 iterations. Why? because we are passing 10 images through our model, so we need to pass batch of 10 images for 100 times to complete 1000 images, which would then complete one epoch.</blockquote><p>Let’s now understand how triangular CLR works. We have a few parameters to keep track of while using CLR.</p><ul><li><strong>Base learning rate: </strong>This is the min learning rate we are going to use.</li><li><strong>Max learning rate: </strong>This is the maximum learning rate you want to use for your model. This can be set to a large value range such as 0.5 or even sometimes 1.0 or 2.0</li><li><strong>Cycle:</strong> A cycle is a complete phase of learning rate where it takes values from minimum to maximum and then from maximum to minimum. We will understand more about the cycle in just a bit.</li><li><strong>Step size: </strong>Step size refers to the half cycle.</li></ul><p>Now to understand CLR, we have a set of formulas to follow (afraid not, we will make it ultra simple for you, promise 👌). We will explain those formulas with more intuitive examples. Let’s define a few values which we will use in the following steps.</p><p>let, Step Size = 1000</p><p>So, One cycle = 2 * Step Size = 2*1000 = 2000</p><p>The total number of iterations = 10010 (it couldn’t be 10000, because? it’s a mad mad mad world. Things are not ideal 😜. Just kidding though, we simply picked up a random value),</p><p>So, total number of cycles = ceil(10010/2000) = ceil(5.005) = 6 cycles.</p><p><em>[NOTE: Here we have used </em><strong><em>ceil </em></strong><em>because the total number of cycles can not have floor since it will eliminate the remaining iterations. You will find floor function in upcoming equations, do not confuse with them.</em><strong><em> The floor is used to get n-th cycle number &amp; not the total number of cycles</em></strong><em>]</em></p><p>This means that throughout our model training, the CLR will have a total of 6 cycles or simply saying, 6 triangles in the graph we have seen above.</p><p>Let’s now understand how all the equations work in CLR.</p><p><strong>Let’s look at the first formula:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/407/1*aXbI5WDgfTlDc8gVwT59PQ.png" /></figure><p>This tells us which cycles we are currently in. So, in our example, its</p><p>Cycle = floor ( 1 + 10010/(2*1000) ) = floor (1 + 5.005) = 6th cycle.</p><p>this means we completed 5 cycles by 10000 iterations and for the remaining 10 iterations, we move into the 6th cycle. If we are on the 5000th iteration, then we would be in,</p><p>cycle = floor (1 + 5000/2000) = floor(3.5) = 3rd cycle. Similarly, based on the current number of iterations, we can find out which cycle we are in.</p><p><strong>Let’s look at our second equation.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/1*zXulRKsVKoLtU6h8oQYXsw.png" /></figure><p>The above equation helps in (does not calculate final learning rate but only helps in the process 😵) calculating the leaning rate. In the equation, we have 3 variables, namely, iterations, stepsize &amp; cycle (this is not the length of the cycle by the nth cycle in which we currently are, derived from the first equation). To understand this equation further, let’s look at the image below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/1*IjC2gZmYf07z1hv8h5oUdA.png" /></figure><p>We have zoomed into one cycle. Each cycle has an uphill and downhill line. At the Y-axis, we have a learning rate. So, at each value of X on the x-axis, we have a different range of learning rates which oscillates between the maximum and minimum learning rate. The second equation helps us in obtaining the values on the x-axis. Let’s understand how.</p><p>With the help of the cycle parameter, we know at which cycle we currently are in. After getting to know at which cycle, we need to find the ratio of iterations/stepsize. This ratio will help us identify on which size of the cycle we are in. Simply saying, the cycle is divided into 2 parts. The left part where the learning rate goes from minimum to maximum and the right part where the learning rate goes from maximum to minimum. The step size is half of the cycle and based on the current number of iterations, we will know which side of the current cycle we are in. For example, previously we found out that if we were on the 5000th iteration, it means we are on the 3rd cycle. So, now we need to find where exactly in this 3rd cycle we are in. That is what X will help us in calculating with the help of step size.</p><p>so, for the 5000th iteration, we get</p><p>X = 1- mod((5000/1000)- 2(3)+1) = 1 — mod(5–6+1) = 1</p><p>1 indicates that new cycle just started. If iteration were 5500, then X would be 0.5, which means we are the peak learning rate and if iterations were slightly less than 6000, then X would be less than 0.05, meaning we are near to the end of the current cycle, in that case, the learning rate should be low (why? just look at the above image and guess it! Write it down in comments what you think be the reason is). If we would take other iteration, we would get a value within the range 0 to 1.</p><p>Since we have X now, we have to simply map the value of X on to the y-axis, that is, to find the corresponding learning rate.</p><p><strong>Let’s look at the third formula, which will help us to do the same.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/433/1*VBNAdyyuqDRTBVxgDev0dg.png" /></figure><p>In the equation,</p><p>ηt = learning rate at time t</p><p>ηmin = minimum learning rate</p><p>ηmax = maximum learning rate</p><p>To find the learning rate at time t, we have 3 steps.</p><ol><li>Calculate the difference between η max &amp; η min.</li><li>Find max (0, x).</li><li>Add η min to the value calculated from the above 2 steps.</li></ol><p>Thus, at each iteration, we get the corresponding learning rate, and the model trains very fast. Smith writes the main assumption behind the rationale for a cyclical learning rate (as opposed to one which <em>only</em> decreases) is “that <strong>increasing the learning rate might have a short term negative effect and yet achieve a long term beneficial effect</strong>.” Additionally, <strong>increasing the learning rate can also allow for “more rapid traversal of saddle point plateaus.”</strong></p><p>Till now we have only seen the triangular policy of CLR. We have another 2 which are generally used, but we won’t discuss them at much greater depth right now. Let’s look at the curves formed by them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/1*PrAifB2dQa9CwOGOGC6v7w.png" /></figure><p>The first figure on the above image shows a fixed decay value set before the start of the model training. This value will decrease the maximum learning rate by a constant value with each iteration. And, the second figure shows an exponential decay. In exponential, the max learning rate does not decrease a lot during the initial training phase and exponentially decreased towards the end phase of training. Both these policies can be mathematically tweaked. <strong>S<em>o, yes, if you can find a better tweak version, then congratulate yourself 👏 and publish a new research paper &amp; most importantly, do let us know in the comments. 😝</em></strong></p><p>In the next article, we will learn about SGD and the warm-start step. We will then see how One Cycle LR (not CLR but One Cycle LR), which is another great learning policy for better, faster, and, more accurate learning. We also will discuss the maths behind each of these and also discuss the most popular algorithms such as SGD, Adam, and RMSProp with great details &amp; mathematics behind them. Stay tuned!</p><p>Hope you enjoyed it. See you soon in the next article.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c5290b492f0b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hitchhikers-guide-to-deep-learning/14-introduction-to-deep-learning-with-computer-vision-learning-rates-mathematics-part-2-c5290b492f0b">14. Introduction to Deep Learning with Computer Vision — Learning Rates &amp; Mathematics — Part 2…</a> was originally published in <a href="https://medium.com/hitchhikers-guide-to-deep-learning">Deep-Learning-For-Computer-Vision</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[13. Introduction to Deep Learning with Computer Vision — Learning Rates & Mathematics — Part 1]]></title>
            <link>https://medium.com/hitchhikers-guide-to-deep-learning/13-introduction-to-deep-learning-with-computer-vision-learning-rates-mathematics-part-1-4973aacea801?source=rss-5fccc8e98750------2</link>
            <guid isPermaLink="false">https://medium.com/p/4973aacea801</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[computer-science]]></category>
            <dc:creator><![CDATA[Inside AI]]></dc:creator>
            <pubDate>Sun, 12 Jul 2020 17:32:36 GMT</pubDate>
            <atom:updated>2022-06-22T05:16:36.533Z</atom:updated>
            <content:encoded><![CDATA[<h3>13. Introduction to Deep Learning with Computer Vision — Learning Rates &amp; Mathematics — Part 1</h3><p>Written by <a href="https://medium.com/u/4ce50580cfab">Nilesh Singh</a> and <a href="https://medium.com/u/f1a8d777cc44">Praveen Kumar</a></p><p>Having spent enough time on <a href="https://medium.com/hitchhikers-guide-to-deep-learning/8-introduction-to-deep-learning-with-computer-vision-activation-functions-22dbcf5f7cd1">Activation functions</a>, <a href="https://medium.com/hitchhikers-guide-to-deep-learning/9-introduction-to-deep-learning-with-compute-vision-normalization-batch-normalization-ba5f60c77cf3">Batch Normalization</a>, <a href="https://medium.com/hitchhikers-guide-to-deep-learning/10-introduction-to-deep-learning-with-computer-vision-types-of-convolutions-atrous-convolutions-3cf142f77bc0">Convolution types,</a><a href="https://medium.com/hitchhikers-guide-to-deep-learning/11-introduction-to-deep-learning-with-computer-vision-image-augmentations-f65530a67409"> Data Augmentation</a>, and <a href="https://medium.com/hitchhikers-guide-to-deep-learning/12-introduction-to-deep-learning-with-computer-vision-receptive-fields-93d227ce8fd5">Receptive fields</a>, we now understand how models use different strategies to learn the features of a class and also improve its learning by understanding more complex features. However, having said that the model learns these features and all works without a snitch, we will now try to gain a deeper understanding of how the model learns.</p><p>We will dedicate a few articles (starting with this one) and continue to learn about the learning rates, Mathematics behind learning rates, popular learning rate methods &amp; finally the backpropagation algorithm along with its mathematics as well.</p><p>Let’s get started…</p><h3>What is the Learning Rate?</h3><p>It is essentially a hyperparameter that tells our model, how fast should it try to capture features and learn about a class. Intuitively, let’s say we have a 1-year-old baby and our task is to teach the baby how to recognize an apple.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/390/0*gy3jkEi8VsYKMM9s.jpg" /></figure><p>Now imagine you are holding the apple in front of the baby for 1 second and hope it has seen and learned all the features of an apple, or you can hold for 1 minute and hope that this time the baby has learned an apple, or even a day or month. Based on the different duration of the apparent holding of our metaphorical apple, the baby will be able to learn different features of the apple. This rate of us showing the apple to the baby is simply the learning rate parameter.</p><p>Now you might feel that the longer we show the apple, the better the baby learns the features. However, it is not the case. The more time we show the apple to the baby, the less slowly it learns the features. Hence it will take more time for the model to learn all the features and we can not keep on waiting for it to learn for ages just to recognize an apple. What happens when we just show the apple for 1 second? In this case, the baby will feel hard done by. It is just 1-year-old and asking it to learn all the features of an apple within 1 second is very harsh. We have other issues as well based on learning rate values. We shall discuss them in detail as we move deeper.</p><p>Hence, choosing the right learning rate is very crucial for the model.</p><blockquote>NOTE: In terms of models, a learning rate of 0.0001 is considered to be very slow (analogous to 1 month in the above example) and 0.9 is considered to be fast (analogous to 1 seconds in the above example). Smaller the value, slower the learning phase, and vice-versa.</blockquote><p>Let’s now put on our technical hats and learn what the fuss about the learning rate on CNN is and what happens in the background.</p><h3>Learning rate &amp; Loss function relation:</h3><p>The loss function is a method of evaluating how well your algorithm models your dataset. That is to say that how well your equation (network) fits the data at hand. If the equation perfectly fits the data being trained on, then the loss will be zero. As a rule of thumb, we don’t want that, because in most cases that would mean a horrible horrible case of ‘overfitting’, or a first step in the creation of Ultron. So, in general, for practical cases, we expect our loss to be approaching zero, but we’ll be wary of it ever actually becoming zero.</p><p>If your model’s predictions are off, the loss function will output a higher number. If they’re pretty good, it’ll output a lower number. What we mean by ‘predictions being off’ is the difference between the predicted model output and the actual expected output. This loss function calculates these values based on a few mathematical formulas, for example, the cross-entropy loss, the mean-squared error, the Huber loss, and the hinge loss. If you want a deeper understanding of loss functions, do check out this excellent <a href="https://algorithmia.com/blog/introduction-to-loss-functions">blog</a>.</p><p>Now things start getting interesting at this point. Let’s take a moment and recall what has happened to this point.</p><p>We have our data at hand, then we wrote a network to fit that data, we enter the training phase, and after each subsequent forward pass, we get a model that is trying hard to fit the data. We then calculate the accuracy of this model by calculating a loss function, which is essentially a function of predicted output and the expected output.</p><p>Alright, now let’s move forward.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/459/0*C5kzgb3kIQ7cLSfv.jpeg" /><figcaption>Fig 1: Learning rate curves</figcaption></figure><p>From Fig 1, we understand, that during the training phase, the loss will increase or decrease based on the learning rate parameter.</p><blockquote><strong>Learning rate</strong> is essentially a value between 0 and 1, which signifies the weight that each calculated loss function carries when the network backpropogates. One can think of it as the magnitude of weight updates that happen during backpropogation. It is essentially a hyper-parameter that is used to minimize the loss function.</blockquote><p>If the learning rate is very high, then loss starts to rise exponentially. Each learning rate has its pros and cons. One could think a low learning rate will be best, however, it will take a lot of time for you to train such a network (Think smaller weight updates, so slower localization). Hence we look for a good learning rate value. There are several methods to help you find out the optimal learning rate. We shall discuss them as we move along.</p><p>Now let’s try to understand the loss function more deeply to have a fair idea of why a small variation in learning rate value could be disastrous or beneficial for the model.</p><p>When we talk about the learning rate, we always refer to ways by which our weights are updated. Eventually, it’s the weights that matter to the model. These weights are updated after each epoch. Here is a simple weight equation:</p><pre>updated_weight= old_weight - learning_rate * derivative_of_loss</pre><p>Mathematically, it can be represented as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/331/1*8wP7eLN5fn5eNxMmPXRuxw.png" /></figure><p>In this equation, <strong>W</strong> is weight, <strong>α</strong> is the learning rate and J is the loss function which is partially derived wrt. weight. The last term is called a gradient. The learning rate is multiplied by the derivative of the loss function. If the learning rate is high, the weights are decreased (or increased, depending on the loss) by a large value or if the learning rate is low, the weights are updated very slowly. Let’s visualize this equation along with our loss function curve.</p><figure><img alt="Image result for weight update loss function" src="https://cdn-images-1.medium.com/proxy/1*t6OiVIMKw3SBjNzj-lp_Fw.png" /><figcaption>Fig 2: The gradient space</figcaption></figure><p>In the above Fig, the blue curve is imagined to be the loss curve (y-axis). So, we would want to reach the bottom of this curve to have the minimum loss, this is called minima. Along the x-axis, we have weights.</p><p>Now let&#39;s visualize scenarios for both high and low learning rates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*-TuVhSQH7rCqXCIc" /><figcaption>Fig. 3: Gradient descent with small (top) and large (bottom) learning rates. Source: Andrew Ng’s Machine Learning course on Coursera</figcaption></figure><p>As you can see, in the first part of the figure, the learning rate is too small, so the convergence takes a lot of time aka steps. This is not the only problem with this approach as we will see further. In the second part of the figure, the learning rate is too large, so our weights are jumping around instead of approaching the global minima.</p><p>We can visualize these mountainous valleys, where we want to get to the bottom of the valley by navigating the tough landscape that lies ahead of us. In relation to the above analogy, let’s try and understand some of the common terminologies that you will be using throughout your journey of deep learning:</p><ol><li><strong>Local vs Global minima:</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/213/1*9DZ-Cg13JtP7xRfB5sUCTA.png" /><figcaption>Fig 5. Descending down the valley</figcaption></figure><p>Let’s assume that you started your journey at point <strong>A</strong>, then followed the path down to point<strong> B</strong>. Now when you look around you, it can be safely concluded that you have reached the absolute bottom of our valley, right? Hold that thought for a moment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*PbKEszC_0b5vbpvnV862ag.png" /><figcaption>Fig 6. The concept of global and local minimum</figcaption></figure><p>Now when we see the whole picture, point <strong>B </strong>is definitely not the point where we want to be, in fact for all essential purposes we are trapped. Point <strong>B </strong>is called the local minima, and point <strong>E</strong> is called the global minima. Our goal should always be to reach the global minima. But it is generally not practical to reach a global minimum, so we at least target a local minimum that is close to the global one.</p><p>One interesting thing to note here is that if our learning rate is small, then there is no way we can come out of our local minimum <strong>B.</strong></p><p>2. <strong>Saddle Points:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/438/1*lnhKaT8-o52X0LREkPltew.png" /><figcaption>Fig. 7: Saddle point.<a href="https://www.machinecurve.com/index.php/2020/02/26/getting-out-of-loss-plateaus-by-adjusting-learning-rates/#loss-plateaus-saddle-points-and-local-minima"> source</a></figcaption></figure><p>If on your journey to the bottom of the valley, you find yourself stuck at a point like that red dot in the figure, then you are stuck at something called a <strong>saddle point</strong>.</p><p>Saddle points are points that have a zero gradient but are not an extremum. That is just a fancy way of saying that, a saddle point is not an extreme point, it is neither a maxima nor minima. In the figure, if you move to the left or right, the loss increases. While for the other two directions, the loss decreases.</p><p>Saddle points are much more difficult to escape. You might think that, what’s the difficulty in that, just go in any of the directions where loss decreases, right? Well…..just look at <strong>point D </strong>in figure 6 above.</p><p>That’s it for this blog, in the next blog, we’ll cover ways to adjust our learning rates so that our model can escape the loss plateaus.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4973aacea801" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hitchhikers-guide-to-deep-learning/13-introduction-to-deep-learning-with-computer-vision-learning-rates-mathematics-part-1-4973aacea801">13. Introduction to Deep Learning with Computer Vision — Learning Rates &amp; Mathematics — Part 1</a> was originally published in <a href="https://medium.com/hitchhikers-guide-to-deep-learning">Deep-Learning-For-Computer-Vision</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[12. Introduction to Deep Learning with Computer Vision — Receptive fields]]></title>
            <link>https://medium.com/hitchhikers-guide-to-deep-learning/12-introduction-to-deep-learning-with-computer-vision-receptive-fields-93d227ce8fd5?source=rss-5fccc8e98750------2</link>
            <guid isPermaLink="false">https://medium.com/p/93d227ce8fd5</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[receptive-field]]></category>
            <dc:creator><![CDATA[Inside AI]]></dc:creator>
            <pubDate>Thu, 18 Jun 2020 19:00:38 GMT</pubDate>
            <atom:updated>2020-06-18T19:00:38.890Z</atom:updated>
            <content:encoded><![CDATA[<h3>12. Introduction to Deep Learning with Computer Vision — Receptive fields</h3><p>written by <a href="https://medium.com/u/f1a8d777cc44">Praveen Kumar</a> &amp; <a href="https://medium.com/u/4ce50580cfab">Nilesh Singh</a>.</p><p>Receptive fields are one of the core concepts in CNN architecture. Over the years there have been a lot of architectures which deploy numerous techniques to improve their accuracy &amp; decrease overall error. However, most of them have one thing in common, clever ideas of manipulating receptive fields in order to boost accuracy.</p><p>As a beginner’s definition, the Receptive filed simply means the ability of the network architecture to see an area in the image. Let’s intuitively refine this definition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/488/1*oPOAVxEgJr6ldjfgX7y7ng.gif" /><figcaption>Fig 1: Receptive field</figcaption></figure><p>In this above image, the eye can see a circular area on the visual field. Here, the visual field is nothing but our main input image and the area which we can see is called the receptive field. The eye on the other hand depicts the various layers of CNN architecture. The last layer (one before output) can see either a part of the input image or the whole image based on how farther it is from the input image. The deeper it is, the receptive field increases.</p><p>We will define few architectures &amp; understand how their perception of the receptive field was wrong until late 2018 &amp; later on we will deep dive into the receptive field &amp; receptive field calculations.</p><h3>Why the receptive field is very important?</h3><p>Let’s look at 3 images to show you object sizes differ based on the input image &amp; see why it matters so much.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IV7POsWuweWRPu-QTT9B5A.jpeg" /><figcaption>Fig 2: Example Image 1</figcaption></figure><p>In Fig 2, the size of the dog is smaller than the size of the image. Hence to recognize that a dog is present in this image, we need a very small receptive field because even looking at a smaller field on the image, we will be obtaining all the features we require to detect a dog.</p><p>Let’s look at our second example image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*fGQqGy-OTuuZQ9hi5Ifhdg.jpeg" /><figcaption>Fig 3: Example Image 2</figcaption></figure><p>In the above image, we see that the size of the dog is the same as the image. In this case, our network needs to look at the whole image to be able to collect all the features and then detect if the dog is present or not. In this case, if we keep a smaller receptive field, we will be able to deduce fewer features and thus, our model will be less sure about the detection. Hence, in this case, we need a receptive field as the size of the image.</p><p>Let’s look at our third image now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/174/1*jG0eq1aJvWJkhahVDJcsVg.jpeg" /><figcaption>Fig 4: Example Image 3</figcaption></figure><p>In the above image, the dog is not completely visible. Now how much should be our receptive field? While you think, we tried our best to draw the following image… but don’t stop thinking about the size of receptive field ;).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qQT6N4SViNc6-tFFlPOVRg.jpeg" /><figcaption>Fig 5: Custom Dog Drawing (Xd)</figcaption></figure><p>Well, if you think that it should be surely larger than the image size, you are right. Let’s gain more understanding of this.</p><p>If we allow a model whose receptive field is the same as the size of the image in Fig 5, it will be able to see parts of a dog face and nothing more. So, it will collect fewer features and then try to detect dog. Hence the confidence score will be lesser in this case compared to the previous 2 images.</p><p><strong>Question: Why did we draw the dog in Fig 5?</strong></p><blockquote>It is to show that when the size of the object is larger than the size of the image, we need a larger receptive field to be able to see all of the features that we need from our input.</blockquote><blockquote>NOTE: It is important to understand that we have not spoken about the location of the object in the image. The location of the object does not matter as long as we have a receptive field to cover the same. However, before 2017, all the papers (ResNet, Inception &amp; Xception, VGG, AlexNet, &amp; Yolo) did not consider the receptive field as the major factor and rather spoke about object location, which is why Inception used Group convolution of different kernel sizes. We shall see all the drawbacks of these architectures as well.</blockquote><p>We should analyze the architectures to get a deeper understanding of why the receptive field is more important in denser architectures.</p><h3>Where did it start?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*g2EGJcbnQ4tliOTt0MUxBw.png" /><figcaption>Fig 6: ILSVRC [<a href="https://www.researchgate.net/figure/Recent-ConvNets-proposed-in-ILSVRC_fig1_338797371">Source</a>]</figcaption></figure><p>The above image shows the journey to different kinds of architecture. When AlexNet came, it was the start of Deep Neural Networks. Before 2012, it was all artificial neural networks. One could equate AlexNet to the discovery of fire in early stone age, simply revolutionary.</p><p>Let’s look at the few architectures and understand what was wrong in them. We will discuss the wrong intuition behind that architectures at the time when the research paper was published.</p><ul><li><strong>VGGNet [</strong><a href="https://arxiv.org/abs/1409.1556"><strong>Paper</strong></a><strong>]</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5lfBlfwmlsfV_AseNoL95g.png" /><figcaption>Fig 7: VGGNet</figcaption></figure><p>This is a plain network with layers stacked onto each other and few max pool layers in between. One of the main problems with this kind of architecture is, they can not retain the information till the last layer. When we have such simple multiple layers, we might feel it is good because the receptive field is larger, but if the object size is very small in the image, would you need an architecture such as this? No. Also, when we keep stacking up layers, we are asking the model to pass on all the features to the last layer. But this is difficult because we do not have a special connection between layers. All we have is a simple in-out connection between each layer &amp; at each layer, features are extracted based on the features received from its previous layers. So, by the time the features hit the last layer, we have lost a lot of information. This is the main issue with simple plain architectures. To summarize, we have 2 main issues with such networks.</p><ol><li>We can not train the model well if we have a variable object size in an image. (Receptive field can not know before building architecture)</li><li>No Special connections. So we lose a lot of information &amp; can not go deeper just thinking about the receptive field.</li></ol><ul><li><strong>Inception V1 [Paper]</strong></li></ul><p>This paper was present by <a href="https://medium.com/u/991272e72e68">Google Developers</a> at ILSVRC Challenge in the year 2014 when the concepts around the Deep Neural Network were not built in the right intuition as they are built today. They pointed out 3 important issues with the current CNN’s(at that time).</p><ol><li>Huge variation in the location of information, so choosing the right size of the kernel is difficult.</li><li>Very Deep networks are prone to overfitting. It is also hard to pass gradient updates.</li><li>Simply stacked architectures are hugely computation expensive.</li></ol><p>Based on our current situation, we know that point 2 &amp; point 3 are true as we have discussed above. However, point 1 claims that the object location varies in the image which is why kernel size can not be fixed. This is the wrong line of intuition as we know that location of the object does not matter if we have a kernel. It will always fire whenever it sees the object in the image, no matter the location.</p><p>So to solve for point 1, they introduced multi-sized kernels &amp; to solve for point 2, they introduced a wider architecture rather than a deeper architecture. Let’s look at building blocks of inception V1 architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/549/1*sxDw20bNgWagfdb4ixzmZA.png" /><figcaption>Fig 8: Naive Inception [<a href="https://arxiv.org/abs/1409.4842">Source</a>]</figcaption></figure><p>To solve different sized kernels, they introduced multiple kernel sizes in the architecture. However, the number of parameters was too large in this naive block. Hence they introduced a 1x1 kernel to save the number of parameters &amp; make the architecture less computation expensive. 1x1 is also very effective in reducing the number of parameters as well we have discussed <a href="https://medium.com/hitchhikers-guide-to-deep-learning/6-introduction-to-deep-learning-with-computer-vision-3x3-is-a-lie-1x1-convolutions-9edd2baf7fd5?source=collection_home---6------4-----------------------">here </a>previously. So updated architecture block is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/527/1*niSXNpdvnsET3ttPBKzo9Q.jpeg" /><figcaption>Fig 9: Reduced parameters based Inception v1 block [<a href="https://arxiv.org/abs/1409.4842">Source</a>]</figcaption></figure><blockquote>Resources:</blockquote><blockquote><a href="https://medium.com/hitchhikers-guide-to-deep-learning/10-introduction-to-deep-learning-with-computer-vision-types-of-convolutions-atrous-convolutions-3cf142f77bc0?source=collection_home---6------0-----------------------">- These types of architectures belong to group convolution.</a></blockquote><blockquote><a href="https://medium.com/coinmonks/paper-review-of-googlenet-inception-v1-winner-of-ilsvlc-2014-image-classification-c2b3565a64e7">- Inception V1 Parameter calculation and deeper understanding of whole architecture.</a></blockquote><p>Let&#39;s try to visualize the receptive field for this architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*Zi8rR8HuiY0Ay1OSAcb5QQ.jpeg" /><figcaption>Fig 10: Receptive field visualization for Inception v1</figcaption></figure><p>The Black outline represents the receptive field formed by 5x5 kernel &amp; Red represents the receptive field by 3x3 kernel. As we can see, now the model is more flexible to different sized objects. This is why if we observe the Inception-V1 architecture, there are several output connections in the middle of the architecture. These outputs are taken because an object can be found in a lower receptive field(after few initial layers), or some objects which could be almost size of the image could be found at a later stage of the model.</p><p>Now, if we consider other architectures such as ResNet, ResNeXT, SENet, these architectures also play with receptive field knowingly or unknowingly. However, the concept of receptive field has never been clearly spoken out in any of the research papers. We wanted to point out that this is one of the key factors which decides how well a model is doing.</p><p>Let’s come back to the Receptive field. Now that we understand that it is very important. Let’s look at the Human eyes receptive field.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/1*TtDTPgRPoDSF4OQ0j8rqjA.png" /><figcaption>Fig 11: Human eye receptive field [<a href="https://arxiv.org/pdf/1711.07767v3.pdf">Source</a>]</figcaption></figure><p>The human brain only has 4 layers[<a href="https://thebrain.mcgill.ca/flash/d/d_02/d_02_cl/d_02_cl_vis/d_02_cl_vis.html">Source</a>], and they are enough for us to detect anything and find any kind of pattern to learn images. However, we have even designed CNN networks with 1000+ layers, and still, it difficult for us to find the right receptive field-based architecture.</p><p>Fig 11 shows 3 different receptive fields of our brain. Few things to observe here</p><ol><li>Our receptive fields are not square boxes. It is circular unlike CNN’s.</li><li>V1 is used to capture precise data(gradients, textures), V3 is used to capture parts of objects(Patterns &amp; Parts of objects), &amp; hV4 is used to capture the whole scenario(Object itself). Note that this is what we already covered in <a href="https://medium.com/hitchhikers-guide-to-deep-learning/5-introduction-to-deep-learning-with-computer-vision-kernels-channels-neural-architecture-41b6bc4befa7">article 5</a>. The same methods are also followed by our brain, just that, in a different way.</li></ol><p><strong>NOTE: We are starting a new telegram group to tackle all the questions and any sort of queries. You can openly discuss concepts with other participants and get more insights and this will be more helpful as we move further down the publication. [Follow this</strong> <a href="https://t.me/joinchat/Kxmt-hWgGSFHAOI_jyxVSw?source=post_page---------------------------"><strong>LINK</strong></a> <strong>to join]</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=93d227ce8fd5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hitchhikers-guide-to-deep-learning/12-introduction-to-deep-learning-with-computer-vision-receptive-fields-93d227ce8fd5">12. Introduction to Deep Learning with Computer Vision — Receptive fields</a> was originally published in <a href="https://medium.com/hitchhikers-guide-to-deep-learning">Deep-Learning-For-Computer-Vision</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[11. Introduction to Deep Learning with Computer Vision — Image Augmentations]]></title>
            <link>https://medium.com/hitchhikers-guide-to-deep-learning/11-introduction-to-deep-learning-with-computer-vision-image-augmentations-f65530a67409?source=rss-5fccc8e98750------2</link>
            <guid isPermaLink="false">https://medium.com/p/f65530a67409</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Inside AI]]></dc:creator>
            <pubDate>Wed, 03 Jun 2020 12:51:32 GMT</pubDate>
            <atom:updated>2020-06-03T12:51:32.577Z</atom:updated>
            <content:encoded><![CDATA[<h3>11. Introduction to Deep Learning with Computer Vision — Image Augmentations</h3><p>Written by <a href="https://medium.com/u/f1a8d777cc44">Praveen Kumar</a> &amp; <a href="https://medium.com/u/4ce50580cfab">Nilesh Singh</a>.</p><p>It’s been a little while since our last publication. In this article, we will see how data augmentation is very important in terms of model deployment and how it changes your strategies during the training phase. We will also see several data augmentation techniques, which we like to categories them as Rich man’s data augmentation(RMDA), Middle man’s data augmentation(MDDA) &amp; Poor man’s data augmentation(PMDA) 😜 . Let’s get started.</p><p>WARNING ⚠️-&gt; This article is going to deeply extensive. Make sure you grab a cup of coffee ☕️.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*cNkPj5G1GdBR9qY4.gif" /></figure><blockquote>Github links are attached based on the Augmentation technique.<br><a href="https://github.com/aleju/imgaug">An excellent resource for you to implement general augmentation</a></blockquote><h3>What is Data Augmentation?</h3><p>Data augmentation is a strategy that enables us to significantly increase the diversity of data available for training models, without actually collecting new data. Let’s take a sample image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/605/1*9O8CdaxOrH7XauhAmDvfug.png" /><figcaption>Fig 1: Data Augmentation [<a href="https://towardsdatascience.com/machinex-image-data-augmentation-using-keras-b459ef87cd22">source</a>]</figcaption></figure><p>In the above image, we see, in our training set, we just had a single image of a lion but we produced 6 more images based on simple operations such as zooming, cropping, rotations.</p><h3>Why Data Augmentations?</h3><p>When models are built during the training phase, they are only exposed to learn a sample of the population, whereas the real world contains more diverse samples. Hence, if a model simply trained on collected images, it will fail badly during deployment.</p><p>For example, if you are training a model to recognize faces, then it is probable you collect all the face images which are under good lighting conditions &amp; the face is properly aligned to the camera. However, In real life, such scenarios are possible but it is also possible that people do not pose straight to the camera, or the lighting conditions may change. So, your model will fail as it has never seen these new scenarios during its training phase and a model is only capable of producing results based on the patterns it has learned during its training.</p><blockquote>NOTE: “Poor man”, “rich man” &amp; “middle man” names are used to express the intuition behind the strategies based on computations &amp; time complexity. We do not wish to categorize or discriminate against any race, ethics, or work done by any person in any field.</blockquote><h3>Types of Data Augmentations</h3><h4>I. Poor Man Data Augmentations</h4><p>I mean why is it called poor man data augmentation? Because Data Augmentation is a compute-intensive strategy and a poor man can not afford high compute power. Hence he has to settle with the strategies which give him a bit better results without having to spend much computer power &amp; time.</p><p>Techniques such as:</p><ul><li><strong>Scaling</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*UUGv7x7qM6xVC0D6vH6kNg.png" /><figcaption>Fig 2: Scaling [<a href="https://github.com/codebox/image_augmentor">Source</a>]</figcaption></figure><ul><li><strong>Translation</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*IYgrFOgKQpsZ_Tnk2l7Xzw.png" /><figcaption>Fig 3: Translate [<a href="https://github.com/codebox/image_augmentor">Source</a>]</figcaption></figure><ul><li><strong>Rotation</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/709/1*Cpa8K9DMdwr6Zrim38QQcQ.png" /><figcaption>Fig 4: Rotation [<a href="https://github.com/codebox/image_augmentor">Source</a>]</figcaption></figure><ul><li><strong>Blurring</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/709/1*njKfAovJSNHEwPD093kuvQ.png" /><figcaption>Fig 5: Blur [<a href="https://github.com/codebox/image_augmentor">Source</a>]</figcaption></figure><ul><li><strong>Image Mirroring</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/340/1*ffe9g1skEAaMtV0-SpVEkg.png" /><figcaption>Fig 6: Mirroring [<a href="https://github.com/codebox/image_augmentor">Source</a>]</figcaption></figure><ul><li><strong>Color Shifting / Whitening</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*lmqGEY182AZ1BHGcIACB1Q.jpeg" /><figcaption>Fig 7: Color shift [<a href="http://datahacker.rs/deep-learning-data-augmentation/">Source</a>]</figcaption></figure><p>falls under the poor man’s data augmentation. These techniques do not require intensive compute power and are best fit in scenarios where the test environment is unchanged and only minor changes on the image object are seen. Image classification such as MNIST handwritten digit’s &amp; Character recognition, Number plat recognition in daylight seems to be the right fit for such augmentations.</p><p>An excellent resource to learn and implement the above data augmentation techniques: [<a href="https://github.com/codebox/image_augmentor">LINK</a>]</p><h4>II. Middle Man Data Augmentations</h4><p>These strategies are mostly used in day to day world nowadays. People having decent enough GPU can perform these Augmentation techniques. Let’s discuss each in chronological order.</p><ul><li><strong>Elastic Distortion(2003)</strong></li></ul><p>Elastic Distortion is a technique in which the input image is deformed by change. Let’s look at an example to get a clear understanding</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/294/1*Lqd0osH01ce9HiTbGvZj7w.gif" /><figcaption>Fig 8: Elastic Distortion</figcaption></figure><p>It is one of the simplest Image augmentation techniques used. PyPI has a package named “elasticdeform” which allows us to easily manipulate the input image. [Check out: <a href="https://pypi.org/project/elasticdeform/">PyPI</a> and <a href="https://gist.github.com/erniejunior/601cdf56d2b424757de5">Github </a>if you want to implement]</p><p>Simple distortions such as translations, rotations, and skewing can be generated by applying affine displacement fields to images. This is done by computing for every pixel a new target location concerning the original location. The new target location, at position (x,y) is given concerning the previous position. For instance, if ∆x(x,y)=1, and ∆y(x,y)=0, this means that the new location of every pixel is shifted by 1 to the right. If the displacement field was: ∆x(x,y)= αx, and ∆y(x,y)= αy, the image would be scaled by α, from the origin location (x,y)=(0,0). Since α could be a noninteger value, interpolation is necessary. [<a href="http://cognitivemedium.com/assets/rmnist/Simard.pdf">Source</a>]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tjTVywi126P_BOiigVi-ug.png" /><figcaption>Fig 9: Elastic Distortion [<a href="http://cognitivemedium.com/assets/rmnist/Simard.pdf">Source</a>]</figcaption></figure><ul><li><strong>Dropout(2014)</strong></li></ul><p>Dropouts have a very interesting concept behind it. We will dive into it more intuitive but before that, we would like you to know <a href="https://medium.com/hitchhikers-guide-to-deep-learning/5-introduction-to-deep-learning-with-computer-vision-kernels-channels-neural-architecture-41b6bc4befa7">Kernels</a> &amp; <a href="https://medium.com/hitchhikers-guide-to-deep-learning/3-introduction-to-computer-vision-with-deep-learning-channels-and-feature-maps-4ae7eb932d41">Channels</a> to fully understand how Dropouts could be helpful in a Fully Connected layer rather than just CNN layers as a whole.</p><p>Let’s take a look at an example. The following image shows the normal behavior of the CNN layer(left).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_8fQOrl8qYyHkr5vOQxxNA.png" /><figcaption>Fig 10: Dropouts</figcaption></figure><p>Now suppose, if we were to blackout a few neurons(set them to 0) in the input layer, what would be the output? Of course, it changes, but it is not all that changes. When we randomly dropout neurons(blackout/set to 0), the model is now forced to find different neurons, which could help it to better classify the input. Each neuron is capable of holding a set of features based on its kernel and if we drop a few of those, the model would rather prefer to classify based on the available neurons.</p><p>This way, we are indirectly saying to the model that if were to give you 2 images of a person, one perfectly fine and second one with all the hands blacked out(pixels are set to 0 wherever hand was there in the image), then are you capable of learning to classify a person even without those hands neurons. This is a more practical scenario, right?</p><p>Dropouts could be helpful in over-fitting as stated by the <a href="https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf">original paper</a>. How? Because we might randomly drop a few redundant features in an image that is very well learned by the model. However, it is a topic of debate. Also remember, dropouts are randomly applied to neurons at each training phase. so it keeps changing the dropping neurons at each training instance. Well, it all seems to just click and work, however, we also assume that we do not randomly drop all the important features. If we do so, then the accuracy may not necessarily increase. Dropouts could be tricky and it would be difficult to judge them based on just neurons. This was recognized by LaCunn in 2015 &amp; he proposed a different strategy to implement Dropouts. He proposed <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.858.5872&amp;rep=rep1&amp;type=pdf">spatial dropouts</a>. Spatial dropouts drop the whole channel itself instead of dropping neurons. Hence, the output of a kernel is not used. Again, this dropping of kernel changes with each training instance. However, spatial dropouts are not frequently used compared to normal dropouts. We have 2 separate functions in Keras framework.</p><pre># for importing normal dropout, use<br>from keras.layers import Dropouts</pre><pre># for importing spatial dropouts, use<br>from keras.layers import SpatialDropout2D</pre><ul><li><strong>Batch Normalization(2015)</strong></li></ul><p>To understand batch normalization, refer to our article on <a href="https://medium.com/hitchhikers-guide-to-deep-learning/9-introduction-to-deep-learning-with-compute-vision-normalization-batch-normalization-ba5f60c77cf3">batch normalization</a>.</p><ul><li><strong>Cutout(2017)</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/0*pniXLkS_T71CfvQw.png" /><figcaption>Fig 11: Cutout [<a href="https://arxiv.org/pdf/1708.04552.pdf">Source</a>]</figcaption></figure><p>The cutout is a very simple augmentation technique. We simply say, cut out a portion in the image. That’s it. How is it helpful? It follows the same reason as we just discuss. If we remove random features from the image, the model looks for other features to be able to classify that image. Hence, cutting out portions in the image forces the model to learn other features as well. This is helpful for the model to learn more features rather than simply learning the dominant features in an image.</p><p>If you want to learn the code, you can make use of the following links.</p><ol><li><a href="https://github.com/uoguelph-mlrg/Cutout">Original research paper implementation</a></li><li><a href="https://github.com/yu4u/cutout-random-erasing">Simpler, easy and direct implementation</a></li></ol><p>NOTE: You must be understanding what your model is learning to be able to integrate cutout in your project. In case your model is not learning well, cutout might haunt it down further. There are interpretability methods to find out what the model is seeing in an image. One of them is GradCam.</p><ul><li><strong>Mixup(2018)</strong></li></ul><p>Mixup is a very interesting augmentation. Let’s try to understand it with the help of an example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/0*GOmPVtlchKb_sP8z" /><figcaption>Fig 12: Mixup [<a href="https://www.google.com/url?sa=i&amp;url=https%3A%2F%2Fmlexplained.com%2F2019%2F06%2F02%2Fpapers-dissected-mixmatch-a-holistic-approach-to-semi-supervised-learning-and-unsupervised-data-augmentation-explained%2F&amp;psig=AOvVaw2XoK36uSHqwHkKKiuhO6D_&amp;ust=1591116792664000&amp;source=images&amp;cd=vfe&amp;ved=0CA0QjhxqFwoTCJCSnI6K4ekCFQAAAAAdAAAAABAI">Source</a>]</figcaption></figure><p>Mixup alpha blends two images to produce a new image. This behavior forces the model to predict 2 classes. Now in a single image, we expect the model to predict 2 outcomes. Randomly mixing up images forces the model to learn features for a class. In the above example, 20% of dog features are visible, so the model is forced to detect those features and should be able to say that a dog is present and it is 20% confident of the prediction. [<a href="https://arxiv.org/pdf/1710.09412.pdf">Mixup source</a>]</p><blockquote>Mixup is very similar to <a href="https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Szegedy_Rethinking_the_Inception_CVPR_2016_paper.pdf">label smoothing</a> technique. If you don’t know label smoothing, let us give a simple example. Imagine, if we have 2 classes to predict to and our model gives out 1 for the class which it thinks is, or 0 if it thinks no for the class. So we have corresponding 0/1 labels with 0 being no class and 1 being class. if any other values are there, the model simply rejects it. So what label smoothing does is, it tell the models to not reject all values but say let’s take all the values which are greater than 0.95, so that model will reject all the values below 0.95 and convert all the values above 0.95 to 1 and accept it.</blockquote><blockquote><a href="https://github.com/yu4u/mixup-generator">Mixup implementation on keras</a></blockquote><ul><li><strong>Smart Augmentation(2017)</strong></li></ul><p>Smart augmentation was a very specific image augmentation technique. It was introduced in 2017. It can only be applied to specific classes and it provides great results. It does this by learning to merge two or more samples in one class. This merged sample is then used to train a target network. The loss of the target network is used to inform the augmenter at the same time. This has the result of generating more data for use by the target network. This process often includes letting the network come up with unusual or unexpected but highly performance augmentation strategies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/417/0*HFfLSjHynfit9X6X" /><figcaption>Fig 13: Smart Augmentation[<a href="https://arxiv.org/pdf/1703.08383.pdf">Source</a>]</figcaption></figure><p>The only drawback is, it can be only applied to images where the object size is constant and the location of the object is fixed.</p><ul><li><strong>Sample pairing(2018)</strong></li></ul><p>In this technique, synthesize a new sample from one image by overlaying another image randomly chosen from the training data (i.e., taking an average of two images for each pixel). By using two images randomly selected from the training set, we can generate N² new samples from N training samples.[<a href="https://arxiv.org/pdf/1801.02929.pdf">Source</a>]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*c11q3pxLn62YKs65wOAPjg.jpeg" /><figcaption>Fig 14: Sample pairing</figcaption></figure><p>This technique could be tricky when you implement it. This will most often give you a low training accuracy, however, when it comes to a validation error, it will help you reduce your testing error. Why so? During the training phase, the model is not able to properly classify both and hence accuracy will be lower. This may discourage you in regards to your model’s performance but now your model is also learning lots of features that could be a part of an object, so during testing, it would help it to classify better. This is what the research paper also quoted. Let’s see some stats.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-zQJSzRav4xqHuEt.png" /><figcaption>Fig 15: Error drop after sample pairing [<a href="https://arxiv.org/pdf/1801.02929.pdf">Source</a>]</figcaption></figure><p>As we can see, the error reduced with sample pairing during validation but increased during training. This is expected behavior and we do not need to worry about our model’s performance.</p><ul><li><strong>RICAP(2018)</strong></li></ul><p><a href="https://arxiv.org/pdf/1811.09030v1.pdf">RICAP</a> crops four training images and patches them to construct a new training image. It selects images and determines the cropping sizes randomly, where the size of the final image is identical to that of the original image. RICAP also mixes class labels of the four images with ratios proportional to the areas of the four images like label smoothing in the mixup. Compared to a mixup, RICAP has three clear distinctions:</p><ol><li>it mixes images spatially</li><li>it uses partial images by cropping</li><li>and it does not create features that are absent in the original dataset except for boundary patching.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/427/0*2li6qF8O2QQszFI-.png" /><figcaption>Fig 16: RICAP[<a href="https://arxiv.org/pdf/1811.09030v1.pdf">Source</a>]</figcaption></figure><p>RICAP shares concepts with cutout, mixup, and label smoothing, and potentially overcomes their shortcomings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*B5LUbTgepeJIf-qR.png" /><figcaption>Fig 17: RICAP results[<a href="https://arxiv.org/pdf/1811.09030v1.pdf">Source</a>]</figcaption></figure><p>As we see, with RICAP, the results are much better.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*acqBd54sZX8jQt_Gsd2clg.png" /><figcaption>Fig 18: RICAP heatmap[<a href="https://arxiv.org/pdf/1811.09030v1.pdf">Source</a>]</figcaption></figure><p>Fig 18 shows the heatmap of where the model is looking to classify the corresponding input image. The heatmaps of the baseline model are not looking at the right location to decide the output class, but after RICAP, the heatmaps are much accurate and the model is looking at the right location.</p><blockquote><a href="https://github.com/koshian2/keras-ricap">RICAP implementation on Keras</a></blockquote><ul><li><strong>CutMix(2019)</strong></li></ul><p>In this technique, patches are cut and pasted among training images where the ground truth labels are also mixed proportionally to the area of the patches. Instead of simply removing pixels, we replace the removed regions with a patch from another image. The ground truth labels are also mixed proportionally to the number of pixels of combined images. CutMix now enjoys the property that there is no uninformative pixel during training, making training efﬁcient while retaining the advantages of regional dropout to attend to non-discriminative parts of objects. The added patches further enhance localization ability by requiring the model to identify the object from a partial view. The training and inference budgets remain the same. CutMix shares similarity with Mixup which mixes two samples by interpolating both the image and labels. While certainly improving classiﬁcation performance, Mixup samples tend to be unnatural. CutMix overcomes the problem by replacing the image region with a patch from another training image.[<a href="https://arxiv.org/pdf/1905.04899.pdf">Source</a>]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/649/0*ozzvazPQ3A4kp1P1.png" /><figcaption>Fig 19: CutMix stats [<a href="https://arxiv.org/pdf/1905.04899.pdf">Source</a>]</figcaption></figure><p>The table shows significant improvements in the results in ImageNet classification, Localization, and VOC detections.</p><blockquote><a href="https://github.com/DevBruce/CutMixImageDataGenerator_For_Keras">Keras implementation of CutMix</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/0*Ff3WDnmez7pJIedQ" /></figure><p>We know you feel this way, but just hold on for few more minutes. Knowledge is worth the risk 😜 😉</p><h4>III. Rich Man Data Augmentations</h4><p>Very rich kids/companies can afford these kinds of techniques. They cost a lot and require 1000’s of GPU hours to find out the optimal augmentation technique. These are strategies rather than argumentation techniques. We run these strategies on the dataset and pass which image augmentation technique we wish to use. These strategies then test 100’s of values for those augmentations and then find out the optimal values we should use for that augmentation technique. So, these strategies analyze the dataset multiple times and give out those values. Even though we explore our search space to find the best values, it comes with its advantages and disadvantages.</p><p>Advantages:</p><ol><li>Results can be directly applied to the dataset.</li><li>Learned policies can be transferred to other datasets</li><li>Archives state-of-the-art results on ImageNet, &amp; Cifar 10, Cifar 100.</li></ol><p>Disadvantages:</p><ol><li>Takes 1000’s of GPU hours.</li></ol><p>Let’s discuss a few rich man’s data augmentations strategies.</p><ul><li><strong>AutoAugment(2018)</strong></li></ul><p>AutoAugment is a reinforcement based search techniques, that takes a set of values as its space search and explore this space search over multiple augmentation techniques (Also called operations). In the <a href="https://arxiv.org/pdf/1805.09501.pdf">paper</a>, they explain how each search space(which is also called a policy) could have multiple sub-policies. Each sub policy consists of 2 parts. First, which operation to be applied(rotations, zoom, scaling, etc), and second, what values of these operations must be explored. The search algorithm finds the best possible values for each operation. These results, when applied to current state-of-the-art results, improve them by a very significant margin, and gives out the new state-of-the-art results. The following tables depict the same.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yvu4NQ9ATGM9L4gB.png" /><figcaption>Fig 20: AutoAugment results [<a href="https://arxiv.org/pdf/1805.09501.pdf">Source</a>]</figcaption></figure><p>The search space algorithm is responsible for finding out the best policy. The search algorithm has two components: a controller, which is a recurrent neural network, and the training algorithm. At each step, the controller predicts a decision produced by a softmax; the prediction is then fed into the next step as an embedding. Let’s see an image to gain more understanding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ud7NE7ZoOMHYPzj-.png" /><figcaption>Fig 21: Policy and sub policy in AutoAugment [<a href="https://arxiv.org/pdf/1805.09501.pdf">Source</a>]</figcaption></figure><p>In Fig 21, we have an original image(policy), and corresponding 5 sub-policies. Each sub-policy has 3 parameters, namely, operation type, probability, and magnitude. In total, the controller has 30 softmax predictions (10*3 batches) to predict 5 sub-policies, each with 2 operations, and each operation requiring an operation type, probability, and magnitude.</p><blockquote>Fact: Operations considered are ShearX/Y, Translate X/Y, Rotate, AutoContrast, Invert, Equalize, Solarize, Posterize, Contrast, Color, Brightness, Sharpness, Cutout, Sample Pairing. Each operation also comes with a default 10 range of magnitudes. There are roughly 2.9x1032 possibilities. 😅 Now we know where do those 1000’s of GPU hours go.</blockquote><p>Moving on…</p><ul><li><strong>Optimal DataAug Policies via Bayesian Optimization(2019) [Bayesian Methods — Type 1]</strong></li></ul><p>Bayesian optimization type 1 follows a similar approach as we have discussed in AutoAugment. However, a few important differences are needed to be list down:</p><ol><li>AutoAugment uses Recurrent Neural Network &amp; Reinforcement Learning for exploring the search space, whereas the Bayesian method uses a global Bayesian optimization algorithm to explore the bayesian space.</li><li>AutoAugment uses 5 sub-policies whereas the Bayesian method uses 3 sub-policies.</li><li>Each sub-policy in the Baysian method uses a group of 2 consecutive operations [Example: (Rotation, scaling) or (Zoom, translation) … These are random combination but the important point is, they are always 2 consecutive operations] instead of a single operation in AutoAugment.</li><li>AutoAugment requires at max 15000’s GPU hours on ImageNet whereas the Bayesian method requires 800 GPU hours.</li></ol><blockquote><a href="https://github.com/zhangxiaozao/BO-Aug">Bayesian Optimization implementation</a></blockquote><ul><li><strong>Optimal DataAug Policies via Bayesian Optimization(2019) [Bayesian Methods — Type 2]</strong></li></ul><p>Type 2 is a very different case than type 1. Do you know about GAN’s? If you do, good and then you also know they are not easy to train. If you don’t know about GAN, do not worry. For now, just understand, its some kind of black boxes, that can learn to produce new images or generate similar images based on an input image.</p><p>So…coming back to Type 2 method. Observe the following image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/1*CgezR0xxrkYe8piM_0ZHVw.png" /><figcaption>Fig 22: Bayesian Optimization-based Augmentation — Type 2</figcaption></figure><p>In Fig 22, we have a training dataset (Observed training set) and a synthesized training set produced by the generator network (GAN). We concatenate these 2 sets to form an Augmented training set, which is later on used to feed the classifier network. Now we have more training images and we can use them to help our model to be trained on more variant samples. So the variance distribution of the dataset increases. Let’s see example images that are produced by the GAN.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/1*0NOBC2pdEJlqxpJFPOWdIA.png" /><figcaption>Fig 23: Images produced by GAN</figcaption></figure><p>In Fig 23, the first row is the original images and the rest of rows are produced by GAN. Fig 23 a) is a simple MNIST dataset, b) is Cifar10 dataset with classes airplane, automobile, bird and ship, c) is Cifar100 dataset with classes apple, fish, rose, and crab. We can see that the GAN produced similar but little variant images based on the original images. This enlarges the datasets and increases variance.</p><blockquote><a href="https://github.com/zhangxiaozao/BO-Aug">Bayesian Optimization implementation</a></blockquote><ul><li><strong>Population-based Augmentation(2019)</strong></li></ul><p>People with extra resources have got to do something crazy. This augmentation is a set example for that. Population-based augmentation(PBA) follows a unique idea to train our models. Let’s understand why and how.</p><p>The general norms of training a model are, to select a set of augmentation techniques, let the model train for 100’s of epochs and get some metrics. So, once we choose a set of augmentation strategies, we randomly shuffle and apply them to our dataset throughout the training phase. But! PBA is from another planet. What PBA proposes is, we make a bag of strategies, and based on the epochs numbers, we will choose different strategies and apply them. So after every N random number of epochs, we change the augmentation strategy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/979/1*mObFb7GXziRL9SQViwOZ1g.png" /><figcaption>Fig 24: PBA implementation [<a href="https://arxiv.org/pdf/1905.05393v1.pdf">Source</a>]</figcaption></figure><p>From Fig 24, we can see that no augmentation strategies were picked from epoch 0–11, and 12–33 has several strategies but with less magnitude. This consumes a lot of resources(less than AutoAugment) but can be very effective if it’s in your power to implement on your project. Look at some state-of-the-art results produced by PBA.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/0*x-tBsNUG6u8ZXcco.png" /><figcaption>Fig 25: PBA results [<a href="https://arxiv.org/pdf/1905.05393v1.pdf">Source</a>]</figcaption></figure><p>PBA example of “Car” class in the Cifar10 dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*1SJW4ToGuB0RZeDLucycKA.png" /><figcaption>Fig 26: PBA example [<a href="https://arxiv.org/pdf/1905.05393v1.pdf">Source</a>]</figcaption></figure><ul><li><strong>Patch-based Gaussian augmentation(2019)</strong></li></ul><p>Patch-based Gaussian augmentation(PGA) rightfully points out that previous techniques such as cutout increases clean accuracy but not robustness. What we mean by that is, adding cutout allows the model to learn more features thus helping it to classify the object more cleanly, but when it comes to variations, the cutout is not robust to classify those variations in the image. PGA overcomes the tradeoff between clean accuracy and robustness. By adding a patch, we increase clean accuracy, and by adding Gaussian noise, we overcome robustness. This produces state-of-the-art performance on Cifar10 &amp; ImageNet. The following image shows robustness and clean accuracy stats on ImageNet of ResNet architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/0*GwEq1CdXGNgXMxMU.png" /><figcaption>Fig 27: PGA results [<a href="https://arxiv.org/pdf/1906.02611v1.pdf">Source</a>]</figcaption></figure><p>Let’s see an example of PGA.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/0*1JmRvFFF0TDvjert.png" /><figcaption>Fig 28: PGA example [<a href="https://arxiv.org/pdf/1906.02611v1.pdf">Source</a>]</figcaption></figure><blockquote><a href="https://github.com/aleju/imgaug">Keras Implementation of Gaussian Augmentation</a></blockquote><p>Now you are free to fly off.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/0*vNsGKxjGDIbTd9ve.gif" /></figure><p>This was by far the most extensive article among all the previous ones. This was tiring but we hope this article would help you understand the right strategies and techniques you need to improve your project. We hope you enjoyed it. See you soon!</p><p><strong>NOTE: We are starting a new telegram group to tackle all the questions and any sort of queries. You can openly discuss concepts with other participants and get more insights and this will be more helpful as we move further down the publication. [Follow this</strong> <a href="https://t.me/joinchat/Kxmt-hWgGSFHAOI_jyxVSw?source=post_page---------------------------"><strong>LINK</strong></a> <strong>to join]</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f65530a67409" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hitchhikers-guide-to-deep-learning/11-introduction-to-deep-learning-with-computer-vision-image-augmentations-f65530a67409">11. Introduction to Deep Learning with Computer Vision — Image Augmentations</a> was originally published in <a href="https://medium.com/hitchhikers-guide-to-deep-learning">Deep-Learning-For-Computer-Vision</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[10. Introduction to Deep Learning with Computer Vision— Types of Convolutions & Atrous Convolutions]]></title>
            <link>https://medium.com/hitchhikers-guide-to-deep-learning/10-introduction-to-deep-learning-with-computer-vision-types-of-convolutions-atrous-convolutions-3cf142f77bc0?source=rss-5fccc8e98750------2</link>
            <guid isPermaLink="false">https://medium.com/p/3cf142f77bc0</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Inside AI]]></dc:creator>
            <pubDate>Sun, 16 Feb 2020 16:37:37 GMT</pubDate>
            <atom:updated>2020-02-16T16:38:48.811Z</atom:updated>
            <content:encoded><![CDATA[<p>Written by <a href="https://medium.com/u/f1a8d777cc44">Praveen Kumar</a> &amp; <a href="https://medium.com/u/4ce50580cfab">Nilesh Singh</a>.</p><p>Even though we have only covered <a href="https://medium.com/hitchhikers-guide-to-deep-learning/5-introduction-to-deep-learning-with-computer-vision-kernels-channels-neural-architecture-41b6bc4befa7">3x3 convolutions</a> &amp; <a href="https://medium.com/hitchhikers-guide-to-deep-learning/6-introduction-to-deep-learning-with-computer-vision-3x3-is-a-lie-1x1-convolutions-9edd2baf7fd5">1x1 Convolutions</a>, we still have a lot of different types of kernels that are used in the Computer vision world. We will go through each and possibly all the pros &amp; cons. Let’s get the hang of convolutions &amp; then dive into its types. We will be covering</p><ol><li>Normal Convolution</li><li>Pointwise convolutions | 1x1 Convolution</li><li>Dilated convolution | Atrous Convolution</li><li>Deconvolution | Transpose convolution | Fractionally Strided convolution</li><li>Pixel Shuffle convolution for Super-resolution</li><li>Depthwise separable convolution</li><li>Spatially separable convolution</li><li>Grouped convolution</li></ol><p>Let’s get started.</p><h3>What are Convolutions?</h3><p>The process of extracting features from input data using <a href="https://medium.com/hitchhikers-guide-to-deep-learning/introduction-to-computer-vision-with-deep-learning-filters-and-kernels-a65eadc9edeb">kernels/filters</a>. Filter moves discretely on top of data-plane or a <a href="https://medium.com/hitchhikers-guide-to-deep-learning/3-introduction-to-computer-vision-with-deep-learning-channels-and-feature-maps-4ae7eb932d41">channel</a> scales the input data equal to the size of its receptive field, sums these results and creates a new <a href="https://medium.com/hitchhikers-guide-to-deep-learning/3-introduction-to-computer-vision-with-deep-learning-channels-and-feature-maps-4ae7eb932d41">feature map</a>.</p><h3>Different types of convolutions</h3><h3><strong>1. Normal Convolutions | 3x3 Convolution:</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*CfNnqc6aEchW56uSlWdpxw.gif" /><figcaption>Fig 1: 3x3 Convolution</figcaption></figure><p>This is simply a 3x3 kernel running over an input image or channel to be specific. The resultant size is reduced by 2 pixels on all sides. We have covered a lot about this in previous blogs as well. Read more <a href="https://medium.com/hitchhikers-guide-to-deep-learning/5-introduction-to-deep-learning-with-computer-vision-kernels-channels-neural-architecture-41b6bc4befa7">here</a>.</p><h3><strong>2. Pointwise Convolutions | 1x1 Convolutions</strong>:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*EFkzKZHHPMXXZFZI2ACWKQ.gif" /><figcaption>Fig 2: 1x1 Convolutions</figcaption></figure><p>The beautiful 1x1 convolutional kernels are used to reduce the number of channels (also increase if needed). Read more about 1x1 from our previous blog <a href="https://medium.com/hitchhikers-guide-to-deep-learning/6-introduction-to-deep-learning-with-computer-vision-3x3-is-a-lie-1x1-convolutions-9edd2baf7fd5">here</a>.</p><blockquote>Before going to learn about more different types of convolutions, let me ask you one question.</blockquote><blockquote>We know that we get a receptive field of 3x3 when we use a 3x3 kernel because a 3x3 kernel is only capable of looking at that 3x3 area segment on the layer before it. Take a look at this image</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*vBgLT6Yys2JpGP575CpXfQ.png" /><figcaption>Fig 3: Receptive Field</figcaption></figure><blockquote>Here, each pixel on layer 2 is only capable of looking at the 3x3 area on layer 1. So we say the local receptive field of layer 2 is 3x3 with respect to layer 1. For layer 3, receptive field with respect to layer 2 is 3x3 but with respect to layer 1, its 5x5. <strong><em>So, the question is, If we have only 2 layers, and a 3x3 kenerl, how can we get a receptive field of 5x5?</em></strong></blockquote><p>Think! Think! &amp; let’s move on…</p><h3><strong>3. Dilated convolutions | Atrous Convolutions:</strong></h3><p>Dilated convolutions are a way of increasing the receptive field of the network. These helps in understanding the overall picture rather than finer details. Thus, they cut the compute cost and provide a global view of the network in much lesser depth (Indeed they are also our answer to the previous question we just asked earlier). Let’s take a look at this animation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QmiemuNYPcpDJlpQOTFCDg.gif" /><figcaption>Fig 4: Dilated Convolutions</figcaption></figure><p>In this, we can see that there is a 3x3 kernel(top) and a 7x7 input image. Generally, when we have a 3x3 kernel, it looks at 9(3x3 area) pixels on the input image without any gap, but in this case, the kernel is looking at 25(5x5 area) pixels at every stride because the dilation rate is 2. Dilation rate is simply the gap. If we have a dilation rate of 1, we get a normal 3x3 kernel. Let’s look at the following image</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*HYT3Fusj0d9awVcHXqw8bQ.jpeg" /><figcaption>Fig 5: Dilation Rate</figcaption></figure><p>With D=1, we get our normal 3x3 receptive field using a 3x3 kernel. When D=2, we get a receptive field of 5x5 using 3x3 kernel &amp; with D=3, we get a receptive field of 7x7 with 3x3 kernel.</p><p>Dilated convolutions can be used where the object size is almost equal to the size of the image. In these cases, learning a few features of the object will give the right prediction. Hence we will not need every feature to make predictions. Thus, dilation convolutions will cut the computation cost &amp; faster predictions. Also, it can be used in Pixel segmentations, Image super resolutions, denoising, keypoint detection, etc.</p><p>In many such applications, one wants to integrate information from different spatial scales and balance two properties:</p><ol><li>local, pixel-level accuracy, such as precise detection of edges, and</li><li>integrating the knowledge of the wider, global context</li></ol><p>Another classic example where dilation is used is: Google’s Wavenet</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*GFM988bCGbEROrzHzrWvow.png" /><figcaption>Fig 6: Wavenet</figcaption></figure><p>Refer to <a href="http://sergeiturukin.com/2017/03/02/wavenet.html">this</a> amazing blog for Wavenet &amp; Dilated convolutions.</p><h3><strong>4. Deconvolution | Transpose convolution | Fractionally Strided Convolution:</strong></h3><p>In the real world, we have image data that has correlations within them. Due to this, normal convolutions are expected to re-learn the data. So, deconvolutions come into the picture, which “optimally” removes pixel-wise and channel-wise correlations before the data is fed to subsequent layers.</p><p>If you need a convolution where you need an output size larger than your input image, deconvolution would be the answer. If you have a 2x2 image and a 3x3 kernel running on top of it, and you need an output of 4x4, we have 2 strategies.</p><ol><li>Use padding — It means that, we pad the borders of the input image by either random values or some specific values based on the image. The following animation depicts it.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/212/1*jVbfcVQJpgLVPHGXgljeFg.png" /><figcaption>Fig 7: Padded Image</figcaption></figure><p>This strategy produces the result we would want but is very inefficient because we are added (most of the time) useless pixels on the border to increase image size.</p><p>2. Use deconvolution — We can do something like dilation. We can spread out the pixels of the original image and pad the spaces with some values. The following animation depicts the same.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/1*Pfp2iGtf67RL_VfA--DI6Q.gif" /><figcaption>Fig 8: Deconvolution</figcaption></figure><p>This is more efficient since the pixels are spread out to different areas. So when our kernel runs over it, it can combine the result in a much better way from different areas rather than the previous approach where all the pixels are centered.</p><p>We also have a disadvantage of using Deconvolution, which is, it leads to something called Checkerboard issue. The <a href="https://distill.pub/2016/deconv-checkerboard">paper</a> “Deconvolution and Checkerboard Artifacts” has an excellent description of this behavior. Please check out this article for more details. Here, I just summarize a few key points.</p><p>Checkerboard artifacts result from an “uneven overlap” of transposed convolution. Such overlap covers some places more than others.</p><p>In the image below, the layer on the top is the input layer, and then layer on the bottom is the output layer after transposed convolution. During transposed convolution, a layer with a small size is mapped to a layer with a larger size.</p><p>For example (a), the stride is 1 and the filer size is 2. As outlined in red, the first pixel on the input maps to the first and second pixels on the output. As outlined in green, the second pixel on the input maps to the second and the third pixels on the output. The second pixel on the output receives information from both the first and the second pixels on the input. Overall, the pixels in the middle portion of the output receive the same amount of information from the input. Here exist a region where kernels overlapped. As the filter size is increased to 3 in the example (b), the center portion that receives most information shrinks. But this may not be a big deal since the overlap is still even. The pixels in the center portion of the output receive the same amount of information from the input.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Lpn_GxK1bf8D-atf.png" /><figcaption>Fig 9: The image is adopted and modified from the paper (<a href="https://distill.pub/2016/deconv-checkerboard/">link</a>).</figcaption></figure><p>Now for the example below, we change stride = 2. In the example (a) where filter size = 2, all pixels on the output receive the same amount of information from the input. They all receive information from a single pixel on the input. There is no overlap of transposed convolution here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*e9i9d71wFeq8P7jD.png" /><figcaption>Fig 10: The image is adopted and modified from the paper (<a href="https://distill.pub/2016/deconv-checkerboard/">link</a>).</figcaption></figure><p>If we change the filter size to 4 in the example (b), the evenly overlapped region shrinks. But still, one can use the center portion of the output as the valid output, where each pixel receives the same amount of information from the input.</p><p>However, things become interesting if we change the filter size to 3 and 5 in the example (c) and (d). For these two cases, every pixel on the output receives a different amount of information compared to its adjacent pixels. One cannot find a continuous and evenly overlapped region on the output.</p><p><em>The transposed convolution has an uneven overlap when the filter size is not divisible by the stride. This “uneven overlap” puts more of the paint in some places than others, thus creates the checkerboard effects. In fact, the unevenly overlapped region tends to be more extreme in two dimensions. There, two patterns are multiplied together, the unevenness gets squared.</em></p><p>Two things one could do to reduce such artifacts while applying transposed convolution. First, make sure you use a filer size that is divided by your stride, avoiding the overlap issue. Secondly, one can use transposed convolution with stride = 1, which helps to reduce the checkerboard effects. However, artifacts can still leak through.</p><h3><strong>5. Pixel Shuffle convolution for high resolution</strong></h3><p>Pixel shuffle is similar to Christopher Nolan&#39;s movies, which means, they are always mind-bending. The main aim of the pixel shuffle is for super-resolution (say, from 100x100 image, we want a 300x300 image without losing a significant amount of information). The pixel shuffle technique follows 2 simple steps. These are:</p><ol><li>Take the input image(MxNxC), run over a 1x1 kernel to extract the CxC number of channels. (We will discuss why 3)</li><li>From these 1,2,3,…, N channels, where N is CxC, take one pixel at a time from each consecutive layer from 1,2 to N and then shuffle them to get the output.</li></ol><p>Let’s look at this with an example</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*pvO9Z0BBWubea9PRtXTVxQ.png" /><figcaption>Fig 11: Pixel Shuffle</figcaption></figure><p>In fig 9, the input image size is 6x6 and 0’s are padded on the border (Final input size: 7x7xC). Now we use 1x1xC kernels over it to get 7x7x(CxC) feature map. The final feature map is from 2nd to the right. Now if you observe the final output image, it consists of shuffled pixels. Here is a closer look</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*8qn4vl3UdaCKJbVgptC_uw.png" /><figcaption>Fig 12: A Closer look at the output</figcaption></figure><p>We see 9 feature maps and each pixel from the consecutive layer is chosen and arranged in a 3x3 subsection in the output. So, the 1st pixel of each feature map is chosen and arranged as a 3x3 grid. Then the 2nd pixel is chosen from all the feature maps and arranged as another 3x3 grid. Since we have a 7x7x9 feature map, our output is 21x21x1 because we took each pixel from 9 feature maps, and arranged each pixel into 3x3 grids. So, we get 7*3 = 21 output size. Look at this animation now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*A6i6__d2OMJOJStUQffrkw.gif" /><figcaption>Fig 13: Pixel shuffle animation</figcaption></figure><p>To read more, read <a href="https://arxiv.org/pdf/1609.05158.pdf">this</a> original paper &amp; <a href="https://www.youtube.com/embed/nG3tT31nPmQ?start=2128">this</a> video.</p><h3><strong>6. Depthwise Separable Convolution</strong></h3><p>Let’s say we have a 7x7 image with 3 channels(7x7x3). So we have to use a 3x3 kernel with 3 channels (3x3x3) to convolve on top of it to get the output. Take a look at this animation which depicts it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*693FMt7oCSP2_4J_lLYSUg.gif" /><figcaption>Fig 14: Normal convolution</figcaption></figure><p>Depthwise convolution says that instead of a 7x7x3 input image, we can say its 3 7x7x1 images. Hence we would now need a 3x3 kernel with just 1 channel for each 7x7x1 image. Let’s look at an example</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/1*nKEaitLzL73W0r-cZQc1ag.png" /><figcaption>Fig 15: Depthwise convolution</figcaption></figure><p>In fig 13, the input image is split into multiple images and then the kernel is convolved over it to get multiple feature maps. These feature maps are stacked over each other and then a 1x1xC (pointwise convolution) is used to get the desired number of channels. Look at the following animation</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*OrIzzK51dQQGjXMiUD3gxg.gif" /><figcaption>Fig 16: Depthwise convolution animation</figcaption></figure><p>One of the main advantages of Depthwise convolution is that it reduced the number of parameters by 2 without even losing a significant amount of information. let’s take a look at an example of parameter calculation.</p><p>Let, Input image be 7x7x3 and we use a 3x3 kernel and we want 32 output feature maps and we are using padding.</p><p><strong><em>In normal convolution:</em></strong></p><p>(7x7x3)x[(3x3x3)x32] =&gt; (7x7x32)</p><p><strong><em>In Depthwise convolution:</em></strong></p><p>3x(7x7x1) x {<strong>[3x(3x3x1)]</strong>+<strong>[1x1x3]x32</strong>} =&gt; (7x7x32)</p><blockquote>{[arg1]+[arg2]} where arg1 is 3x(3x3x1) &amp; arg2 is (1x1x3)x32, which is the number of parameters required for convolving the 3x3 kernel over multiple images and then 1x1 pointwise convolution to obtain the required number of feature maps, which results in a number of parameters calculated from arg2.</blockquote><p>If we divided LHS, we get a ratio of 864/91 or 9.49. We see that the number of parameters is reduced by a 9.49 factor, which can be approximated to 10 and it is more significant when we have a larger input size. This allows us to have a deeper or wider architecture without worrying about parameters. Architecture like Inception, Xception &amp; mobile net uses these convolutions a lot.</p><h3><strong><em>7. Spatially Separable Convolution</em></strong></h3><p>Spatially separable convolution is similar to the Depthwise convolution. It is also used when the number of parameters is a matter of concern.</p><p>Spatially Separable convolution makes use of 2 different kinds of kernels to produce the output. Let’s go deeper into it with the help of an example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*sWZJ-vFNUCErNV9_WJ0wdw.gif" /><figcaption>Fig 17: Spatially Separable Convolution</figcaption></figure><p>In fig 15, the input is a 7x7x3 image. we convolve a 1x3x1 kernel to get an output of 7x5x1 ( Only the column dimension is reduced by 2 but rows are unchanged). Then we use a 3x1x1 kernel to get the final output as 5x5x1.</p><p>The choice of using (1x3) or (3x1) at first depends on you.</p><p>This architecture is also heavily used in Xception-Inception architectures.</p><h3>8. Grouped Convolutions</h3><p>Grouped convolutions were initially mentioned in AlexNet and later reused in ResNext. The main motivation of such convolutions is to reduce computational complexity while dividing features into groups. Each group is expected to find out specific features from the network. These group features are then combined to form a single and denser feature map.</p><p>Here we describe how the grouped convolutions work. First of all, conventional 2D convolutions follow the steps showing below. In this example, the input layer of size (7 x 7 x 3) is transformed into the output layer of size (5 x 5 x 32) by applying 32 filters (each filter is of size 3 x 3 x 3).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R3lJHQf0g5Ef5MQtrO1ZVw.png" /><figcaption>Fig 18: Standard 2D convolution.</figcaption></figure><p>In grouped convolution, the filters are separated into different groups. Each group is responsible for conventional 2D convolutions with a certain depth. The following examples can make this clearer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qwsgAF_9F1gNlfOmmgOdAw.png" /><figcaption>Fig 19: Grouped convolution with 2 filter groups.</figcaption></figure><p>Above is the illustration of grouped convolution with 2 filter groups. In each filter group, the depth of each filter is only half of that in the nominal 2D convolutions. They are of depth Din / 2. Each filter group contains Dout /2 filters. The first filter group (red) convolves with the first half of the input layer ([:,:, 0: Din/2]), while the second filter group (blue) convolves with the second half of the input layer ([:,:, Din/2: Din]). As a result, each filter group creates Dout/2 channels. Overall, two groups create 2 x Dout/2 = Dout channels. We then stack these channels in the output layer with Dout channels.</p><p>This was a long one. Grab a nap for ourselves and let all these different kinds of convolutions sink in.</p><p>Hope you enjoyed it. See you soon…</p><p><strong><em>NOTE: we are starting a new telegram group to tackle all the questions and doubts. You can openly discuss concepts with other participants and get more insights and this will be more helpful as we move further down the publication.[Follow this </em></strong><a href="https://t.me/joinchat/Kxmt-hWgGSFHAOI_jyxVSw"><strong><em>LINK</em></strong></a><strong><em> to join]</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3cf142f77bc0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hitchhikers-guide-to-deep-learning/10-introduction-to-deep-learning-with-computer-vision-types-of-convolutions-atrous-convolutions-3cf142f77bc0">10. Introduction to Deep Learning with Computer Vision— Types of Convolutions &amp; Atrous Convolutions</a> was originally published in <a href="https://medium.com/hitchhikers-guide-to-deep-learning">Deep-Learning-For-Computer-Vision</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[9. Introduction to Deep Learning with Compute Vision — Normalization & Batch Normalization]]></title>
            <link>https://medium.com/hitchhikers-guide-to-deep-learning/9-introduction-to-deep-learning-with-compute-vision-normalization-batch-normalization-ba5f60c77cf3?source=rss-5fccc8e98750------2</link>
            <guid isPermaLink="false">https://medium.com/p/ba5f60c77cf3</guid>
            <dc:creator><![CDATA[Inside AI]]></dc:creator>
            <pubDate>Tue, 21 Jan 2020 17:27:01 GMT</pubDate>
            <atom:updated>2020-01-21T17:27:01.661Z</atom:updated>
            <content:encoded><![CDATA[<h3>9. Introduction to Deep Learning with Compute Vision — Normalization &amp; Batch Normalization</h3><p>Written by <a href="https://medium.com/u/4ce50580cfab">Nilesh Singh</a> &amp; <a href="https://medium.com/u/f1a8d777cc44">Praveen Kumar</a>.</p><p>After learning about <a href="https://medium.com/hitchhikers-guide-to-deep-learning/8-introduction-to-deep-learning-with-computer-vision-activation-functions-22dbcf5f7cd1?source=friends_link&amp;sk=0967760bb51d3e55048e9848ff2b41ec">Activation functions</a>, which are very crucial for our deep neural network as we move on to build more complex and amazing architectures, its time we focus on more important concepts.</p><h3><strong>What is normalization?</strong></h3><p>Okay, let us suppose you and your friend are working in a company on different projects. After the completion of the project, you both are supposed to get a bonus. Let’s suppose you get a bonus of 1000 rupees per month, not good but you still accept it. Then you head over to your friend and ask about his bonus, apparently, he also got a bonus of 1000 rupees, ahh the relief. But as it turns out he got it on a daily basis. Ughh, not fair, so not fair.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*pz4Y4wwL1QbrXt9pTYSMFQ.gif" /></figure><p>Normalization is used to prevent the exact same thing happening inside a neural network. Normalization, specifically <strong><em>pixel normalization</em></strong> is used to assign every pixel of an RGB image to an appropriate point in the 0 to 255 scale. This is done by scaling the brightest pixel to 255 and the darkest pixel to 0. In simple terms, it redistributes all the pixels throughout the whole scale and helps in preventing pixel intensity clustering.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*R9ArsxFRoBrJ6R_NdXGT1w.png" /><figcaption>Fig 1: Normal Image Vs Normalized Image</figcaption></figure><p>You can observe the difference in the quality of features in image just with normalization, where the bottom image is of-course pixel normalized.</p><p>Now let us talk about <strong><em>image normalization</em></strong>,</p><p>let us suppose, we have a dataset consisting of the following images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/597/1*lU04H0grNBntg_KCxusiFA.png" /><figcaption>Fig 2: Face images</figcaption></figure><p>The idea behind image normalization is that we calculate the mean of all the images given in the dataset, then we calculate the variance of all images in the dataset. The mean (left) and variance (right) would look somewhat like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*nCZ6kVNDuVbDU1H7YEGE3w.png" /><figcaption>Fig 3: Mean of Face Images(LEFT) &amp; Variance of Face Images(RIGHT)</figcaption></figure><p>Then we subtract the mean from each image and divide it by the standard deviation. What does it do?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/597/1*SrbzqKIlS58eGsA2b38n3g.png" /><figcaption>Fig 4: New Face Images after subtracting Mean &amp; Variance</figcaption></figure><p>As is evident from the updated dataset, the features on the face are highlighted, that is, the relevant features are highlighted and distortion of features happens on the sides, i.e. irrelevant features are weeded out.</p><p>Data normalization, in turn, ensures weight normalization inherently. If our relevant features are uniform and not overblown, then the weight corresponding to them will also be uniform. This ensures that backpropagation works efficiently.</p><p>For example, if we have two weights <em>W1=0.5 and W2=100</em>. After a backpropagation step, if the intended weight update is 0.3, then <em>W1=0.5+0.3=0.8</em> and <em>W2=100+0.3=100.3</em>. Both weights are updated by the same value but the effect on W1 is far more pressing than the effect on W2 thus rendering the backpropagation step much less potent <em>(think about the bonus example)</em>.</p><p>But, if our data would have been normalized in the first place then W1 and W2 would have been of similar scale and the backpropagation could have worked its magic with much ease.</p><p>So far we’ve discussed normalization that happens at the beginning of each epoch, but what happens during the epoch, who’s to stop the weights from growing non-uniformly?</p><p>Well, this is where Batch Normalization comes into the picture.</p><h3>Batch Normalization (BN)</h3><p>Before going into BN, we would like to cover <strong>Internal Covariate Shift</strong>, a very important topic to understand why BN exists &amp; why it works.</p><p>Whenever we want to train a network, we provide a batch of input from the complete dataset and these batch keep changing until the epoch is completed. In simpler words, the whole dataset is divided into batches of images with each image in a batch being processed parallelly on the GPU. Training is complicated by the fact that the inputs to each layer are affected by the parameters of all preceding layers. So, small changes in the network parameters amplify as the network becomes deeper. The training proceeds in steps, at each step considering a <strong><em>mini-batch</em></strong>. Thus, each mini-batch has a different distribution from different parts of the dataset. So, the network has to continuously adapt to new distribution. whenever this input distribution changes, it is called <strong>Covariate Shift</strong>. Internal covariate shift simply means the changes inside network layers. Let’s look at an example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*7EASRl5uhCRcBjs4RXKhZA.gif" /><figcaption>Fig 5: Covariate Shift</figcaption></figure><p>From fig 5, we can see that the distribution varies from different datasets. This would lead to a failure model when deployed since the type of distribution the model is trained on is very different than test distribution.</p><p>Another intuitive way of saying is, we have many many kernels in our neural network and a kernel is responsible for extracting features from input images. Suppose we are trying to predict a cat. So, we can have different colors of cats as well. For simplicity, assume we have 3 different kinds of colored cats, that is, a white cat, a red cat, and a black cat. So our kernel is supposed to shout that I have found a cat in this image. But the confidence value will change as different colored cats will appear in an image. This is because, for a black cat, the input pixel values will be much larger(200–255) when compared to a white cat(0–50), whose pixel values will be less. Thus, the kernel will have different output values and when these output values are passed on the successive layers, the information is lost between different kinds of layers and activations. So we need to make sure all 3 cat inputs are between the same distribution.</p><blockquote>Thus our aim is to centralize these distributions(at least during training &amp; validation) which vary on each mini-batch. This is the main reason why BN comes into the picture.</blockquote><p>Let’s dive into BN now and the maths behind it. To understand it, follow us carefully here with the following example.</p><p>Assume your batch size is 10. So this means that at any given time, you are giving 10 images as input to the network. Also, assume that you have 3 layers in your network and 32,64, and 128 are the number of channels in each consecutive layer respectively. So this means Layer 1 has 32 channels, Layer 2 has 64 channels &amp; Layer 3 has 128 channels. Now we will focus on one layer, say the first layer, which has just 32 channels and our batch size is 10 images. Carefully look at the following diagram (<em>read it from left to right</em>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*59qVTuoXWAxoBs7546YPNw.png" /><figcaption>Fig 6: Batch Normalization Visualization</figcaption></figure><p>The first question is how are there 320(32 channels x 10 images for each channel) images even though our batch size was 10, right? This is what we call mini-batches. We know that GPU works in parallel. So when we specify 32 channels, we are making 32 different kernels inside GPU parallelly where each kernel is responsible for extracting a different feature. So each kernel is passed those 10 images instead of passing those 10 images one by one. Thus, 32 kernels work on the same images parallelly. Hence in the memory, it basically forms 320 images. Thus, we still have 10 images as our batch size, but now 32 mini-batches of the same 10 images. This is why BN has the term B in it, which means we work on batches inside layers, otherwise, it could simply be coined as Normalization.</p><p>So, BN calculates mean &amp; Variance for each mini-batch(x1,x2,…,x32) and updates that mini-batch(x1`,x2`,…,x32`) and passes to next layer and so on. Thus a very important point to note here is:</p><blockquote>The number of Means &amp; Variance is equal to the number of channels in a particular layer. Also, for each mini-batch, Mean &amp; Variance values are different. So, Mean &amp; Variance for x1 is different from the mean &amp; variance of x2 and soo on.</blockquote><p>Let’s look at what the original paper suggests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bri-58JU3HSHqGVq9lh4fA.png" /><figcaption>Fig 7: Batch Normalization Math</figcaption></figure><p>Fig 7 shows exactly what we have seen in the previous figure. For each layer and for each mini-batch, we subtract the mean &amp; divide by variance to normalize that current mini-batch and similar for all the mini-batches. We pass this updated &amp; normalized mini-natch and ask the model to learn features from it. Do not mind the parameters Gamma &amp; Beta. Those are simply the learnable parameters for the model. Basically they exist to make sure that this mini-batch normalization does not skew the data at hand towards a wrong direction. If our normalization is perfect, then Beta would be 0 and gamma would be 1. Let’s look at the batch normalization algorithm now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XKr2e_wMF_HLX6mz4doMLg.png" /><figcaption>Fig 8: Batch Normalization Algorithm</figcaption></figure><p>The algorithm could not be much simpler for us now. We have just seen those exact steps. For each mini-batch, we calculate the mean &amp; variance. We then subtract from the mini-batch to get updated mini-batch and then use this updated mini-batch to learn features.</p><p>NOTE: <em>We have an epsilon term with Variance in the denominator because we try to avoid the pitfall of divide by zero exception.</em></p><p>That’s all is Batch Normalization. Easy when visualized and tricky enough to confuse.</p><p>If you need a deeper understanding of Batch normalization &amp; the complete math behind it, please go through this <a href="http://proceedings.mlr.press/v37/ioffe15.pdf">paper</a> presented by Google at the 32nd International Conference on Machine Learning, Lille, France, 2015.</p><p>Hope you enjoyed it. See you soon!</p><p><strong>NOTE: We are starting a new telegram group to tackle all the questions and any sort of queries. You can openly discuss concepts with other participants and get more insights and this will be more helpful as we move further down the publication. [Follow this</strong> <a href="https://t.me/joinchat/Kxmt-hWgGSFHAOI_jyxVSw?source=post_page---------------------------"><strong>LINK</strong></a> <strong>to join]</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba5f60c77cf3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hitchhikers-guide-to-deep-learning/9-introduction-to-deep-learning-with-compute-vision-normalization-batch-normalization-ba5f60c77cf3">9. Introduction to Deep Learning with Compute Vision — Normalization &amp; Batch Normalization</a> was originally published in <a href="https://medium.com/hitchhikers-guide-to-deep-learning">Deep-Learning-For-Computer-Vision</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[8. Introduction to Deep Learning with Computer Vision — Activation Functions]]></title>
            <link>https://medium.com/hitchhikers-guide-to-deep-learning/8-introduction-to-deep-learning-with-computer-vision-activation-functions-22dbcf5f7cd1?source=rss-5fccc8e98750------2</link>
            <guid isPermaLink="false">https://medium.com/p/22dbcf5f7cd1</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[computer-science]]></category>
            <dc:creator><![CDATA[Inside AI]]></dc:creator>
            <pubDate>Sat, 23 Nov 2019 06:19:29 GMT</pubDate>
            <atom:updated>2019-11-23T07:27:27.579Z</atom:updated>
            <content:encoded><![CDATA[<h3>8. Introduction to Deep Learning with Computer Vision — Activation Functions</h3><p>Written by <a href="https://medium.com/u/f1a8d777cc44">Praveen Kumar</a> &amp; <a href="https://medium.com/u/4ce50580cfab">Nilesh Singh</a></p><p>After our audacious first attempt at building an actual network (<a href="https://medium.com/hitchhikers-guide-to-deep-learning/7-introduction-to-deep-learning-with-computer-vision-mnist-handwritten-digit-recognition-654efb1d2d30?source=friends_link&amp;sk=c0957719c206e896e8bb16fdb76869cc">MNIST from scratch</a>), we are finally back to theory.</p><p>In this blog we’ll try to understand:</p><p><em>What are activation functions? How do they work? Why do we require them? What are their different types? </em>and<em> which one should you choose for your network?</em></p><h3>What is the activation function?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/342/1*mIRtaiH63vUezMmERipvVg.png" /><figcaption>Fig 1</figcaption></figure><p>In simplest of terms, the activation function is a function that takes an input signal and converts it to an output signal. According to Wikipedia, “<em> In artificial neural networks, the activation function of a node defines the output of that node given an input or set of inputs.</em>”</p><p>In Fig 1, we have 3 inputs for node 2. Node 2 performs some arithmetic operations and derives the activated output. The final output is simply an activated output or an output of the activation function.</p><p>Inside node 2, we can see “<em>f activation function</em>” written. Let’s try to understand how they work and their different types.</p><h3>How do activation functions work?</h3><p>Activation functions are simply mathematical functions that act as a gateway for some gradients (the input values). For example, consider the step function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*0iOzeMS3s-3LTU9hYH9ryg.png" /><figcaption>Fig 2</figcaption></figure><p>In Fig 2, if we get an input value which is less than 0, we always return 0 and if we get input as 0 or any positive integer, we always return 1.</p><p>That’s how all the activation functions work. Each activation function is a simple mathematical formula based threshold gateway, which changes behavior as per on the input at hand and gives an activated output.</p><p>Let’s take another example to completely understand how output differentiate based on the activation function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/257/1*gkrVQrMoVDD42iU2BbTl7Q.png" /><figcaption>Fig 3</figcaption></figure><p>In Fig 3, we have a sigmoid activation function. This is very very similar to our threshold step function, but the only difference is the output for input values in the range -4 to +4. If we give input in the rage (+-4), the sigmoid function returns between 0.1 to 0.9 and at input value 0, it returns 0.5 whereas, in a threshold step function, it always returns 0 for negative input values and 1 for input values in range 0 to +infinity.</p><h3>Why do we require activation functions?</h3><p>For an artificial neural network to work and do complicated computations, it requires more than a linear representation. Activation functions provide the network with the said non-linearity. Without activation functions, our neural networks would cease to be anything more than a simple linear regression model.</p><blockquote><strong><em>Activation functions introduce non-linearity in our network.</em></strong></blockquote><p>Consider the following 2 cases:</p><ol><li><em>The parameters of a linear equation</em></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*UDIO6rwBgiOwFTHYwSHvsg.jpeg" /><figcaption>Fig 4</figcaption></figure><p>If we were to learn the parameters of the above equation, we would not require an activation function and this could be simply done by a predefined linear model.</p><p>2. <em>Learning a human face</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*l7D81pmOkAw-WOkZ2xJUZA.png" /><figcaption>Fig 5</figcaption></figure><blockquote>If we were to learn a human face, you would require a function that can allow such complex and non-linear information to pass between layers.</blockquote><p>In such cases, we want to move the complex non-linear information between layers. This is not achievable by a simple predefined linear function. And that’s when activation comes to our rescue. Activation functions also give you the ability to abstract complex mappings from data.</p><h3>Types of activation functions</h3><ol><li><strong>Sigmoid function:</strong></li></ol><p>Sigmoid is one of the earliest activation function that exists. In almost all cases, newer and more reliable functions are used over it. Nonetheless, it won’t hurt to know a little about it.</p><p>The curve for Sigmoid looks like an S.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/485/1*BwZmaMBaVW8f8lBAQUoytw.png" /><figcaption>Sigmoid Activation Function</figcaption></figure><p>As it is evident from the figure, the range of output values produced by sigmoid is 0 to 1.</p><p>Mathematically it can be expressed as:</p><p><strong><em>f(x) = 1 / 1 + exp(-x)</em></strong></p><p>Although, Sigmoid has some good characteristics such as being differentiable at any point (this comes in handy while backpropagating). But it’s pros are heavily outweighed by its vices. Sigmoid activation can cause a vanishing gradient problem and it can kill gradients as it is very sensitive to even tiny changes in weights. The convergence rate of the sigmoid is also very less.</p><p>2. <strong>Hyperbolic tangent Activation or tanh:</strong></p><p>The curve for tanh is very similar to sigmoid with the only real difference lying in output range for both activation functions. The tanh function can map input values to a output range between -1 and 1. It is centered at 0.</p><p>Mathematical representation:</p><p><strong><em>f(x) = 1 — exp(-2x) / 1 + exp(-2x) (or) 2 *sigmoid(2x)-1</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/1*lDreOpbvgVF1_I3HgjvD5A.jpeg" /><figcaption>Comparison between sigmoid and tanh curves</figcaption></figure><p>Even with the increased range, the tanh activation function suffers from semi chronic versions of the same ailments as sigmoid function and hence is not very popularly used.</p><p>3. <strong>ReLU (Rectified Linear Unit) Activation Function</strong></p><p>The name sounds quite complicated, doesn’t it? Trust me, the working is not half as complicated. Think of ReLU as a gatekeeper who knows only two rules:</p><ol><li>If the number trying to enter is greater than 0, then open the gate and let it in.</li><li>If the number is negative, then don’t let it in, instead, just send a 0 in its place.</li></ol><p>Mathematical Representation:</p><p><strong><em>f(x) = max(0,x)</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/439/1*8zOYy34M7TADT4S3y3l10A.png" /><figcaption>Curve for ReLU</figcaption></figure><p>Despite its very simple nature, ReLU outperforms nearly all other activation functions and is the most commonly used activation function across the DL community. This does not suffer from vanishing gradient problem or slow convergence rate. Also, a very big plus point with ReLU is that it is fast as it does not involve any complex calculations as tanh or sigmoid.</p><p>It does suffer from a problem called <em>dying ReLU</em>. To quote <a href="https://en.wikipedia.org/wiki/Rectifier_(neural_networks)">Wikipedia</a>:</p><blockquote>“ReLU neurons can sometimes be pushed into states in which they become inactive for essentially all inputs. In this state, no gradients flow backward through the neuron, and so the neuron becomes stuck in a perpetually inactive state and “dies.” In some cases, large numbers of neurons in a network can become stuck in dead states, effectively decreasing the model capacity. This problem typically arises when the learning rate is set too high.”</blockquote><p>4. <strong>All other kinds of ReLU:</strong></p><p>To counter the dying ReLu problem, <strong><em>Leaky ReLu</em></strong> was introduced which allowed a small negative value to pass through the neuron. The slope of this negative value depends on a parameter <em>alpha. </em>The value of <em>alpha </em>is 0.0 in the case of vanilla ReLU, if it is made 1.0 then essentially the curve on the negative side becomes x=y and we get a linear function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/332/1*t-K5jJLBADJCgYEqFZAtPQ.png" /><figcaption>Leaky ReLU with diff. alpha values</figcaption></figure><p>We have a plethora of other over-engineered versions of ReLU. They include <em>Threshold ReLu, ELU (Exponential Linear Unit), SELU (Scaled Exponential Linear Unit) </em>and so on. These activation functions are barely used and it sometimes feels that they were engineered just for the sake of it.</p><p>If you wish to read more about all activation functions in detail you can visit this <a href="https://towardsdatascience.com/deep-study-of-a-not-very-deep-neural-network-part-2-activation-functions-fd9bd8d406fc">link</a>.</p><h3>Which one should you choose?</h3><p>As as a beginner you are most likely going to use ReLU over other activation functions. Sometimes people also prefer Sigmoid and Tanh. But then that’s it. Due to their complex nature, sigmoid and tanh are actually big culprits of pulling back the DL revolution by many years. ReLU is currently the state-of-the-art and for fun, people also say it the simple-of-the-art.</p><p>Let it sink in for a while now. It is a needy concept that changes according to your problem statement. As we go further, we will also highlight some of the other pros and cons of ReLU as well as other activation functions. Let’s take some rest and get back to more concepts in upcoming articles.</p><p>Hope you enjoyed it. See you soon!</p><p><strong>NOTE: We are starting a new telegram group to tackle all the questions and any sort of queries. You can openly discuss concepts with other participants and get more insights and this will be more helpful as we move further down the publication. [Follow this</strong> <a href="https://t.me/joinchat/Kxmt-hWgGSFHAOI_jyxVSw?source=post_page---------------------------"><strong>LINK</strong></a> <strong>to join]</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=22dbcf5f7cd1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hitchhikers-guide-to-deep-learning/8-introduction-to-deep-learning-with-computer-vision-activation-functions-22dbcf5f7cd1">8. Introduction to Deep Learning with Computer Vision — Activation Functions</a> was originally published in <a href="https://medium.com/hitchhikers-guide-to-deep-learning">Deep-Learning-For-Computer-Vision</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[7. Introduction to Deep Learning with Computer Vision — MNIST Handwritten digit recognition…]]></title>
            <link>https://medium.com/hitchhikers-guide-to-deep-learning/7-introduction-to-deep-learning-with-computer-vision-mnist-handwritten-digit-recognition-654efb1d2d30?source=rss-5fccc8e98750------2</link>
            <guid isPermaLink="false">https://medium.com/p/654efb1d2d30</guid>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[diy]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Inside AI]]></dc:creator>
            <pubDate>Wed, 30 Oct 2019 20:46:11 GMT</pubDate>
            <atom:updated>2020-05-05T12:00:10.636Z</atom:updated>
            <content:encoded><![CDATA[<h3>7. Introduction to Deep Learning with Computer Vision — MNIST Handwritten digit recognition hands-on</h3><p>Written by <a href="https://medium.com/u/4ce50580cfab">Nilesh Singh</a> &amp; <a href="https://medium.com/u/f1a8d777cc44">Praveen Kumar</a>.</p><p>Prerequisites: <a href="https://medium.com/hitchhikers-guide-to-deep-learning/introduction-to-computer-vision-with-deep-learning-filters-and-kernels-a65eadc9edeb">Filters &amp; Kernels</a>, <a href="https://medium.com/hitchhikers-guide-to-deep-learning/3-introduction-to-computer-vision-with-deep-learning-channels-and-feature-maps-4ae7eb932d41">Channels &amp; Features</a></p><p>In this blog, we will learn to code and create our very first neural network. The primary task of our network will be to demonstrate the fact that it works and as a bonus, it will also identify handwritten digits autonomously.</p><p>So let’s cut the chatter and talk code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*3OBkq6AeDcwaeWwL7HJcGw.jpeg" /></figure><p>The first thing to bear in mind is that we will be using <em>Python 3.x on a google Colab</em> platform. We will be using <a href="https://keras.io/"><em>Keras</em></a> as our primary library for writing the network. It is a simple, elegant, yet extremely powerful API which runs on top of TensorFlow or Theano.</p><p>The first logical thing to do would be to fire up <a href="https://colab.research.google.com/">colab</a> if not already done.</p><p>You should get a ‘<em>Welcome to the Colaboratory’</em> page. On the top left corner, look for the ‘file’ option and go to <em>file-&gt;New Python 3 Notebook.</em></p><p>Now as the notebook is up and running, let us look at shortcuts for some commonly used tasks that can prove instrumental while using the environment.</p><p>To run each cell individually: <em>SHIFT+ENTER</em></p><p>To run all cells: <em>CTRL+F9</em></p><p>To delete a cell: <em>CTRL+M +D</em></p><p>To insert a cell below: <em>CTRL+M+B</em></p><p>To insert a cell above: <em>CTRL+M+A</em></p><p>The good news is, if you find it complicated to remember them, you can customize each shortcut using <em>preferences</em> setting or use <em>CTRL+M+H</em> to open up customization settings for all the shortcuts.</p><h3>Let&#39;s build a neural network with 7 hidden layers</h3><p><em>Click </em><a href="https://colab.research.google.com/drive/1_n8OpMUmoEtdgVbmODm8sphvvlheQkeY"><em>here</em></a><em> to find the notebook with all the code used in this article.</em></p><p>In the first block, write this import statement and press <em>shift+enter.</em></p><pre>import keras</pre><p>We will now import a few more modules. Don’t worry about understanding everything at one go, we will ease you into it as and when we use it.</p><pre>import numpy as np<br>​<br>from keras.models import Sequential<br>from keras.layers import Flatten<br>from keras.layers import Convolution2D<br>from keras.utils import np_utils<br>​<br>from keras.datasets import mnist</pre><p>Here the line worth noting is the last import statement. In the last statement, we are importing the whole <a href="https://en.wikipedia.org/wiki/MNIST_database">MNIST dataset</a>. It contains all the images and their labels which we will be using to train our handwritten digit recognition model. MNIST consists of 60k training and 10k testing images. All images are black and white and are 28X28 in size.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TLs98e2HOPsV-DH2a7UbqQ.png" /></figure><p>Now that we have all the required libraries and dataset in place. We will be splitting the dataset into training and testing parts. We will build a model that will be trained on the training set. It&#39;s like teaching a baby and asking it to learn how does an apple look like, the same goes for our model, which will learn by seeing multiple instances of handwritten digits and hopefully learn to classify them from 0 to 9.</p><p>Even though we are confident that our model has accurately learned all the digits, we need to gauge its learning by showing it some unseen images. If the model accurately classifies these new unseen images, we can confidently say, we nailed it!!!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/1*QSXxOderidihfo1itIE7cw.png" /></figure><p>Since we are using inbuilt dataset from Keras, it can be loaded and split into training and test sets with just one line of code.</p><pre>##splitting MNIST dataset into training and testing sets, images are loaded into memory as well<br>(X_train, y_train), (X_test, y_test) = mnist.load_data()</pre><p>Here<em> X_train</em> and <em>y_train</em> contains the training and testing images respectively while <em>X_test</em> and <em>y_test</em> contain their labels and follow the same nomenclature. Also, all the variables created in this step are numpy arrays.</p><p>In the next step, we add a depth dimension of 1, which signifies that our data has only one <a href="https://medium.com/hitchhikers-guide-to-deep-learning/3-introduction-to-computer-vision-with-deep-learning-channels-and-feature-maps-4ae7eb932d41">channel</a>. This is done using the <a href="https://docs.scipy.org/doc/numpy/reference/generated/numpy.reshape.html">reshape</a> function of numpy arrays.</p><pre>##Basically adding a depth dimension to the data, so our MNIST data are single channeled<br>X_train = X_train.reshape(X_train.shape[0], 28, 28,1)<br>X_test = X_test.reshape(X_test.shape[0], 28, 28,1)</pre><p>Next, we scale our data from a range of (0–255) (think RGB) to the range of (0–1). why? Because scaling allows us to scale down all the images into a single ratio of the same proportion. It helps faster training and better accuracy prediction (sometimes counter-intuitive). So we divide our images by 255 because pixels in images of RGB color model can have a maximum value of 255(2⁸-1).</p><pre>X_train = X_train.astype(&#39;float32&#39;)<br>X_test = X_test.astype(&#39;float32&#39;)<br>X_train /= 255<br>X_test /= 255</pre><p>Now, once we have scaled our images, we go on to convert our linear image labels to a categorical one. These labels are the ground truth to our model. Let’s just do this and not get into why and what. We will explain as we progress deeper into the sessions.</p><pre># Convert 1-dimensional class arrays to 10-dimensional class matrices(making the data categorical)<br>Y_train = np_utils.to_categorical(y_train, 10)<br>Y_test = np_utils.to_categorical(y_test, 10)</pre><p>In the next code cell, we will define and create a very simple neural network. This may seem complex, and weird, but believe us, this is the simplest of architectures you would ever write. Let’s write the code and then try to understand it briefly.</p><pre>from keras.layers import Activation, MaxPooling2D<br>​<br>model = Sequential() <br>model.add(Convolution2D(32, 3, 3, activation=&#39;relu&#39;, input_shape=(28,28,1))) #receptive field=3,input channel=1<br>model.add(Convolution2D(64, 3, 3, activation=&#39;relu&#39;)) #receptive field=5,input channel=32<br>model.add(Convolution2D(128, 3, 3, activation=&#39;relu&#39;)) #receptive field=7,input channel=64<br>​<br>model.add(MaxPooling2D(pool_size=(2, 2))) #receptive field=14<br>​<br>model.add(Convolution2D(256, 3, 3, activation=&#39;relu&#39;)) #receptive field=16,input channel=128<br>model.add(Convolution2D(512, 3, 3, activation=&#39;relu&#39;)) #receptive field=18,input channel=256<br>model.add(Convolution2D(1024, 3, 3, activation=&#39;relu&#39;)) #receptive field=20,input channel=512<br>model.add(Convolution2D(2048, 3, 3, activation=&#39;relu&#39;)) #receptive field=22,input channel=1024<br>model.add(Convolution2D(10, 3, 3)) #receptive field=24,input channel=2048<br>​<br>model.add(Flatten())<br>model.add(Activation(&#39;softmax&#39;))<br>​<br>model.summary()</pre><p>In the 2nd line, we are declaring a neural network and making it <em>Sequential</em>. Sequential simply means that the layers are connected to one another in a 1:1 ratio, which means, each layer takes input from the previous layer and its output goes to the next immediate layer. There are no connections that may lead to skipping of layers. It follows a cascaded layer design.</p><p>Then in the next line, we create a convolutional layer by using the <em>Convolution2D()</em> function and add it to the model using <em>model.add()</em>.</p><p>In <strong>Convolution2D(32, 3, 3, activation=’relu’, input_shape=(28,28,1))</strong>:</p><ul><li>32 represents that we need 32 kernels</li><li>3,3 means that the 32 kernels should be of size 3X3, so we are performing a 3X3 convolution here</li><li>Activations are basically non-linear functions that tell the network how data is passed between layers. Without its presence, all neural networks will be a simple linear regression model. The activation function <em>ReLu</em> used here is quite simple, it sends all the positive values over the layer and discards all negative values.</li></ul><p>We repeat these layers, linking one to the next.</p><p>Towards the end, we add a <em>Flatten()</em> layer which basically converts the output of the final layer into a 1D array of 10 numbers (because there are 10 categories in our data [numbers from 0 to 9]).</p><p>When the last statement is executed, we get a nice and structured summary of the model with details of each layer, something very similar to this.</p><figure><img alt="Summary of the model created" src="https://cdn-images-1.medium.com/max/576/1*NoYzYpKa9WOwaypiorjvpA.png" /></figure><p>We just created an architecture of the model. Now let us complete the process by compiling it.</p><pre>##we are compiling our model using adam as optimizer<br>model.compile(loss=&#39;categorical_crossentropy&#39;,<br>             optimizer=&#39;adam&#39;,<br>             metrics=[&#39;accuracy&#39;])</pre><p>We have some loss metrics which will be used by our model in ‘<em>backpropagation’</em> (watch this <a href="https://youtu.be/Ilg3gGewQ5U">wonderful video</a>), so that every time our model learns (predicts) the wrong label (ground truth), we punish it by giving it a penalty. We have <em>accuracy</em> as metrics which is essentially just our performance parameter based on which we assess the model. As it so happens, after every epoch, we are monitoring and judging the performance of the model using the accuracy of its prediction. Let’s ignore optimizers for the time being since we will cover them deeply in upcoming sessions. If you are keen then please do read about them <a href="https://medium.com/analytics-vidhya/optimization-algorithms-for-deep-learning-1f1a2bd4c46b">here</a>.</p><p>With our model properly compiled, we say our prayers to the heaven and with enormous gratitude to all the old gods and new, we finally train our model.</p><pre>##Training the model in this cell<br>model.fit(X_train, Y_train, batch_size=32, nb_epoch=10, verbose=1)</pre><p>Finally, we are doing some training??</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*mZ9HFIzY3K0oAcQYY58AEQ.jpeg" /></figure><p>We train the model for 10 epochs i.e. we go through all the 60k training images 10 times over in the hope that our model will learn all the numbers from 0 to 9.</p><p>Training process should take around 15–20 mins, and it should look something like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1y2rrq3IFhaNb1yV2nE04g.png" /></figure><p>The accuracy printed at the far right of every line after each epoch is our training accuracy. These values actually play a monumental role in diagnosing the network but should not be looked upon as final accuracy.</p><p>Let&#39;s move on…take a test of our model. Let’s test it.</p><pre>##Evaluating the trained model on test data<br>score = model.evaluate(X_test, Y_test)<br>​<br>##predicting the labels of test dataset<br>y_pred = model.predict(X_test)<br>​</pre><p>This should print the score of the model. It should be somewhat around 99.2%, not all bad for the first network.</p><p><strong>Please note that we have deliberately left explanations for some functions and statements as we have not built up the base to understand this; hopefully we will cover them in near future.</strong></p><h3>Congratulations!!!</h3><p>You made it!!!!!</p><p>You are a tough cookie and you nailed it. You deserve a pat on the back.</p><p>Get some rest!!!</p><p>Hope you enjoyed it.</p><p>Interesting comment: Do check out your model’s summary and try different kernel values and let us know how many parameters you are training today.</p><p><strong>NOTE: We are starting a new telegram group to tackle all the questions and any sort of queries. You can openly discuss concepts with other participants and get more insights and this will be more helpful as we move further down the publication. [Follow this</strong> <a href="https://t.me/joinchat/Kxmt-hWgGSFHAOI_jyxVSw?source=post_page---------------------------"><strong>LINK</strong></a> <strong>to join]</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=654efb1d2d30" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hitchhikers-guide-to-deep-learning/7-introduction-to-deep-learning-with-computer-vision-mnist-handwritten-digit-recognition-654efb1d2d30">7. Introduction to Deep Learning with Computer Vision — MNIST Handwritten digit recognition…</a> was originally published in <a href="https://medium.com/hitchhikers-guide-to-deep-learning">Deep-Learning-For-Computer-Vision</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[6. Introduction to Deep Learning with Computer Vision — 3x3 is a lie! 1x1 convolutions]]></title>
            <link>https://medium.com/hitchhikers-guide-to-deep-learning/6-introduction-to-deep-learning-with-computer-vision-3x3-is-a-lie-1x1-convolutions-9edd2baf7fd5?source=rss-5fccc8e98750------2</link>
            <guid isPermaLink="false">https://medium.com/p/9edd2baf7fd5</guid>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Inside AI]]></dc:creator>
            <pubDate>Thu, 17 Oct 2019 15:50:48 GMT</pubDate>
            <atom:updated>2019-10-17T15:50:48.539Z</atom:updated>
            <content:encoded><![CDATA[<h3>6. Introduction to Deep Learning with Computer Vision — 3x3 is a lie! 1x1 convolutions</h3><p>written by <a href="https://medium.com/u/4ce50580cfab">Nilesh Singh</a> and <a href="https://medium.com/u/f1a8d777cc44">Praveen Kumar</a>.</p><p>Prerequisites: <a href="https://medium.com/hitchhikers-guide-to-deep-learning/introduction-to-computer-vision-with-deep-learning-filters-and-kernels-a65eadc9edeb">Filters &amp; Kernels</a>, <a href="https://medium.com/hitchhikers-guide-to-deep-learning/3-introduction-to-computer-vision-with-deep-learning-channels-and-feature-maps-4ae7eb932d41">Channels &amp; Features</a></p><p>So far, we’ve looked at a very specific type of convolution with a kernel size of 3, more commonly called 3X3 convolutions. We also said that this the only type of convolution that we are going to use. But what if I told you that we were indeed lying.</p><p>There is a very important kind of convolution that is under some circumstances more important than our feature extractor 3X3.</p><p>Before looking at what it is, let’s look at the problem faced with 3X3 convolutions.</p><p>Let’s say that we have an image with some arbitrary size, also let’s assume that we are doing a series of feature extractions on it using 3X3 kernels. We know that it is in the best interest of the model to increase the number of channels in each convolutional layer.</p><p>So, if we generate 32 channels in 1st convolution then our GPU will have 32 images after 1st convolution. In the second iteration if we generate 64 channels, then for each of 32 images generated in the previous step 64 new channels will be generated.</p><p>total no. of images in GPU=32X64</p><p>if in the 3rd convolution, we generate 128 channels, then no of images=32X64X128.</p><p>We have networks with 1000s of such layers, just try to imagine the number of images that GPU will have to handle. Even state of art Titan X GPUs is tiny when confronted with such a huge number.</p><p>What do we do?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rdcFhkFJy3mz8Y-cbphubA.jpeg" /></figure><p>No seriously, what do we do?</p><p>Well, we thank Google and the University of North Carolina for releasing <a href="https://arxiv.org/pdf/1409.4842.pdf">their paper</a> and introducing the world to the magic of 1X1 convolutions.</p><p>A 1X1 convolution is used to reduce the dimensionality of channels. We can pass 1000 channels to it and tell it to give 10 channels and it will do so without losing any information.</p><p>Let’s look at an animation explaining the process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*AjaTIcaz2oHFuBwTiGfL3w.gif" /></figure><p>1X1 convolutions a linear traversal of each feature maps (channels) from top to bottom and left to right. During this traversal, it takes a weighted summary of each of the channels. This weight is decided by the number of channels that we are expecting as output.</p><p>The very same thing is happening in the animation above. We initially have 10 feature maps (channels), then we tell our 1X1 convolution that we want 4 channels as it’s output. What then happens is that all those 10 feature maps are scanned 4 times and are summarized each time using weights, this is done to ensure that all 4 output channels have features that are different.</p><p>Yeah, trust me when I say, I know this confusion.</p><p>Let it sink in for a while now. It is a difficult concept, actually, counter-intuitive to grasp in one go and it requires a wee bit of practice as well.</p><p>Here are some fantastic resources to read more about it.</p><ul><li><a href="https://machinelearningmastery.com/introduction-to-1x1-convolutions-to-reduce-the-complexity-of-convolutional-neural-networks/">Machine Learning Mastery</a></li><li><a href="https://iamaaditya.github.io/2016/03/one-by-one-convolution/">Google Inception Model</a></li></ul><p>Hope you enjoyed it. See you soon again.</p><p><strong>NOTE: We are starting a new telegram group to tackle all the questions and any sort of queries. You can openly discuss concepts with other participants and get more insights and this will be more helpful as we move further down the publication. [Follow this</strong> <a href="https://t.me/joinchat/Kxmt-hWgGSFHAOI_jyxVSw?source=post_page---------------------------"><strong>LINK</strong></a> <strong>to join]</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9edd2baf7fd5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hitchhikers-guide-to-deep-learning/6-introduction-to-deep-learning-with-computer-vision-3x3-is-a-lie-1x1-convolutions-9edd2baf7fd5">6. Introduction to Deep Learning with Computer Vision — 3x3 is a lie! 1x1 convolutions</a> was originally published in <a href="https://medium.com/hitchhikers-guide-to-deep-learning">Deep-Learning-For-Computer-Vision</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5. Introduction to Deep Learning With Computer Vision — Kernels, Channels & Neural Architecture]]></title>
            <link>https://medium.com/hitchhikers-guide-to-deep-learning/5-introduction-to-deep-learning-with-computer-vision-kernels-channels-neural-architecture-41b6bc4befa7?source=rss-5fccc8e98750------2</link>
            <guid isPermaLink="false">https://medium.com/p/41b6bc4befa7</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[convolution-neural-net]]></category>
            <dc:creator><![CDATA[Inside AI]]></dc:creator>
            <pubDate>Sun, 21 Jul 2019 12:30:14 GMT</pubDate>
            <atom:updated>2019-07-21T12:30:14.443Z</atom:updated>
            <content:encoded><![CDATA[<h3>5. Introduction to Deep Learning With Computer Vision — Kernels, Channels &amp; Neural Architecture</h3><p>by <a href="https://medium.com/u/f1a8d777cc44">Praveen Kumar</a> and <a href="https://medium.com/u/4ce50580cfab">Nilesh Singh</a></p><p>Prerequisites:</p><p><a href="https://medium.com/hitchhikers-guide-to-deep-learning/introduction-to-computer-vision-with-deep-learning-filters-and-kernels-a65eadc9edeb">Filters and kernels</a></p><p><a href="https://medium.com/hitchhikers-guide-to-deep-learning/3-introduction-to-computer-vision-with-deep-learning-channels-and-feature-maps-4ae7eb932d41">Channels and feature</a></p><p><a href="https://medium.com/hitchhikers-guide-to-deep-learning/4-introduction-to-computer-vision-with-deep-learning-datasets-and-dl-libraries-7000cc858200?source=collection_home---6------0-----------------------">Datasets</a></p><p>Hello everyone, welcome back. Till now we’ve been trying to get a hang of basics and get used to some common terminologies used. This article covers the core concepts and officially marks the beginning of our journey. While building our model, we’ll be interacting directly with many entities discussed here, so they’re tangible in a sense. For the very same reason, we will first re-look previously covered topics and build this article gradually using those concepts as the base.</p><h3>Concepts from previous sessions</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AL77N3gHx6xHUnm6.gif" /></figure><p>In the above animation, what do you see? Hard to remember?</p><p>Alright, so let’s imagine that our image is divided into a matrix of 5X5, the block on right. This is the base image for our kernel (recall that we’ll only be using 3X3 kernels). Now the aim of this animation is to convince you that the white matrix, forming on left is not the kernel, but rather it is an output of the shadowed mask (kernel) moving over our original 5X5 matrix.</p><p>So, what is that white block formed?</p><p>That block is actually our output image formed via an operation called convolution (explained later), which involves kernels and the base image.</p><p>The new image (3X3) retains some of the features of the original image (5X5) because the kernel are forced to extracts only a specific set of features. What exactly did we do here? Let’s break it down into simpler steps.</p><ol><li>First, we choose a kernel, which is used to extract features from an image. How does a kernel extract features? By the magic of convolution.</li><li>Then it creates a new image with the extracted features, the size of the new image is reduced by 2 (why? explained later).</li><li>This new image now contains only those extracted features and is fed forward to other kernels waiting in line which extracts even more finer features.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*-EDKlIUCtUMi5dCG.gif" /></figure><p>Seems complicated? Well, fret not, it gets easier with time as you learn to muscle your way in.</p><p>So don’t worry. Let’s continue…</p><p>In neural architecture, we have numerous kernels and all of those kernels produce one image each, all having a different set of features. These images accumulate in our network and become an eternal source of sorrow for our computer hardware. This is the main reason why people prefer to use high-end GPU’s and High Ram Storage capacity. GPU is much faster than RAM at processing images and those images are stored in RAM before and after processing, but remember that no amount of GPU is enough. We have thousands (sometimes millions) of kernel images, hence even with the kind of hardware provided by colab, it’s quite plausible that more often than not, you’ll get similar messages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QNxSM9UPsc2IvLYq.png" /></figure><p><strong>What about the image size? and Is it generalized?</strong></p><p>Each time we perform a convolution, a new feature extracted image is produced whose size is reduced.</p><p>By how much?</p><p>It depends on the kernel. Say size of the kernel is <em>K X K</em>, and size of image used is <em>M X M</em>,</p><p>Then, size of new image after convolution will be -</p><p><em>(M-(K-1)) X (M-(K-1))</em></p><p>In our case,</p><p><em>image size= 5X5</em></p><p><em>kernel size=3X3</em></p><p>then, <em>output image size=(5-(3–1)) X (5-(3–1)) ==&gt; (5–2) X (5–2) ==&gt; </em><strong><em>3X3</em></strong></p><p><strong>We loose 2 pixels on both <em>x-axis &amp; y-axis</em> when we perform convolution with the 3x3 kernel <em>(now would be an apt time to go and look back at that first image)</em>.</strong></p><p><strong>It is important to highlight that image width and height dimensions are only changed during the normal 3x3 kernel, not the depth. Depth remains the same. What is the depth of an image? It&#39;s the number of channels an image posses. Generally, we have 3 channels (RGB). Hence in more precise terms, we say,</strong></p><p><strong>Size of image is not simply 5x5 but 5x5x3 where 3 is the number of channels. Now wait….</strong></p><p><strong>I’m confused….. if the image has channels, then our kernel must-have channels right? Exactly, our kernel also has same number of channels as our input image. So our size of the kernel is not just 3x3 but 3x3x3, where last 3 is same as the number of channels in the input image.</strong></p><h3>Wait! First, tell me what is convolution?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*_TwkD3pVs4QBV1-H.gif" /></figure><p>You’d be disappointed if you came looking for complicated code and fancy concept. In actuality, convolution is really boring. It is just a bunch of multiplication and addition. We can define it as:</p><p><strong><em>Convolution is multiplication between kernel matrix and base image matrix followed by successive element-wise addition</em></strong>.</p><p>Consider the following image</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/0*n1bd2zJoqOFzSa6b.jpg" /></figure><p>Our image size is 7x7 and kernel size is 3x3.</p><p>When our kernel is at the top left corner… it extracts features from this top left corner of the image. How it extracts? Look at the calculations on the top right of the image. Each pixel in the image is multiplied with each pixel in the kernel and the final output is obtained by adding all the multiplications.</p><p>That’s all there is to convolution.</p><p>Another example of convolution…</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*Zx-ZMLKab7VOCQTxdZ1OAw.gif" /></figure><p>We can see above that our 3x3 kernel has these values:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/191/1*600OnDWGvqSiubz74xOekA.png" /></figure><p>Whenever our kernel is stopping on a 3x3 area, we are looking at 9 multiplications and the sum of the resulting 9 multiplications being passed on to the output (green) channel as shown in the image above.</p><p>The values in the output channel can be considered as the “confidence” of finding a particular feature. Higher the value, higher the confidence, and lower (or more negative) the value, <strong>“higher”</strong> the confidence of the <strong>non-existence</strong> of the feature.</p><h3>Why do we need layers? &amp; How many?</h3><p>We say that we need kernels to extract features to be able to detect an object or classify an image. Imagine you need to detect numerical digits which looks something like</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/530/0*NckzLyNausGetv-6.png" /></figure><p>Now, if you were to detect 8 from these images, how will you do? As humans are very smart, it is easier for us to know, but for a computer system to do this, it’s very difficult. Let’s see how machines are able to detect 8. The following steps are involved.</p><ol><li>It first tries to find edges and gradients. Every digit (including 8) is made up of thousands of small edges joined together. Gradients are the intensities or colors in an image. Each intensity or color have different values between the min-max (0–255) range.</li><li>Based on these edges and gradients, it tries to find small textures. It basically gives us spatial information about the arrangement of colors or image intensity values.</li><li>Textures are then combined to form patterns. Patterns for number 8 could be arcs and curves.</li><li>From patterns, we obtain parts of objects. Parts of the object for number 8 could be 2 circles.</li><li>Parts of objects combine to form the object itself. Here it’s 8.</li></ol><p>So the process by which a computer vision system is able to detect any object or classify an image involves these 5 steps (some of them are skipped depending on images and task at hand):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*lB7wjevubAbJu7jxpEJJZA.png" /><figcaption>Steps involved in detection by a Computer Vision system</figcaption></figure><p>In the following image, we are trying to detect 4 objects namely Faces, Cars, Elephants, and Chairs.</p><p>We look that only 3 steps are shown here. Tracing the image of Faces from bottom to top, we first obtain edges and gradients of faces, then directly we form parts of objects. Here textures and patterns are not shown but they are also performed before going to the parts of the object step. And finally, we see faces as the complete object itself. The same procedure is followed for other 3 objects as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/0*B3lBEmgsuBzPeTjD.jpg" /><figcaption>Leaning of Object parts</figcaption></figure><p><strong>IMPORTANT NOTE:</strong> Not all the images need 5 steps. Some images might require a direct jump from Edges&amp;Gradients to Patterns. This is because some images are not rich in features based on the class. For example, if you are detecting CAR, you will need all those 5 steps in sequence but detecting just TYRE of a CAR would not need all those 5 steps. Hence it depends on the image or the Dataset you are working.</p><p>Whatever we have seen up to now is based on the discussion of using just one kernel. But in reality, kernels are stacked together to perform all the 5 steps. Just one kernel can not perform all the 5 steps. So, we form different layers which consist of kernels and channels, where each layer (or set of layers) are responsible for detecting those 5 steps.</p><p>Hence, In a Neural Network architecture, each of these 5 steps is carried out by a set of layers. One of the main tasks of DL researchers is to figure out how many layers does it take to carry out each of these steps. This number is dynamic and varies for each dataset.</p><p>Example of neural architecture (also called a Network or Just simply Architecture)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*D1aPMrclTJu5ukEy" /></figure><p>Here input data is our matrix of the Image. We have several convolution layers starting from conv1 to conv5. Ignore the last 3 layers. We will cover them in upcoming articles. Do not wonder about the depth of each convolution layer. We are still yet to cover all concepts of depth and working with depth in an image and kernels.</p><p>In the above image, each convolution layer detects the corresponding 5 steps which we just discussed.</p><h4>Wait….. So……if we have too many layers, we will get more features and the better our architecture is ????</h4><p>No. As already mentioned above, each image has some set of features. If an image has fewer features and you try to detect too many, it will lead to bad network accuracy &amp; Vice versa.</p><h3>So the question still remains, how many layers should use?</h3><p>This is one of the most interesting questions in computer vision. The way we define the number of layers in a network is chosen based on an algorithm called <strong>MMZ</strong>. It is a rather interesting algorithm and takes some time to master it.</p><p>So, what is <strong>MMZ</strong>?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*mnz6OVHa9278mtd8.jpg" /></figure><p>You must be thinking if we are being serious? Yes, we indeed are. <strong>MMZ is expanded as Meri MarZi.</strong></p><p>It depends on your wish and whims to choose the number of layers you want in your network. It all depends on the dataset you are working on. No one can tell you what will work best, you have to choose wisely based on your intuition and understanding of the dataset. If you feel you are working on hard images, you might require more layers &amp; if you working on simple datasets such as MNIST (The numerical digit), you require comparatively lesser number of layers. Take the last statement with a pinch of salt, it is true in most cases but will produce awful results in others.</p><p>Always remember, <strong>Your most valuable treasure is your intuition.</strong></p><h3>Alright, Time to wrap up. Usshhh !! That was intense</h3><p>That was intense but we expect you to digest a lot more than this article as we delve deeper and deeper. We will cover the receptive field, 1x1 kernel &amp; MaxPool convolutions and calculations involving the same.</p><p>To summarize,</p><ol><li>The output of a kernel is produced by the process of convolution.</li><li>The output is a new image with a reduced dimension but contains a specific set of features extracted from the original image.</li><li>A calculation involving dimension reduction based on the size of the Kernel.</li><li>Learning process of convolution &amp; understanding 5 main steps involved in object detection by a computer vision system.</li><li>Concept and intuition of the number of layers as well as the beautiful MMZ algorithm.</li></ol><p>Hope you enjoyed it. See you soon again.</p><p><strong>NOTE: We are starting a new telegram group to tackle all the questions and any sort of queries. You can openly discuss concepts with other participants and get more insights and this will be more helpful as we move further down the publication. [Follow this</strong> <a href="https://t.me/joinchat/Kxmt-hWgGSFHAOI_jyxVSw?source=post_page---------------------------"><strong>LINK</strong></a> <strong>to join]</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=41b6bc4befa7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hitchhikers-guide-to-deep-learning/5-introduction-to-deep-learning-with-computer-vision-kernels-channels-neural-architecture-41b6bc4befa7">5. Introduction to Deep Learning With Computer Vision — Kernels, Channels &amp; Neural Architecture</a> was originally published in <a href="https://medium.com/hitchhikers-guide-to-deep-learning">Deep-Learning-For-Computer-Vision</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>