<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Antoine Labatie on Medium]]></title>
        <description><![CDATA[Stories by Antoine Labatie on Medium]]></description>
        <link>https://medium.com/@antoinelabatie?source=rss-297373e290f6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*zJ1YqNYdKXjDkbNFcRueDg.jpeg</url>
            <title>Stories by Antoine Labatie on Medium</title>
            <link>https://medium.com/@antoinelabatie?source=rss-297373e290f6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 03:52:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@antoinelabatie/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Removing Batch Dependence in CNNs by Proxy-Normalizing Activations]]></title>
            <link>https://medium.com/data-science/removing-batch-dependence-in-cnns-by-proxy-normalising-activations-bf4824eb0ba4?source=rss-297373e290f6------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf4824eb0ba4</guid>
            <category><![CDATA[normalization]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[thoughts-and-theory]]></category>
            <category><![CDATA[batch-normalization]]></category>
            <category><![CDATA[convolutional-neural-net]]></category>
            <dc:creator><![CDATA[Antoine Labatie]]></dc:creator>
            <pubDate>Thu, 24 Jun 2021 12:49:00 GMT</pubDate>
            <atom:updated>2022-05-09T11:30:21.713Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/thoughts-and-theory">Thoughts and Theory</a></h4><h4>Memory-efficient convolutional neural network training</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8zLEJD_fvWpatQfdx0DvDA.jpeg" /></figure><p>This post aims at presenting the novel technique “Proxy Norm” that we introduced in our paper “<a href="https://arxiv.org/abs/2106.03743">Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence</a>” published at NeurIPS 2021.</p><p>Proxy Norm paves the way for more memory-efficient training of convolutional neural networks. It retains the benefits of batch normalisation while removing the complication of batch dependence that previously led to inefficient execution. Proxy Norm could help AI engineers to ensure efficiency of execution in future as machine learning model sizes continue to grow and datasets keep getting larger.</p><h3>Challenges in Normalisation</h3><p>Normalisation is critical for successfully scaling neural networks to large and deep models. While the scope of normalisation was originally limited to input processing [1], it was taken to another level with the introduction of the technique Batch Normalisation [2] that maintains intermediate activations normalised throughout the network.</p><p>The specific normalisation that Batch Norm imposes is a <strong>channel-wise normalisation</strong>. This means concretely that Batch Norm normalises intermediate activations by subtracting channel-wise means and dividing by channel-wise standard deviations. Notably, Batch Norm achieves this channel-wise normalisation without altering the neural network’s expressivity. This means that the expressivity of batch-normalised networks is the same as the expressivity of unnormalised networks. We will see that these two properties, namely the <strong>channel-wise normalisation</strong> and the <strong>preservation of expressivity</strong>, are both beneficial.</p><p>Batch Norm comes, however, with an equally significant complication: <strong>batch dependence</strong>. Because the channel-wise means and variances over the whole dataset are not easily computable, Batch Norm approximates these statistics by considering the current mini-batch as a proxy for the whole dataset (cf Figure 1). Given the use of mini-batch statistics in Batch Norm’s computation, the output associated by the neural network to a given input depends not only on this input but also on all the other inputs in the mini-batch. In other words, the approximation of full-batch statistics with mini-batch statistics introduces a <strong>batch dependence</strong> in the neural network’s computation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BSCfR3IKXhL56lmeMCaDDQ.png" /><figcaption>Figure 1: Each subplot corresponds to a specific technique of normalisation characterised by its groups of components sharing the same normalisation statistics. In each subplot, the tensor of intermediate activations is constituted of the batch axis B, the channel axis C, and the spatial axes (H, W). Image by author, adapted from [3].</figcaption></figure><p>Let’s now look in more detail at Batch Norm to understand why the channel-wise normalisation and the preservation of expressivity are beneficial, and why batch dependence is a complication. Following this, we will then introduce the novel technique Proxy Norm, as presented in our NeurIPS 2021 paper “<a href="https://arxiv.org/abs/2106.03743">Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence</a>”.</p><h3>First benefit of Batch Norm: channel-wise normalisation</h3><p>As mentioned above, Batch Norm maintains the intermediate activations channel-wise normalised “close” to the nonlinearity, at each layer. This channel-wise normalisation has two benefits:</p><ol><li>Since the nonlinearity “acts” on channel-wise distributions close to normalised, it can be effectively nonlinear with respect to these channel-wise distributions. Each layer then adds expressive power and the neural network effectively uses its whole depth.</li><li>Since different channels have comparable variance, channels are well-balanced and the neural network effectively uses its whole width.</li></ol><p><strong>In short, channel-wise normalisation allows the neural network to effectively use its whole capacity.</strong> However, this benefit is not retained with the prototypical batch-independent alternatives to Batch Norm (see Figure 1). In fact, while channel-wise normalisation is retained with Instance Norm [4], it is not retained with Layer Norm [5] or Group Norm [6]. In the top left plot of Figure 2, this translates into non-negligible channel-wise squared means with Layer Norm and Group Norm.</p><h3>Second benefit of Batch Norm: preservation of expressivity</h3><p>As previously explained, the channel-wise normalisation with Batch Norm does not come at the cost of an alteration of the neural network’s expressivity. This means that any unnormalised network can be equivalently expressed as a batch-normalised network (in the full-batch setting) if Batch Norm’s scale and shift parameters are chosen appropriately. Conversely, any batch-normalised network (in the full-batch setting) can be equivalently expressed as an unnormalised network if the convolutional weights and biases are chosen appropriately. <strong>In short, Batch Norm amounts to a plain reparameterisation of the neural network’s space of solutions.</strong></p><p>This preservation of expressivity is the second benefit of Batch Norm. To appreciate why such preservation of expressivity is beneficial, it is useful to understand why alterations of the expressivity with batch-independent alternatives to Batch Norm are detrimental. In the case of Group Norm, a symptom of the alteration of expressivity is the lack of variance of the instance’s mean and standard deviation, as can be seen in the two rightmost subplots of Figure 2. This lack of variance of instance statistics is detrimental for learning as it tends to be incompatible with the expression of high-level concepts in deep layers of the neural network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EohJJO9kMllxdV9aE2ZqBw.gif" /><figcaption>Figure 2: Squared mean (left) and variance (right) of instance means (top) and instance standard deviations (bottom) during the training of ResNet-50 with various norms on ImageNet. Instance statistics are computed after the normalisation at different depths of layer (x-axis). Image by author.</figcaption></figure><h3>Batch Norm’s complication: batch dependence</h3><p>The main symptom of Batch Norm’s batch dependence is the presence of a noise stemming from the random choice of the different inputs in each mini-batch. This noise is propagated in between Batch Norm layers and “fuelled” at every Batch Norm layer when full-batch statistics are approximated with mini-batch statistics. <strong>The noise is therefore stronger when the mini-batch is smaller.</strong> This phenomenon translates into a specific regularisation of Batch Norm [7], whose strength depends on the noise amplitude and thus on the mini-batch size.</p><p>Unfortunately, this regularisation cannot be easily controlled. When the goal is to decrease the strength of this regularisation, the only way to achieve this is to increase the mini-batch size. Depending on the task and the strength of regularisation needed, Batch Norm then imposes a lower bound on the mini-batch size compatible with optimal performance. When the “compute” mini-batch size is below this lower bound, maintaining optimal performance requires an “expensive” synchronisation of statistics across multiple workers to yield a “normalisation” mini-batch larger than the “compute” mini-batch [8]. <strong>The main issue that can result from batch dependence is therefore an inefficiency of execution.</strong></p><p>Resolving this issue would make a real difference when using Graphcore’s IPU, as the extra acceleration and energy savings that the IPU provides are obtained in exchange for tighter memory constraints. Even with alternative accelerators relying less on local memory, this issue could become critical in the future. As datasets get larger and larger, we indeed expect that stronger memory constraints will be imposed by the use of larger and larger models. The use of larger datasets for a given model size also implies that less regularisation will be required, resulting in the requirement of larger and larger “normalisation” mini-batch to guarantee optimal performance when using Batch Norm.</p><h3>Proxy Norm: retaining Batch Norm’s benefits, while removing batch dependence</h3><p>So how can we retain Batch Norm’s benefits while removing batch dependence?</p><p>The two benefits of Batch Norm (i.e. the channel-wise normalisation and the preservation of expressivity) cannot be simultaneously retained with the prototypical batch-independent norms. On the one hand, Layer Norm does a good job at preserving expressivity, but only at the cost of channel-wise denormalisation. On the other hand, Instance Norm guarantees channel-wise normalisation, but only at the cost of a strong alteration of the expressivity. While Group Norm provides a better trade-off between Layer Norm’s issue and Instance Norm’s issue, it still cannot square the circle. In short, all the prototypical batch-independent norms lead to a performance degradation.</p><p>To resolve this, we need a batch-independent norm that can avoid channel-wise denormalisation while simultaneously preserving expressivity. We can make these two requirements more precise by noting that:</p><ol><li>The main operations responsible for channel-wise denormalisation are: (i) the learnable affine transforms; (ii) the activation functions.</li><li>The expressivity is preserved when inserting affine operations into the neural network.</li></ol><p>These observations have guided our design of the novel technique Proxy Norm. Proxy Norm assimilates the output of the normalisation operation to a Gaussian “proxy” variable that is assumed close to channel-wise normalised. This Gaussian proxy is fed into the same two operations as the real activations, namely the same learnable affine transform and the same activation function. After these two operations, the mean and variance of the proxy are finally used to normalise the real activations themselves. This is depicted in Figure 3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8JjZoPDJy4yHSXO2iytdpw.png" /><figcaption>Figure 3: Proxy Norm is incorporated into the neural network by adding the “cheap” red operations on top of the pre-existing black operations. Image by author.</figcaption></figure><p>Proxy Norm enables to preserve expressivity while compensating for the two main sources of channel-wise denormalisation, namely the learnable affine transforms and the activation functions. This leads us to adopt a batch-independent normalisation approach that <strong>combines Proxy Norm with either Layer Norm or Group Norm with a small number of groups</strong>. As can be seen in Figure 2, this batch-independent approach preserves channel-wise normalisation while minimising any alteration of expressivity. With this approach, we retain Batch Norm’s benefits while removing batch dependence.</p><p>The next question is whether this approach also leads to good performance in practice. Comparing Batch Norm to batch-independent approaches requires specific care to be taken to properly account for the added regularisation stemming from batch dependence with Batch Norm. To “subtract away” this regularisation effect, we therefore include extra regularisation in each of our experiments.</p><p>As can be seen in Figure 4, when that specific care is taken, the ImageNet performance of our batch-independent approach consistently matches or exceeds that of Batch Norm across various model types and sizes (note that our variant of EfficientNet is introduced in another <a href="https://arxiv.org/abs/2106.03640">paper</a>). This means that our batch-independent approach matches Batch Norm not only in behaviour, but also in performance!</p><p><strong>We see as a side product of our analysis that while an efficient normalisation is necessary for good ImageNet performance, it must also come with appropriate regularisation.</strong> On even larger datasets, we expect that an efficient normalisation will be sufficient in itself, given that less regularisation will be required [9,10].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MwGRTZoohU04a7OzKIz2gQ.png" /><figcaption>Figure 4: ImageNet performance of various model types and sizes with Batch Norm, Group Norm, and Group Norm + Proxy Norm. Image by author.</figcaption></figure><h3>Conclusion</h3><p>In this work, we have delved into the inner workings of normalisation in convolutional neural networks. We have gathered both theoretical and experimental evidence that an efficient normalisation should: (i) maintain channel-wise normalisation and (ii) preserve expressivity. While Batch Norm maintains these two properties, it also introduces the complication of batch dependence.</p><p>When investigating the prototypical batch-independent alternatives to Batch Norm, we have found that the channel-wise normalisation and the preservation of expressivity are difficult to obtain simultaneously. This has led us to design the novel technique Proxy Norm, which preserves expressivity while maintaining channel-wise normalisation. We have then adopted a batch-independent normalisation approach that combines Proxy Norm with either Layer Norm or Group Norm with a small number of groups. <strong>We have found that such an approach consistently matches Batch Norm in both behaviour and performance, while maintaining batch independence at all times.</strong></p><p>Our approach paves the way for more memory-efficient training of convolutional neural networks. This memory efficiency gives an edge to accelerators such as Graphcore’s IPU that leverage local memory to boost the efficiency of execution. In the long term, we expect that this memory efficiency will be critical even on alternative hardware.</p><p><a href="https://arxiv.org/abs/2106.03743">Read the paper</a></p><h3>Thank you</h3><p>Thank you to <a href="https://medium.com/@dominic.masters.gc/about">Dominic Masters</a>, Zach Eaton-Rosen and Carlo Luschi who also contributed to this work, and thank you to our other colleagues for the insightful discussions.</p><h3>References</h3><p>[1] Y. LeCun, L. Bottou, G. B. Orr and K.-R. Müller, <a href="http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf">Efficient BackProp</a> (1998), Neural Networks: Tricks of the Trade</p><p>[2] S. Ioffe and C. Szegedy, <a href="http://proceedings.mlr.press/v37/ioffe15.pdf">Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift</a> (2015), ICML 2015</p><p>[3] N. Dimitriou and O. Arandjelovic, <a href="https://arxiv.org/pdf/2007.08554.pdf">A New Look at Ghost Normalization</a> (2020), arXiv 2020</p><p>[4] D. Ulyanov, A. Vedaldi and V. S. Lempitsky, <a href="https://arxiv.org/pdf/1607.08022v2.pdf">Instance Normalization: The Missing Ingredient for Fast Stylization</a> (2016), arXiv 2016</p><p>[5] L. J. Ba, J. R. Kiros and G. E. Hinton, <a href="https://arxiv.org/pdf/1607.06450.pdf">Layer Normalization</a> (2016), arXiv 2016</p><p>[6] Y. Wu and K. He, <a href="https://arxiv.org/pdf/1803.08494.pdf">Group Normalization</a> (2018), ECCV 2018</p><p>[7] P. Luo, X. Wang, W. Shao and Z. Peng, <a href="https://openreview.net/pdf?id=HJlLKjR9FQ">Towards Understanding Regularization in Batch Normalization</a> (2019), ICLR 2019</p><p>[8] C. Ying, S. Kumar, D. Chen, T. Wang and Y. Cheng, <a href="https://arxiv.org/pdf/1811.06992.pdf">Image Classification at Supercomputer Scale</a> (2018), NeurIPS 2018 Workshop on Systems for ML and Open Source Software</p><p>[9] A. Kolesnikov, L. Beyer, X. Zhai, J. Puigcerver, J. Yung, S. Gelly and N. Houlsby, <a href="https://arxiv.org/pdf/1912.11370.pdf">Big Transfer (BiT): General Visual Representation Learning</a> (2020), ECCV 2020</p><p>[10] A. Brock, S. De, S. L. Smith and K. Simonyan, <a href="https://arxiv.org/pdf/2102.06171.pdf">High-Performance Large-Scale Image Recognition Without Normalization</a> (2021), ICML 2021</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf4824eb0ba4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/removing-batch-dependence-in-cnns-by-proxy-normalising-activations-bf4824eb0ba4">Removing Batch Dependence in CNNs by Proxy-Normalizing Activations</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Serendipitous Effectiveness of Weight Decay in Deep Learning]]></title>
            <link>https://medium.com/data-science/the-serendipitous-effectiveness-of-weight-decay-in-deep-learning-8b6b4234a3f9?source=rss-297373e290f6------2</link>
            <guid isPermaLink="false">https://medium.com/p/8b6b4234a3f9</guid>
            <category><![CDATA[making-sense-of-big-data]]></category>
            <category><![CDATA[l2-regularization]]></category>
            <category><![CDATA[weight-decay]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[editors-pick]]></category>
            <dc:creator><![CDATA[Antoine Labatie]]></dc:creator>
            <pubDate>Wed, 30 Dec 2020 23:06:06 GMT</pubDate>
            <atom:updated>2021-01-06T20:40:00.057Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/making-sense-of-big-data">Making Sense of Big Data</a></h4><h4>Coming full circle after unintended changes of effects</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yg7lLOo6SW-L6nrfmQynOQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@joshcala?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Josh Calabrese</a> on <a href="https://unsplash.com/s/photos/spiral-tennis-ball?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Weight decay is without question an integral part of the deep learning toolkit. It might seem like the prototypical example of a simple idea whose effectiveness endured time. If we look closely, however, comes a big surprise: weight decay’s effectiveness in deep learning owes more to <strong>serendipity </strong>than to its original rationale.</p><h3>Original rationale of weight decay</h3><p>In machine learning, the weight decay term λ∥<strong>w</strong>∥², with strength given by hyperparameter λ&gt;0, can be<strong> </strong>added to the loss function <em>L</em>(<strong>w</strong>)<strong><em> </em></strong>to penalize the norm of model parameters <strong>w.</strong> The minimization of <em>L</em>(<strong>w</strong>) is then replaced by the minimization of <em>L</em>(<strong>w</strong>) + λ∥<strong>w</strong>∥².</p><p>Adding such weight decay can be seen as imposing a <strong>soft constraint</strong> on the norm of <strong>w</strong>.<strong> </strong>This soft constraint reduces the model’s effective capacity (i.e. its representation power), and with it, its tendency to overfit.</p><p>To get a better intuition, let’s consider the hypothetical case where the regularization strength becomes infinite λ → ∞. In that case, the minimization of <em>L</em>(<strong>w</strong>) + λ∥<strong>w</strong>∥² becomes dominated by the term λ∥<strong>w</strong>∥² and the solution <strong>w</strong>=0 becomes independent of the training data. Without adaptation to the training data, no overfitting!</p><p>Historically, this idea of penalizing the norm of model parameters as a regularizer emerged in many different contexts independently. It can be traced back at least to the work of Tikhonov in 1943 in the context of inverse problems, hence its alternate naming of <strong>“Tikhonov regularization”</strong>. The idea didn’t leave the front of the stage thereafter, being notably at the core of support vector machines that dominated the machine learning landscape before the advent of deep learning.</p><p>On a theoretical level, it can be proven that any class of models with convex and Lipschitz loss function <em>L</em>(<strong>w</strong>) on a bounded domain is “agnostic PAC-learnable” through the minimization of <em>L</em>(<strong>w</strong>) + λ∥<strong>w</strong>∥² [1]. In simple terms, this means that the minimizer of <em>L</em>(<strong>w</strong>) + λ∥<strong>w</strong>∥² only has a very small probability of overfitting given enough data. Crucially, the number of data points required not to overfit is <strong>independent of the number of parameters</strong>. This property is very appealing in the context of deep learning with models constituted of millions or even trillions of parameters.</p><p>So what’s the catch? As we said, one of the hypotheses underlying this rationale is that <em>L</em>(<strong>w</strong>) is Lipschitz. And, as we shall see shortly, this hypothesis is <strong>fundamentally wrong</strong> for deep models with normalization layers. So the original rationale of weight decay becomes essentially null in modern state-of-the-art models that — nearly always — incorporate normalization layers, e.g. with batch normalization or group normalization in Convolutional Neural Networks or layer normalization in Transformers.</p><h3>Weight decay in deep learning</h3><h4>Normalization → scale invariance</h4><p>Normalization plays a crucial role in the successful “scaling” of neural networks to very large and deep models. Its primary purpose is to ensure that, no matter how wide and deep a neural network is, the signal propagated through it remains normalized.</p><p>For this purpose to be achieved, each normalization layer transforms its input <strong>x </strong>to produce the output Norm(<strong>x</strong>)=(<strong>x</strong><em>-</em>μ(<strong>x</strong>))/σ(<strong>x</strong>), with μ(<strong>x</strong>), σ(<strong>x</strong>) the mean and standard deviation of <strong>x </strong>over some subsets of its components. The inclusion of this simple operation in a neural network has profound impacts, one of which being the <strong>emergence of the</strong> <strong>scale invariance property.</strong></p><p>To see how this property emerges, let’s assume — as is typical — that <strong>x</strong> itself is the result of a convolution with weights <strong>w</strong>. If <strong>w </strong>is scaled by some positive constant ρ&gt;0, then so is the convolution output, now equal to ρ<strong>x</strong>.<strong> </strong>In turn, the statistics of ρ<strong>x</strong> are scaled by ρ compared to the statistics of <strong>x</strong>, and Norm(ρ<strong>x</strong>) remains equal to (<strong>x</strong><em>-</em>μ(<strong>x</strong>))/σ(<strong>x</strong>).</p><p>So when normalization is present, the output of the neural network with weights <strong>w </strong>is always equal to the output of the neural network with weights ρ<strong>w</strong>. This means that the loss of both networks are equal: <em>L</em>(<strong>w</strong>)= <em>L</em>(ρ<strong>w</strong>) and that <em>L</em>(<strong>w</strong>) is <strong>“scale-invariant”.</strong></p><p>A profound implication of this property is that <em>L</em>(<strong>w</strong>) is fully determined by its values on the unit sphere ∥<strong>w</strong>∥=1. Indeed,<strong> </strong>the value of <em>L</em>(<strong>w</strong>) for any <strong>w</strong> can be recovered from the value of its projection on the unit sphere <strong>w</strong>/∥<strong>w</strong>∥, given that <em>L</em>(<strong>w</strong>)=<em>L</em>(<strong>w</strong>/∥<strong>w</strong>∥) by scale invariance.</p><p>From this follow two implications on the gradient ∇<em>L</em>(<strong>w</strong>) of <em>L</em>(<strong>w</strong>):</p><ul><li>∇<em>L</em>(<strong>w</strong>) <strong>is orthogonal to w </strong>since <em>L</em>(<strong>w</strong>)<strong> </strong>does not change along the direction of <strong>w </strong>used for the projection on the unit sphere</li><li>∇<em>L</em>(<strong>w</strong>) <strong>scales as </strong>1/∥<strong>w</strong>∥ since the rate of variation of the loss is multiplied by 1/∥<strong>w</strong>∥ at point <strong>w </strong>compared to point <strong>w</strong>/∥<strong>w</strong>∥ (which explains why scale-invariant functions cannot be Lipschitz!)</li></ul><h4>Weight decay + scale invariance → increased effective learning rate</h4><p>In the remainder of this post, we focus on the context widely encountered in deep learning of stochastic gradient descent (SGD). In this context, we denote <strong>w</strong>_t the model parameters at iteration t, and ∇<em>L</em>(<strong>w</strong>_t|<strong>B</strong>_t), ∇<em>L</em>(<strong>w</strong>_t|<strong>D</strong>) the gradients at point <strong>w</strong>_t computed respectively on the mini-batch <strong>B</strong>_t and the whole dataset <strong>D</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ub67e7AECeA552_LjYN_ow.png" /><figcaption>Fig. 1: Weight equilibrium of SGD with learning rate <em>η </em>and weight decay strength λ</figcaption></figure><p>As can be seen on Fig. 1, SGD without weight decay would solely follow the direction of the mini-batch gradient ∇<em>L</em>(<strong>w</strong>_t|<strong>B</strong>_t), that — by orthogonality to <strong>w</strong>_t — has a small centrifugal component with respect to the interpolation of model parameters between t and t+1. As a consequence, <strong>w</strong>_t would “spiral out” to ∥<strong>w</strong>_t∥→ ∞, causing gradients to vanish: ∥∇<em>L</em>(<strong>w</strong>_t|<strong>B</strong>_t)∥∝1/∥<strong>w</strong>_t∥→ 0, and the optimization to slow down.</p><p>When weight decay is added, on the other hand, its centripetal force compensates the centrifugal force of gradients such that an <strong>equilibrium </strong>with approximately constant ∥<strong>w</strong>_t∥ can be reached. This equilibrium is stable due to the opposite dependence on ∥<strong>w</strong>_t∥ of the weight decay and gradient forces.</p><p>When this equilibrium is reached, it can be shown that the update during iteration t of the projection <strong>w</strong>_t/∥<strong>w</strong>_t∥ on the unit sphere is approximately equal to -<em>η_e</em>∇<em>L</em>(<strong>w</strong>_t/∥<strong>w</strong>_t∥|<strong>B</strong>_t), with the <strong>“effective learning rate” </strong><em>η_e</em> equal to <em>η/</em>∥<strong>w</strong>_t∥² [2]. This means that the update during iteration t of <strong>w</strong>_t/∥<strong>w</strong>_t∥ is equivalent to an update of SGD with learning rate <em>η</em>_<em>e </em>at point <strong>w</strong>_t/∥<strong>w</strong>_t∥.</p><p>Furthermore, the effective learning rate <em>η_e </em><strong>can be shown to scale as the square root of </strong><em>η</em>λ, due to the fact that ∥<strong>w</strong>_t∥² at equilibrium scales as the square root of <em>η</em>/λ (cf [3] and Fig. 2). The primary effect of weight decay in deep learning is therefore not to reduce the model capacity, but to <strong>increase the effective learning rate!</strong></p><p>Let’s now see why this unintended change of effects leads in the end still to regularization, enabling <strong>weight decay to go full circle</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cr3kzy0cDo384uGizi7F_Q.png" /><figcaption>Fig. 2: For various choices of (<em>η,</em>λ), we plot ∥<strong>w</strong>_t∥² and the ratio of ∥<strong>w</strong>_t∥² by the square root of <em>η/</em>λ during the first 100 epochs of training of ResNet-8 on CIFAR-10, with <strong>w</strong>_t the concatenation of all 3x3 conv kernels in the network</figcaption></figure><h4>Increased effective learning rate → regularization</h4><p>Let’s start by considering an increase of learning rate in the case of networks with neither scale invariance nor weight decay.</p><p>In that case, SGD “descends” at each iteration the mini-batch gradient by making a step -<em>η</em>∇<em>L</em>(<strong>w</strong>_t|<strong>B</strong>_t). Since the mini-batch gradient ∇<em>L</em>(<strong>w</strong>_t|<strong>B</strong>_t) is only an approximation of the dataset gradient ∇<em>L</em>(<strong>w</strong>_t|<strong>D</strong>), this results in the introduction of a<strong> stochastic noise</strong> with zero expectation but nonzero variance in the trajectory (cf Fig. 3).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-Ri6xtzosNE8fekM3svG6A.gif" /><figcaption>Fig. 3: Noise in the trajectory of SGD with learning rate <em>η</em></figcaption></figure><p>To understand the effect of an increase of learning rate, let’s now contrast the cases: (i) where we perform two iterations with learning rate <em>η</em>, and (ii) where we perform one iteration with learning rate 2<em>η. </em>While the expected length of trajectory is roughly the same in both cases, the <strong>stochastic noise is increased</strong> in case (ii) compared to case (i). Indeed, the noise variance scales as 4<em>η</em>² when performing one iteration with learning rate 2<em>η</em>, while it scales as 2<em>η</em>² — by independence of each iteration — when performing two iterations with learning rate <em>η.</em></p><p>Thus, an increase of the learning rate leads to an increased stochasticity of SGD per unit length of trajectory. In turn, this increased stochasticity favors the convergence to “flat” regions of low loss, that are less prone than sharper regions to overfitting [4]. <strong>An increase of the learning rate is therefore a form of regularization.</strong></p><p>How does that translate to networks with scale invariance and weight decay? Recall that the update during iteration t of <strong>w</strong>_t/∥<strong>w</strong>_t∥ is equivalent to an update of SGD with learning rate <em>η</em>_<em>e </em>at point <strong>w</strong>_t/∥<strong>w</strong>_t∥. Consequently, an increase of the effective learning rate <em>η</em>_<em>e — </em>obtained in particular when increasing the weight decay strength λ — leads to an increased stochasticity of the trajectory of <strong>w</strong>_t/∥<strong>w</strong>_t∥. In turn, this favors the convergence of <strong>w</strong>_t/∥<strong>w</strong>_t∥ to “flatter” regions of low loss, that are less prone overfitting.<strong> An increase of the weight decay strength </strong>λ<strong> is therefore still a form of regularization.</strong></p><h3>Conclusion</h3><p>To sum up, weight decay’s primary effect has changed in deep learning compared to classical machine learning. <strong>It is not anymore to limit the model capacity</strong>, <strong>but rather to</strong> <strong>increase the effective learning rate</strong>. Since an increase of the effective learning rate still leads to finding solutions less prone to overfitting, <strong>weight decay has serendipitously remained a form of regularization</strong>.</p><h3>Disclaimer</h3><p>For the sake of clarity and without altering the analysis, a few complexities have been ignored in the main text.</p><ul><li><strong>Convex-Lipschitzness of </strong><em>L</em>(<strong>w</strong>) <strong>and agnostic PAC learnability</strong>. Technically, for agnostic PAC learnability, the convexity and Lipschitzness of <em>L</em>(<strong>w</strong>) is required for any dataset distribution.</li><li><strong>Scale and shift parameters of the norm. </strong>Normalization is usually followed by an affine transform with parameters β, γ, such that it produces the output: γ(<strong>x</strong><em>-</em>μ(<strong>x</strong>))/σ(<strong>x</strong>)+β. This does not alter our argument on the emergence of the scale invariance property.</li><li><strong>Parameters not subject to scale invariance. </strong>Typically, some parameters in a neural network are not subject to scale invariance (e.g. the final fully-connected layer). Our argument on the scale of equilibrium still applies for all the parameters subject to scale invariance.</li><li><strong>Flatness and generalization. </strong>To guarantee generalization, flatness is not sufficient and an additional constraint on the norm of model parameters is required [4]. In networks with scale invariance, such constraint is present for <strong>w</strong>_t/∥<strong>w</strong>_t∥ since the norm of <strong>w</strong>_t/∥<strong>w</strong>_t∥ remains equal to 1.</li></ul><h3>References</h3><p>[1] Shai Shalev-Shwartz, Shai Ben-David, <a href="https://www.cambridge.org/core/books/understanding-machine-learning/3059695661405D25673058E43C8BE2A6">Understanding Machine Learning: From Theory to Algorithms</a> (2014), Cambridge University Press</p><p>[2] Elad Hoffer, Ron Banner, Itay Golan, Daniel Soudry, <a href="http://papers.neurips.cc/paper/7485-norm-matters-efficient-and-accurate-normalization-schemes-in-deep-networks">Norm Matters: Efficient and Accurate Normalization Schemes in Deep Networks</a>, NIPS 2018</p><p>[3] Twan van Laarhoven, <a href="https://arxiv.org/abs/1706.05350">L2 Regularization versus Batch and Weight Normalization</a>, arXiv 2017</p><p>[4] Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, Nathan Srebro, <a href="https://papers.nips.cc/paper/2017/hash/10ce03a1ed01077e3e289f3e53c72813-Abstract.html">Exploring Generalization in Deep Learning</a>, NIPS 2017</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8b6b4234a3f9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/the-serendipitous-effectiveness-of-weight-decay-in-deep-learning-8b6b4234a3f9">The Serendipitous Effectiveness of Weight Decay in Deep Learning</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[It’s Necessary to Combine Batch Norm and Skip Connections]]></title>
            <link>https://medium.com/data-science/its-necessary-to-combine-batch-norm-and-skip-connections-e92210ca04da?source=rss-297373e290f6------2</link>
            <guid isPermaLink="false">https://medium.com/p/e92210ca04da</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Antoine Labatie]]></dc:creator>
            <pubDate>Sat, 31 Aug 2019 16:06:36 GMT</pubDate>
            <atom:updated>2021-06-23T11:06:04.310Z</atom:updated>
            <content:encoded><![CDATA[<h3>It Is Necessary to Combine Batch Normalization and Skip Connections</h3><h4>These techniques must go hand in hand</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uCBnIvJrePuVBie4p4Qe5Q.jpeg" /><figcaption>Image by geralt from pixabay</figcaption></figure><p>This post is based on my <a href="https://arxiv.org/abs/1811.03087">paper</a> published in ICML 2019 [1]. It attempts to provide novel insights to answer the following questions:</p><ul><li>Why do specific architecture of deep neural networks (DNNs) work and others don’t?</li><li>What’s the role played by batch norm [2] and skip connections [3]?</li></ul><p>Answering these questions is a hot topic of research with a lot at stake. Indeed such answers would advance the theoretical understanding of DNNs and potentially unlock further improvement in their design.</p><h3>How to assess the quality of an architecture?</h3><p>To get started, the first key observation is that any DNN mapping from input to output requires two elements to be specified: (1) the architecture; (2) the values of model parameters — weights and biases — inside the architecture. The set of all DNN mappings obtained by fixing the architecture and varying the model parameters inside the architecture is referred as the <strong>hypothesis class</strong>.</p><p>The purpose of the hypothesis class is to impose a constraint on training. Indeed training consists in finding a DNN mapping that simultaneously: (1) belongs to the hypothesis class; (2) agrees with the training data. This constraint that DNN mappings must belong to the hypothesis class is a way of expressing the prior knowledge that the true mapping itself belongs to the hypothesis class. It is essentially this prior knowledge that makes possible the <strong>induction </strong>of the test data using only the training data. For this reason, the prior knowledge is commonly referred as the <strong>inductive bias </strong>(for more details, I refer the reader to this great <a href="https://www.cs.huji.ac.il/~shais/UnderstandingMachineLearning/">book</a> by Shalev-Shwartz and Ben-David).</p><p>Returning to our initial goal, we may assess the quality of an architecture by assessing the quality of its inductive bias. This assessment may be performed using the following procedure: fix the architecture and randomly sample model parameters inside the architecture. If most DNN mappings sampled with this procedure have bad properties, it means that there’s an inductive bias towards bad properties, i.e. that bad properties will be favored during training. In turn, this will lead either to <strong>untrainability</strong> — i.e. underfitting — when bad properties are incompatible with low training loss, or to <strong>poor generalization </strong>— i.e. overfitting — when bad properties are compatible with low training loss but unlikely to generalize.</p><p>Before being able to apply this procedure, we still need to make more precise the notion of “bad properties” of DNN mappings.</p><h3>What are bad properties of a DNN mapping?</h3><p>Let’s consider a fixed DNN mapping, specified by <strong>fixing the architecture</strong> and <strong>fixing the model parameters </strong>inside the architecture.<em> </em>This fixed DNN mapping receives a <strong>random input </strong>and propagates this input throughout its layers. We keep track of the propagation by defining:</p><ul><li>The<strong> random signal</strong><em>:</em> <strong>y</strong>^<em>l = </em>Φ^<em>l</em>(<strong>x</strong>), obtained by applying the fixed DNN mapping Φ^<em>l</em> up to layer <em>l </em>to the random input <strong>x</strong></li><li>The <strong>random noise</strong>: d<strong>y</strong>^<em>l = </em>Φ^<em>l</em>(<strong>x</strong>+d<strong>x</strong>) -Φ^<em>l</em>(<strong>x</strong>)<strong>, </strong>obtained as the corruption at layer <em>l</em> originating from the random corruption d<strong>x</strong> of the random input <strong>x</strong></li></ul><p>Now “bad properties” of the DNN mapping may be defined as unwanted behavior of the signal<em> </em>and noise: either with the signal <strong>y</strong>^<em>l</em> becoming non meaningful, or with the noise d<strong>y</strong>^<em>l </em>getting out of control. More precisely, two “pathologies” may be defined with such “bad properties” pushed to the extreme (we focus on these two “pathologies” since they are observed in our context, but other “pathologies” could be defined and observed in other contexts):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z71mlOLJhBcKVc_-UIDYxw.png" /><figcaption>Pathological Signal</figcaption></figure><ul><li><strong>Pathological Signal: </strong>the signal <strong>y</strong><em>^l </em>looses its directions of variance and gets concentrated along a single line in deep layers. This pathology is e.g. incompatible with the one-hot target of multiclass classification (having a number of directions of variance typically equal to the number of classes minus 1). An inductive bias towards this pathology likely leads to untrainability.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NBh1JRqaxal2VulI-ShpnA.png" /><figcaption>Pathological SNR</figcaption></figure><ul><li><strong>Pathological SNR: </strong>the noise d<strong>y</strong><em>^l </em>explodes with respect to the signal<em> </em><strong>y</strong><em>^l</em>, with the signal-to-noise ratio SNR<em>^l </em>decaying exponentially with<em> l.</em> This pathology may be compatible with low training loss, but any input corruption d<strong>x </strong>on the test set will lead to the corrupted signal<strong> y</strong><em>^l+</em>d<strong>y</strong><em>^l = </em>Φ^<em>l</em>(<strong>x</strong>+d<strong>x</strong>) becoming pure noise — i.e. meaningless. An inductive bias towards this pathology likely leads to poor generalization.</li></ul><h3>Applying our procedure for various architectures</h3><p>Now let’s apply our procedure for various architectures of <strong>convolutional</strong> <strong>DNNs </strong>—<strong> </strong>including <strong>fully-connected DNNs </strong>as special case with spatial size equal to 1 — with <strong>ReLU</strong> activation function:</p><ul><li>DNNs <strong>without</strong> batch norm and <strong>without</strong><em> </em>skip connections suffer from <strong>pathological signal</strong> — i.e. # directions of variance of <strong>y</strong><em>^l </em>close to 1 in deep layers</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aP27OheI25rZEeoLY_zsAA.png" /></figure><ul><li>DNNs <strong>without </strong>batch norm<em> </em>and <strong>with </strong>skip connections similarly suffer from <strong>pathological signal</strong> — i.e. # directions of variance of <strong>y</strong><em>^l</em> close to 1 in deep layers</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q7nKB-qlW_hFWq53J00l8Q.png" /></figure><ul><li>DNNs <strong>with</strong> batch norm and <strong>without </strong>skip connections suffer from<strong> pathological SNR </strong>—<strong> </strong>i.e. SNR<em>^l</em> / SNR<em>^</em>0<em> </em>decaying exponentially with <em>l</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_u2vGAR-632kqw1qIFjg1w.png" /></figure><ul><li>DNNs <strong>with</strong> batch norm and <strong>with</strong> skip connections do not suffer from any pathology — i.e. remain <strong>well-behaved </strong>at all depths</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qJO_YEmhHbJj1K6iRrFBpw.png" /></figure><h3>What’s going on?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HbXDEfEWwiE254lEPBg8Eg.png" /></figure><p>The main force attracting towards pathology is the <strong>multiplicativity </strong>of feedforward layer composition<strong> </strong>(Conv and ReLU layers can be seen respectively as multiplication by a random matrix and multiplication by a Bernouilli random vector):</p><ul><li>DNNs <strong>without</strong> skip connections are pathological in deep layers since they are subject to plain feedforward multiplicativity</li><li>DNNs<strong> without </strong>batch norm and <strong>with</strong> skip connections are pathological in deep layers since the roughly equal variance in residual and skip connection branches <strong>does not</strong> effectively counter feedforward multiplicativity</li><li>DNNs <strong>with </strong>batch norm and <strong>with</strong> skip connections remain well-behaved at all depths since the decaying ratio ∝ 1/(<em>l</em>+1) of signal variance between residual and skip connection branches <strong>does</strong> effectively counter feedforward multiplicativity</li></ul><h3>Conclusion</h3><p>Let’s summarize our results (to dig deeper, I refer the interested reader to the <a href="https://arxiv.org/abs/1811.03087">paper</a> and <a href="https://github.com/alabatie/moments-dnns">code</a>):</p><ul><li>The combination of batch norm and skip connections encodes a<strong> well-behaved</strong> inductive bias<strong> </strong>in deep nets</li><li>The benefits of these techniques, however, are <strong>difficult to disentangle</strong>. It is only when they are combined that — by diluting the residual branch into the skip connection branch — they counter feedforward multiplicativity</li></ul><p>I hope that these results will unlock new perspectives in the understanding of deep nets.</p><h3>Disclaimer</h3><p>For ease of exposition (and without altering the analysis), some notations have been simplified in this post compared to the paper:</p><ul><li>The pre-activation perspective has been adopted, with each layer <em>l </em>starting after the convolution and ending again after the convolution</li><li># directions of variance of <strong>y</strong><em>^l </em>corresponds in the paper to the effective rank of <strong>y</strong><em>^l</em></li><li>SNR<em>^l / </em>SNR^0 corresponds in the paper to the inverse of the squared normalized sensitivity</li></ul><h3>References</h3><p>[1] A. Labatie, <a href="https://arxiv.org/abs/1811.03087">Characterizing Well-Behaved vs. Pathological Deep Neural Networks</a> (2019), ICML 2019</p><p>[2] S. Ioffe and C. Szegedy, <a href="https://arxiv.org/abs/1502.03167">Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift</a> (2015), ICML 2015</p><p>[3] K. He, X. Zhang, S. Ren and J. Sun, <a href="https://arxiv.org/abs/1603.05027">Identity Mappings in Deep Residual Networks</a> (2016), ECCV 2016</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e92210ca04da" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/its-necessary-to-combine-batch-norm-and-skip-connections-e92210ca04da">It’s Necessary to Combine Batch Norm and Skip Connections</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>