<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dominic Masters on Medium]]></title>
        <description><![CDATA[Stories by Dominic Masters on Medium]]></description>
        <link>https://medium.com/@dominic.masters.gc?source=rss-b23ad1d05ffe------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*6rO3bS74eexpstP560bIIg.jpeg</url>
            <title>Stories by Dominic Masters on Medium</title>
            <link>https://medium.com/@dominic.masters.gc?source=rss-b23ad1d05ffe------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 04:33:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dominic.masters.gc/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Accelerating Block Sparse Matrix Multiplication with Graphcore IPU and the PopSparse library]]></title>
            <link>https://medium.com/data-science/accelerating-block-sparse-matrix-multiplication-with-graphcore-ipu-and-the-popsparse-library-3791284a2127?source=rss-b23ad1d05ffe------2</link>
            <guid isPermaLink="false">https://medium.com/p/3791284a2127</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[sparsity]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[matrix-multiplication]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Dominic Masters]]></dc:creator>
            <pubDate>Wed, 12 Apr 2023 16:38:19 GMT</pubDate>
            <atom:updated>2023-04-12T16:38:19.644Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>Faster sparse matrix multiplication with the new library PopSparse</strong></h4><p>Authors: <strong>Zhiyi Li</strong>, AI Engineer, <strong>Douglas Orr</strong>, Research Scientist, <strong>Valeriu Ohan</strong>, Software Engineer, and <a href="https://medium.com/u/b23ad1d05ffe"><strong>Dominic Masters</strong></a>, Research Scientist</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3JhWugN3CyvCippoPGS7lQ.jpeg" /><figcaption>Image by author.</figcaption></figure><p>In light of the significant growth in deep learning model size as well as the increased uptake of artificial intelligence across a broad range of applications, the incentives to reduce the computational cost of running these models have never been greater.</p><p>One method that has shown significant promise to reduce these costs is to use <em>sparsity</em>. Sparsity typically refers to the method of inducing as many zeros as possible in the model weights through a process of <em>pruning</em>, then skipping these values when doing the computation itself.</p><p>While much success has been achieved in using sparsity to reduce theoretical measures of cost like FLOPs and parameter count, achieving actual speed improvements while maintaining acceptable task performance has been much more difficult, particularly on general-purpose accelerators such as GPUs using low-precision number formats.</p><p>To help realise these theoretical benefits in practice and enable further investigation of sparse techniques on Graphcore IPUs, we introduce PopSparse, a library that enables fast sparse operations by leveraging both the unique hardware characteristics of IPUs as well as any block structure defined in the data. We target two different types of sparsity: static, where the sparsity pattern is fixed at compile-time; and dynamic, where it can change each time the model is run. We benchmarked sparse dense matrix multiplication for both modes on IPU. Results indicate that the PopSparse implementations are faster than dense matrix multiplications on IPU at a range of sparsity levels with large matrix size and block size. Furthermore, static sparsity in general outperforms dynamic sparsity. While previous work on GPUs has shown speedups only for very high sparsity (typically 99% and above), we demonstrate that Graphcore’s static sparse implementation outperforms equivalent dense calculations in FP16 at lower sparsity (around 90%).</p><p>These results were published online on <a href="https://arxiv.org/abs/2303.16999">arXiv</a> and accepted to the <a href="https://www.sparseneural.net/?">Sparse Neural Network workshop at ICLR 2023</a>.</p><h3><strong>Graphcore IPU for sparse dense matrix multiplication in deep neural network</strong></h3><p>The notion of sparsity in deep learning most commonly refers to the idea of sparsifying the model weights with the aim of reducing the associated storage and compute costs. This is typically done through a single pruning step at the end of training (Zhu &amp; Gupta, 2017) or potentially through some sparse training regime (Evci et al., 2019). These approaches have typically achieved between 90–99% reduction in model weights while maintaining and acceptable level of accuracy (Hoefler et al., 2021).</p><p>While the benefits to model size can be easily realised, increased computational efficiency can often be more difficult to achieve as the resulting unstructured sparsity patterns do not align well with the vectorised instruction sets in highly parallel deep learning accelerators (Qin et al., 2022). Therefore, one approach to improving sparse compute efficiency is to enforce some degree of structure on the sparsity pattern. This has motivated a wide range of structured pruning techniques such as neuron (Ebrahimi &amp; Klabjan, 2021), channel-wise (He et al., 2017), block (Gray et al., 2017), 2:4 (Mishra et al., 2021). However, these methods generally incur a task performance penalty compared to equivalent unstructured methods.</p><p>Here we present PopSparse, a library for the effective acceleration of sparse dense matrix multiplication (SpMM) on Graphcore IPUs. IPUs have multiple architectural features that help accelerate sparse operations:</p><ul><li>Large, high bandwidth on-chip SRAM that improves performance for communication heavy, low arithmetic efficiency operations.</li><li>Fine-grained MIMD processing model that partitions work over 1472 independent compute units (tiles) that enables high degrees of parallelism.</li><li>Ahead-of-time compilation model that allows further optimisations to be employed when sparsity structure is known at compile time.</li></ul><h3><strong>SpMM in PopSparse library on IPU</strong></h3><p>The SpMM can be written as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n-9Wa6uI7xEVWWmXv8uMnQ.png" /></figure><p>where ⊙ denotes elementwise multiplication, * for inner product, <em>Y ∈ Rᵐ </em>ˣ<em>ⁿ</em>, <em>X ∈ Rᵏ </em>ˣ<em>ⁿ</em> are dense output and input matrices respectively. The sparse weight matrix (<em>M</em>⊙<em>W</em>) is defined via <em>M ∈ Bᵐ </em>ˣ<em>ᵏ (B = {</em>0,1<em>})</em>, a mask that represents the sparsity pattern, itself derived from <em>M̂ </em>∈<em> B</em>⁽ᵐᐟᵇ⁾ˣ⁽ᵏᐟᵇ⁾, a block mask and <em>W ∈ Rᵐ </em>ˣ<em>ᵏ </em>defines weight values.</p><p>In this formulation, (<em>M</em>⊙<em>W</em>) has a block-sparse structure, where contiguous square blocks of weights of shape <em>b</em> × <em>b</em> contain all the non-zero elements, given block-size <em>b</em>. The dimensions <em>m</em> and <em>k</em> are referred to as output and input feature size and n is referred to as batch size, corresponding to their typical role in weight-sparse neural network computation.</p><p>We define density, <em>d </em>= ∑ᵢⱼMᵢⱼ/(<em>m⋅k</em>), where the standard term sparsity corresponds to <em>(</em>1 <em>— d)</em>. We use floating point operation (FLOP) count to quantify the amount of useful arithmetic work in an SpMM as: 2<em>⋅m⋅k⋅n⋅d</em>. Note that this only counts operations performed on non-zero values and does not depend on block size <em>b</em>. The PopSparse library supports unstructured (block size <em>b</em> = 1) or structured sparsity (<em>b</em> = 4, 8, 16).</p><h4>Static sparsity</h4><p>With static sparsity, during compile time, the sparsity pattern and matrix shapes are known to the partitioner, which splits the non-zero elements (specific values not known yet) of the sparse matrix across <em>k</em> dimension into <em>qₖ</em> partitions and the dense matrix across n dimension into <em>qₙ </em>partitions. Figure 1a illustrates the distribution, computation and reduction of the sparse matrix in the static sparse case. Splits over the k dimension do not have to be evenly sized and are chosen to ensure a balanced distribution of the non-zero elements (denoted with coloured small blocks in the figure). Once the model is compiled, during execution, the specific non-zero values of <em>W</em> are provided. As the sparsity pattern is known in advance, no extra exchange of data is needed. Local dot product computation with the dense matrix is then executed and there is a final reduction across tiles to give the dense output matrix.</p><h4>Dynamic sparsity</h4><p>With dynamic sparsity, the maximum number of non-zero elements for the sparse operand is fixed, while the sparsity pattern is updated during execution. To minimise compute cycles, we employ a planner during compile time to optimise the partitioning while accommodating for the full range of sparsity patterns that can be defined during execution. As the sparsity pattern could change for each run, compared to static sparsity, there is an additional step to distribute sparse matrix indices and non-zero values by the partitioner. This is demonstrated in Figure 1b. Furthermore, as fixed size partitions over m and k dimensions are required, the number of non-zero elements in each partition may not be balanced. In Figure 1b the <em>B</em> elements on the second partition of <em>k</em> dimension do not fit on Tile1. Three of the <em>B</em> elements overflow to Tile2. However, the Tile2 does not have all the partition information (slices of <em>m, k, n</em>) corresponding to B elements to compute the result. Consequently, extra steps of propagation are induced to finish the computation.</p><p>Overall, our dynamic sparse implementation has several additional costs compared to the static case:</p><ul><li>Additional control flow after compilation, which incurs some cost overhead.</li><li>We must compile for the largest inter-tile communication volume possible and thus are less efficient on average.</li><li>When data is unbalanced, additional propagation and compute phases are required to rebalance the data. The number of steps required is dependent on the degree of the imbalance.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/1*6XvoUhci5OlOLnp4Isp9Vw.png" /><figcaption>Figure 1: Illustration of static and dynamic sparse partitioning of the input feature dimension, <em>k</em>. Image by author.</figcaption></figure><h3><strong>Benchmarks</strong></h3><p>Our micro benchmarking experiments explore a single sparse-dense matmul SpMM: <em>Y=(M</em>⊙<em>W)</em>*<em>X</em> on IPU and GPU. The APIs used for IPU experiments and GPU baselines are shown in Table 1. The range of parameters that were benchmarked is detailed in Table 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*fwL1LXIa8D4MLRFs6bsmqw.png" /><figcaption>Table 1: APIs used when benchmarking IPU and GPU.</figcaption></figure><p><strong>IPU</strong>: The operation is executed a single time on a single Bow IPU in a Bow-2000 chassis, with a randomly generated sparsity pattern and values. We extract cycle count information and convert these cycle counts into TFLOP/s values given a constant clock of 1.85 GHz. Host transfers are excluded from measured time.</p><p><strong>GPU</strong>: Baseline sparse implementations on GPU are provided by the cuSPARSE library v11.6.2 (NVIDIA, 2022). These are divided according to the sparse format used, with no distinction between static and dynamic sparsity patterns. We consider compressed sparse row (CSR) and block sparse row (BSR) for unstructured and block sparsity, respectively. We generate a random sparsity pattern and values, copy to device, execute 25 iterations of the operation and copy results to host for validation. We start timing after 5 iterations and measure wall clock time using <em>cudaEventRecord</em>. All experiments were performed on a single A100 GPU running in a 4 x A100-SXM4–40G chassis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/490/1*O878Ta8MJuacobXhEYaDBQ.png" /><figcaption>Table 2: Ranges of parameters swept for benchmarks. Image by author.</figcaption></figure><p>The key results of this work are shown in Figure 2. These plots show the compute FLOP/s (in log scale) vs density with fixed feature size, with each line presenting different implementations of the same operation. By construction, the dense implementation shows linear scaling with density — since zeros are not counted in the FLOP/s calculation. While on the other hand, for a sparse implementation, perfect scaling would predict constant FLOP/s as density varies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kl9CS5RpftrCyFHF9MXVnQ.png" /><figcaption>Figure 2: IPU FP16 and GPU block-sparse matmul performance as density varies for various block size b, square feature size m = k = 4096, best over batch size n. Image by author.</figcaption></figure><p>For the dense baselines, while we broadly see chip-for-chip parity between IPU and GPU in FP16, it&#39;s important to highlight that the IPUs used are approximately <a href="https://docs.paperspace.com/gradient/machines/">half the price in the cloud</a>, highlighting their cost efficiency for these basic operations. Furthermore, for the sparse operations, we see even larger benefits from using IPUs.</p><p>Specifically, for IPUs (Figure 2a), we see that using static sparse operations and block sparsity both individually improve performance. For example, the block size 16, dynamic case shows benefits over the dense baseline across all densities considered and unstructured (b = 1) static sparsity outperforming dense in the low-density regime (&lt;5%). In addition, when both static and block sparsity methods are used, we see benefits of as much as 5x in throughput over the dense baseline.</p><p>For the GPU (Figure 2b), we see much less favourable performance with the sparse operations, with no case tested outperforming the FP16 dense baseline. However, we did find that the FP32 block sparse (BSR) case scaled very well as density decreased. It looks possible that for very low densities, this may outperform the dense baseline.</p><p>Comparing the two platforms, we can therefore see that IPUs achieve as much as a 5x throughput benefit over the dense GPU baseline and as much as a 25x throughput improvement when only sparse operations are considered. With the additional price benefits, we believe this presents a compelling case for investigating sparse applications on IPUs.</p><p>It should however be noted that even on IPU, the sparse operations only outperform the dense baselines under certain conditions. More extensive result can be found in <a href="https://arxiv.org/abs/2303.16999">our paper</a>, but as an approximate guide, sparsity typically brings speedup over the dense baseline on IPU (FP16, large batch size for both) when:</p><ul><li>Static sparsity, block size b = 1, features (m = k) ≥ 4096, density d ≤ 1/32.</li><li>Static sparsity, block size b ≥ 4: features (m = k) ≥ 4096, density d ≤ 1/8.</li><li>Dynamic sparsity, block size b ≥ 8, features (m = k) ≥ 4096, density d ≤ 1/32.</li></ul><p>Importantly, these ranges are somewhat challenging to use today in the context of weight sparse training and inference. This is primarily because finding block sparse patterns that do not degrade performance is a hard problem and that, on theoretical efficiency metrics (per non-zero FLOP), unstructured sparsity will always appear as good or better than block sparsity. We hope that this work shows that block sparsity is a promising method for achieving practical acceleration of sparse models and will motivate further investigation into effective block sparse pruning algorithms in the research community.</p><p>Come and <a href="https://console.paperspace.com/github/graphcore-research/notebooks?container=graphcore%2Fpytorch-jupyter%3A3.2.0-ubuntu-20.04&amp;machine=Free-IPU-POD4&amp;file=%2Fsparsity_benchmarks%2FSpMM.ipynb">try our PopSparse operations</a> for yourself on IPUs on Paperspace and then why not try integrating them into your own models? If you are actively pursuing research in sparsity that could benefit from IPU support, please consider signing up for our <a href="https://www.graphcore.ai/academics">Academic program</a> where we support researchers to achieve great things using IPUs.</p><h3>Try it yourself</h3><p>To try SpMM on IPU — <a href="https://console.paperspace.com/github/graphcore-research/notebooks?container=graphcore%2Fpytorch-jupyter%3A3.2.0-ubuntu-20.04&amp;machine=Free-IPU-POD4&amp;file=%2Fsparsity_benchmarks%2FSpMM.ipynb">run on Paperspace</a>.</p><h4><strong>References</strong></h4><p>[1] Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression, 2017. URL <a href="https://arxiv.org/abs/1710.01878.">https://arxiv.org/abs/1710.01878.</a></p><p>[2] Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners, 2019. URL <a href="https://arxiv.org/abs/1911.11134.">https://arxiv.org/abs/1911.11134.</a></p><p>[3] Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks, 2021. URL <a href="https://arxiv.org/abs/2102.00554">https://arxiv.org/abs/2102.00554</a>.</p><p>[4] Eric Qin, Raveesh Garg, Abhimanyu Bambhaniya, Michael Pellauer, Angshuman Parashar, Sivasankaran Rajamanickam, Cong Hao, and Tushar Krishna. Enabling flexibility for sparse tensor acceleration via heterogeneity, 2022. URL <a href="https://arxiv.org/abs/2201.08916.">https://arxiv.org/abs/2201.08916.</a></p><p>[5] Abdolghani Ebrahimi and Diego Klabjan. Neuron-based pruning of deep neural networks with better generalization using kronecker factored curvature approximation, 2021. URL <a href="https://arxiv.org/abs/2111.08577.">https://arxiv.org/abs/2111.08577.</a></p><p>[6] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks, 2017. URL <a href="https://arxiv.org/abs/1707.06168.">https://arxiv.org/abs/1707.06168.</a></p><p>[7] Scott Gray, Alec Radford, and Diederik P. Kingma. GPU kernels for block-sparse weights. 2017.</p><p>[8] Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks, 2021. URL <a href="https://arxiv.org/abs/2104.08378">https://arxiv.org/abs/2104.08378</a>.</p><p>[9] NVIDIA. API reference guide for cuSPARSE, 2022. URL <a href="https://docs.nvidia.com/cuda/cusparse/">https://docs.nvidia.com/cuda/cusparse/</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3791284a2127" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/accelerating-block-sparse-matrix-multiplication-with-graphcore-ipu-and-the-popsparse-library-3791284a2127">Accelerating Block Sparse Matrix Multiplication with Graphcore IPU and the PopSparse library</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we made EfficientNet more efficient]]></title>
            <link>https://medium.com/data-science/how-we-made-efficientnet-more-efficient-61e1bf3f84b3?source=rss-b23ad1d05ffe------2</link>
            <guid isPermaLink="false">https://medium.com/p/61e1bf3f84b3</guid>
            <category><![CDATA[thoughts-and-theory]]></category>
            <category><![CDATA[convolution-neural-net]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[convolution-network]]></category>
            <category><![CDATA[normalization]]></category>
            <dc:creator><![CDATA[Dominic Masters]]></dc:creator>
            <pubDate>Fri, 25 Jun 2021 12:43:52 GMT</pubDate>
            <atom:updated>2021-06-28T20:48:38.667Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/thoughts-and-theory">Thoughts and Theory</a></h4><h4>Better EfficientNet performance in practice</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cyrUCauY9GkUb4Yc-EVaBg.jpeg" /><figcaption>Image by author.</figcaption></figure><p>In our <a href="https://arxiv.org/abs/2106.03640">new paper</a> “Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Training”, we take the state-of-the-art model EfficientNet [1], which was optimised to be — theoretically — efficient, and look at three ways to make it more efficient in practice on IPUs.</p><p>For example, adding group convolutions, <a href="https://www.graphcore.ai/posts/delving-deep-into-modern-computer-vision-models">which have been shown to perform well on IPUs</a>, achieved up to a 3x improvement in practical training throughput with minimal difference in the theoretical compute cost.</p><p>Combining all three methods investigated, we achieve up to a 7x improvement in training throughput and 3.6x improvement on inference on IPUs, for comparable validation accuracy.</p><p>The theoretical cost of model training, typically measured in FLOPs, is easy to calculate and agnostic to the hardware and software stack being used. These characteristics make it an appealing, complexity measure that has become a key driver in the search for more efficient deep learning models.</p><p>In reality, however, there has been a significant disparity between this theoretical measure of the training cost and the cost in practice. This is because a simple FLOP count does not take into account many other important factors, such as the structure of the compute and data movement.</p><h3>Introducing Group Convolutions</h3><p>The first method we investigate is how to improve the performance associated with depthwise convolutions (in other words, group convolutions with group size 1). EfficientNet natively uses depthwise convolutions for all spatial convolution operations. They are well known for being FLOP and parameter efficient and so have been successfully utilised in many state-of-the-art convolutional neural networks (CNNs). However, they present several challenges for acceleration in practice.</p><p>For example, as each spatial kernel is considered in isolation, the length of the resulting dot product operations, that are typically accelerated by vector multiply-accumulate hardware, is limited. This means this hardware cannot always be fully utilised, resulting in “wasted” cycles.</p><p><a href="https://www.graphcore.ai/posts/delving-deep-into-modern-computer-vision-models">Depthwise convolutions also have very low arithmetic intensity</a> as they require a significant amount of data transfer relative to the number of FLOPs performed, meaning memory access speed is an important factor. While this can limit throughput on alternative hardware, the IPU’s In-Processor Memory architecture delivers high-bandwidth memory access which can significantly improve performance for low arithmetic intensity operations like these.</p><p>Finally, depthwise convolutions have been found to be most effective when they are sandwiched between two dense pointwise “projection” convolutions to form an MBConv block. These pointwise convolutions increase and decrease the dimensionality of the activations by an “expansion factor” of 6 around the spatial depthwise convolution. While this expansion leads to good task performance, it also creates very large activation tensors, which can dominate memory requirements and, ultimately, limit the maximum batch size that can be used.</p><p>To address these three issues, we make a simple but significant alteration to the MBConv block. We increase the size of the convolution groups from 1 to 16; this leads to better IPU hardware utilisation. Then, to compensate for the increase in FLOPs and parameters, and address the memory issues, we reduce the expansion ratio to 4. This leads to a more memory efficient and computationally compact version of EfficientNet that we refer to as G16-EfficientNet.</p><p>While these alterations were primarily motivated by throughput improvements, we also found that they enabled us to achieve higher ImageNet validation accuracy than the vanilla group size 1 (G1-EfficientNet) baseline across all the model sizes. This modification leads to significant improvements in practical efficiency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k9juS4ZYDua3EvhGb3N4_w.png" /><figcaption>Comparison of theoretical (left) and practical (right) efficiency of G1-EfficientNet (baseline) vs G16 variant (ours). Images by author.</figcaption></figure><h3>Proxy Normalised Activations</h3><p>Normalising the outputs of convolution and matrix multiply operations has become an essential element of modern CNNs, with Batch Normalisation the most common form method used for this purpose. However, the constraints on batch size introduced by Batch Norm are a well-known issue that sparked a string of innovations in batch-independent alternatives. While many of these methods work well with ResNet models, we found that none of them achieve the same performance as Batch Norm for EfficientNet.</p><p>To address this lack of alternative to Batch Norm, we leverage the novel batch-independent normalisation method Proxy Norm, introduced in a recent <a href="https://arxiv.org/abs/2106.03743">paper</a>. This method builds on the already successful methods of Group (and Layer) Normalisation.</p><p>Group Norm and Layer Norm suffer from an issue where the activations can become channelwise denormalised. This issue becomes worse with depth since the denormalisation gets accentuated at every layer. While this issue could be avoided by simply reducing the size of groups in Group Norm, such a reduction in the size of groups would, however, alter the expressivity and penalise performance.</p><p>Proxy Norm provides a better fix by preserving expressivity while counteracting the two main sources of denormalisation: the affine transformation and the activation function that follow Group Norm or Layer Norm. Concretely, the denormalisation is counteracted by assimilating the outputs of Group Norm or Layer Norm to a Gaussian “proxy” variable and by applying the same affine transformation and the same activation function to this proxy variable. The statistics of the denormalised proxy variable are then used to correct the expected distributional shift in the real activations.</p><p>Proxy Norm allows us to maximise the group size (i.e. use Layer Norm) and to preserve expressivity without the issue of channel-wise denormalisation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/1*DeUbOQvpB2ACAtklqmxzMw.png" /><figcaption>Convolution block with additional Proxy-Normalised Activation operations shown in red. Image by author.</figcaption></figure><p>This novel normalisation technique is explored in detail in the associated paper [2].</p><p>Importantly, this overall approach does not emulate any of the implicit regularisation characteristics of Batch Norm. For this reason, additional regularisation is required — in this work we use a combination of mixup and cutmix. When comparing the performance of Layer Norm + Proxy Norm (LN+PN) to two Batch Norm (BN) baselines with standard pre-processing and AutoAugment (AA), we find that LN+PN matches or exceeds the performance of BN with standard pre-processing across the full range of model sizes. Furthermore, LN+PN is nearly as good as BN with AA, despite AA requiring an expensive process of “training” the augmentation parameters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ENq0WC_4T5y9hClt-PXDOA.png" /><figcaption>Comparison of different normalisation methods for varying sizes of EfficientNet. Image by author.</figcaption></figure><h3>Reduced Resolution Training</h3><p>Touvron et al. (2020) [3] showed that significant accuracy gains could be achieved via a post-training fine-tuning of the last few layers using larger images than originally trained on. As this fine-tuning stage is very cheap, it was clear that this would achieve some practical training efficiency benefits. This raised a number of further interesting research questions. How should the training resolution be chosen to maximise efficiency? Given that larger images are slower to test on, how does this impact efficiency at inference?</p><p>To investigate these questions, we compared training at two different resolutions, either the “native” resolution (as defined in the original EfficientNet work) or at approximately half the pixel count. We then fine-tuned and tested at a broad range of image sizes. This allowed us to investigate the direct effect of training resolution on efficiency and determine the Pareto optimal combinations that achieved the best speed-accuracy trade-offs for training and inference.</p><p>When comparing training efficiency, we considered two testing scenarios: testing on the Native resolution or selecting the “best” resolution to maximise validation accuracy across the full sweep of resolutions.</p><p>When testing at the native resolution, we see that training with half-size images yields considerable theoretical and practical efficiency improvements. Remarkably, for a given model size, we find that training at half resolution and fine-tuning at the native resolution even yields higher final accuracy than training, fine-tuning and testing all at the native resolution. This conclusion suggests that, for ImageNet training, we should always be testing at a higher resolution than we train at. We now hope to understand if this applies to other domains too.</p><p>If we next allow ourselves to test at the “best” image resolution, we see that training at native resolution yields a significant improvement in final accuracy, narrowing the gap in the Pareto front.</p><p>It should, however, be noted that to achieve this, the “best” testing resolutions for the “native” training scheme end up being much larger than those that correspond to the half training resolution cases. This means they will be more expensive at inference time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P-MC8Me2gABGFpr3Kk5ttQ.png" /><figcaption>Images by author.</figcaption></figure><p>These results highlight the improvements to training efficiency achieved by the three improvements investigated (i) group convolutions [G16 (ours) vs G1]; (ii) proxy normalized activations [LN+PN (ours) vs GN] and (iii) half resolution training [Half (ours) vs Native]. Note that the baseline results have no fine-tuning and use the native image resolution.</p><p>Comparing the efficiency of inference on its own, we see that training at half resolution yields Pareto-optimal efficiency across the full range of accuracies. This is a remarkable result as there is no direct FLOP advantage in inference at all. Furthermore, the points along the half-resolution inference-efficiency Pareto front remain optimal for training throughput.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*Ljo3zn5hlFYPTkQWLMoSDg.png" /><figcaption>Theoretical and practical inference efficiency. Tested at all resolutions; lines highlight Pareto fronts. Images by author.</figcaption></figure><p>Across all efficiency metrics, the models with Proxy Norm perform either equivalently to or slightly better than the models with Group Norm. This stems from the improved accuracy at only a small cost in throughput of ~10%. Importantly, however, models with Proxy Norm use fewer parameters across the whole Pareto front, highlighting an additional benefit of Proxy Norm in terms of efficiency with respect to model size.</p><h3>How to make EfficientNet more efficient</h3><p>In carrying out this research, we have looked at several modifications to the EfficientNet model to improve the overall efficiency in training and inference:</p><ul><li>By adding group convolutions and reducing the expansion ratio in the MBConv blocks, we have improved IPU hardware utilisation of the spatial convolutions and reduced the memory consumption.</li><li>By training with images of half the resolution, we have cut training time and remarkably achieved better final accuracy.</li><li>By leveraging the novel normalisation method Proxy Norm, we matched Batch Norm performance without any dependency on the batch information. To our knowledge, this is the first method to achieve this for EfficientNet.</li></ul><p>Using all these methods in combination, we have achieved up to a 7x improvement in practical training efficiency and 3.6x improvement in practical inference efficiency on IPU. These results show that EfficientNet can deliver training and inference efficiency when using hardware suited to processing group convolutions, like IPU, taking it beyond the theory and towards practical, real-world applications.</p><p><a href="https://arxiv.org/abs/2106.03640">Read the paper</a></p><h3>Thank you</h3><p>Thank you to <a href="https://medium.com/@antoinelabatie">Antoine Labatie</a>, Zach Eaton-Rosen and Carlo Luschi who also contributed to this research, and thank you to our other colleagues at Graphcore for their support and insights.</p><h3>References</h3><p>[1] M. Tan, Q. V. Le, <a href="https://arxiv.org/abs/1905.11946">EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks</a> (2019), arXiv 2019</p><p>[2] A. Labatie, D. Masters, Z. Eaton-Rosen, C. Luschi, <a href="https://arxiv.org/abs/2106.03743">Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence</a> (2021), arXiv 2021</p><p>[3] H. Touvron, A. Vedaldi, M. Douze, H. Jégou, <a href="https://arxiv.org/abs/2003.08237">Fixing the train-test resolution discrepancy: FixEfficientNet</a> (2020), NeurIPS 2019</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=61e1bf3f84b3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-we-made-efficientnet-more-efficient-61e1bf3f84b3">How we made EfficientNet more efficient</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>