<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dr. Mario Michael Krell on Medium]]></title>
        <description><![CDATA[Stories by Dr. Mario Michael Krell on Medium]]></description>
        <link>https://medium.com/@dr.mario?source=rss-24372940c2b3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*BgUmuufAK9uZ3tL6xU2miQ.png</url>
            <title>Stories by Dr. Mario Michael Krell on Medium</title>
            <link>https://medium.com/@dr.mario?source=rss-24372940c2b3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 03:35:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dr.mario/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Accelerating ResNet-50 Training on the IPU: 
Behind our MLPerf Benchmark]]></title>
            <link>https://medium.com/data-science/accelerating-resnet-50-training-on-the-ipu-behind-our-mlperf-benchmark-2cefe43ab2b2?source=rss-24372940c2b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/2cefe43ab2b2</guid>
            <category><![CDATA[resnet50]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[mlperf]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Dr. Mario Michael Krell]]></dc:creator>
            <pubDate>Thu, 13 Jan 2022 17:08:14 GMT</pubDate>
            <atom:updated>2022-09-06T02:15:35.370Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Accelerating ResNet-50 Training on the IPU: Behind our MLPerf Benchmark</strong></h3><h4>A technical guide on efficient hardware scaling, memory optimization strategies and performance tools</h4><p>Written by: <a href="https://medium.com/u/24372940c2b3">Dr. Mario Michael Krell</a>, Zhenying Liu, Emmanuel Menage, and Bartosz Bogdanski</p><p>Graphcore engineers delivered outstanding performance at scale for the latest MLPerf v1.1 training results published in December 2021 [9], with our IPU-POD16 outperforming Nvidia’s flagship DGX A100 on ResNet-50.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tl0tLtrpB5TruENrcEGckg.jpeg" /><figcaption>Image by author.</figcaption></figure><p>Here, we will explain how our team achieved these accelerations at scale for this popular computer vision model. In this technical guide, we’ll uncover the various techniques and strategies used by Graphcore engineers covering efficient hardware scaling, memory optimization, experiment tracking, performance optimisation and more. While focusing on benchmark optimizations for ResNet-50 in MLPerf for the IPU [9], this guide gives a general perspective behind the thought process for benchmark optimisations that can also translate to other applications and hardware.</p><h4>Accelerating ResNet-50</h4><p>For our MLPerf v1.1 results, published in December 2021 [9], we achieved a time to train of 28.3 minutes for ResNet-50 training on ImageNet (RN50) with 30k images per second throughput and 38 epochs till convergence at 75.9% validation accuracy on an IPU-POD16. Before we started on our first MLPerf project in 2020, throughput for ResNet-50 training was around 16k images per second and it took approximately 65 epochs to reach convergence. Putting these numbers together gives us a 3.2x overall improvement. We then scaled the application from an IPU-POD16 to our IPU-POD256, which gave us another 7.5x improvement in time to train and 12x improvement in throughput. So how did we achieve this speed-up?</p><p>The tools that we developed along the way are provided in our <a href="https://github.com/graphcore/examples/tree/master/vision/cnns/tensorflow1/training">public examples</a> and are not that specific to RN50 or the MLPerf benchmark. We thought other IPU developers out there might find them useful for their models. The key ingredients are:</p><ul><li>Optimized job distribution and communication</li><li>Optimized host-IO and data layout with distribution over multiple hosts</li><li>Experiment tracking over Weights and Biases</li><li>Small effective batch sizes</li><li>Switching from SGD with Momentum to the LARS optimizer</li><li>Distributed batch-norm with caching of the statistics</li><li>Recomputation checkpoints instead of pipelined processing</li><li>Stochastic rounding</li><li>Optimized compilation and memory profiles</li></ul><h3>IPU-POD: Built for AI acceleration up to supercomputing scale</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jGJrPVCRxgPwwiHME12iNw.png" /><figcaption>Scaling the application from an IPU-POD16 to our IPU-POD256. Image by author.</figcaption></figure><h4>How IPU Hardware scales Machine Learning Applications</h4><p>To scale our application, numerous features had to be put in place. A major challenge when scaling is the increased batch size and communication which requires time for data transfer and memory for the code. For the local replicas, we want to keep the micro batch size as large as possible to obtain good statistical estimates for the batch norm which is a key part of the ResNet-50 model. The gradients for each replica need to be aggregated and the optimizer performs its update.</p><p>This comes with computational and communication overhead. One trick to reduce this is gradient accumulation on the single replicas and have a bigger effective batch size. By adding more replicas, the effective batch size would scale linearly. Whereas this approach would guarantee almost linear scaling, the higher effective batch size increases the number of epochs required to get convergence. Hence, gradient accumulation needs to be reduced for larger systems and effective communication of the gradients is essential.</p><p>In our experiments, the best compromise for IPU-POD128 and IPU-POD256 was to have a gradient accumulation count of 2, to have a slightly reduced communication but to keep the effective batch size as small as possible. Given the larger batch size, the changes mentioned in “LARS Optimizer and Hyperparameter Optimization” were necessary since SGD would increase the required epochs much more than LARS. Also, we made excessive use of our load distribution tool PopDist as described in “Scaling Host-IPU Communication”. By increasing the number of hosts with local data storage, we improve the data loading bandwidth and make sure that there is no slowdown in data transfer, no matter how many accelerators are used.</p><h4>Hardware Scaling: Connecting IPU-POD64 platforms</h4><p>Graphcore’s IPU-PODs have been designed from the ground up with that scalability in mind, as shown by the incredible scaling exhibited in our latest MLPerf submission. Within a rack, IPUs are directly connected to each other to form two rings via high-bandwidth IPU-Link connectivity. Using these direct connections, each IPU-POD64 provides a huge IPU-Link bandwidth of 10.24TB/s, with ultra-low latency (250ns). On top of these vertical intra-rack connections, IPUs are connected across racks (directly or through a switch) to form a horizontal ring. This means that doubling the number of racks more than doubles the amount of bandwidth available.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*vH414SuT1GppDPshXOmAmw.png" /><figcaption>Image by author.</figcaption></figure><p>Graphcore Communication Library (GCL) has been designed to make full use of these hardware features.</p><p>IPU-POD128 and IPU-POD256 are built from multiple IPU-POD64s. A single IPU-POD64 consists of 16 IPU-M2000 stacked vertically and connected with IPU-Links to form a 2D-Torus. The flexibility of our scale-out solution allows us to connect IPU-POD64s together horizontally to form larger systems. There are two possible methods to connect such systems: directly (GW-Links on one IPU-M connect directly to IPU-M2000s on other IPU-POD64 systems) or over a switched fabric where GW-Links on all IPU-M2000s connect to switches that then forward the traffic to correct destination IPU-M2000s on other IPU-POD64s. In both cases, the syncs between multiple IPU-POD64s are sent over the GW-Links without requiring physical sync lines to enable a simpler scale-out. For the IPU-POD128 and IPU-POD256 we use for MLPerf work, we use the switched solution as it allows us to be more flexible and provides better fail-over.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sQLJ_BaluGTs-yE9cZrANw.png" /><figcaption>Graphcore’s scale-out solution for IPU-POD128 to IPU-POD512. Image by author.</figcaption></figure><p>Looking at the individual aspects of IPU-POD scale-out topology, now we can take a snapshot of what this looks like in practice. This illustrates that even with all the flexibility and great options deploying over an existing switched fabric brings, it is actually not as intensive as might at first be thought in terms of ‘extra kit’ needed. This system has 8x IPU-POD64 racks so we can call it an IPU-POD512. Interconnecting all of the GW-Links between each IPU-M2000 in an IPU-POD512 can be achieved using just two 128 port- 100GbE switches.</p><h4>Scaling communication with the Graphcore Communication Library (GCL)</h4><p>To speed up training of AI models, one idea is to replicate the model on many IPUs, each processing a different batch. After having processed a batch, replicas share what they have learned from the batch (the gradients) using an all reduce. In the ideal case, if the All-Reduce was instantaneous, we would have perfect scaling in the sense of throughput: doubling the number of replicas would allow us to process double the amount of training samples.</p><p>Several optimizations enabling the scale-out beyond a single IPU-POD64 have been implemented in GCL. All model replica internal data exchange is known and detailed by Poplar at compile time and with the assumption that all collective communication, ML model details, and network topology also are known at compile time, the GCL execution is deterministic and can be scheduled and optimized accordingly. GCL also exposes a public API for specifying communication groups as sub-sets of the total set of replicas giving us the flexibility to perform operations on large systems. For the larger scaling to IPU-POD256, we use a phased all reduce where the tensor reduction over a multi-POD system is performed as three distinct operations.</p><p>First, a ReduceScatter is done over the IPU-Links within each of the IPU-POD64s that are the building blocks of an IPU-POD256. The output of the Reduce Scatter becomes an input to the All-Reduce that is run over the GW-Links. This exchanges the data elements between multiple IPU-PODs making sure every replica at the same level contains the same data. Finally, an All-Gather collective operation is performed on the output of the All-Reduce and again it runs within each individual IPU-POD64 so that every replica in the end contains the same copy of the data.</p><p>A crucial characteristic of our implementation is that the time spent in the All-Reduce is a function of the total amount of data to share divided by the total bandwidth available. Because doubling the number of IPUs means double the amount of bandwidth, the time for the All-Reduce remains constant. Having that constant time All-Reduce is the key feature of Graphcore’s offering: it enables innovators to experiment faster by using more hardware effectively reducing the feedback loop ultimately leading to exciting discoveries.</p><p>Given our small gradient accumulation count of 2 for large systems, any optimization of the communication of the gradients had a major impact on our scaling performance. For example, we grouped tensors of different types and shapes together and did the communication in one shot. Also, to make full use of the bandwidth, we optimized the gateway between host and IPU, to immediately forward the data and thus significantly reducing the latency across a variety of applications. Each IPU-POD64 comes with its own gateway. So, this change is mostly benefiting the scaling from IPU-POD16 to IPU-POD64.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QcRSOo6s6FDMXmJwein8Iw.png" /><figcaption>Visual representation of the Three-Phase All-Reduce. Image by author.</figcaption></figure><h4>Memory Optimizations for Scaling</h4><p>Whereas the memory required by the data stays the same when scaling, additional memory is required for the exchange code. Thus, memory optimizations as for example mentioned in “Other Memory and Speed Optimizations” were crucial for our scaling success, especially since the LARS optimizer required even more memory than SGD. A specific optimization for scaling to IPU-POD128 and IPU-POD256 was the reduction of the code size for the exchange between IPU-PODs.</p><h4>Scaling the Validation to an IPU-POD256</h4><p>For validation, no backward pass is needed and in the batch norm layers, the synchronized moving mean and variance are applied. Thus, the computational layout is substantially different. To not waste time with graph swapping, we instead apply an offline evaluation scheme, i.e., we store the checkpoints during training and afterwards, we evaluate the performance using the validation data. We also keep the same configuration of the IPUs to avoid time for reconfiguration. That means for the case of the POD256, we have 128 host processes where each one is responsible for one validation file and results then get aggregated to calculate the overall performance. Since we are working with static graphs, the 50000 validation images cannot be perfectly split even with a batch size of 1, which would be inefficient. However, for the MLPerf setting, all images need to be processed and samples cannot be omitted. Thus, we pad our data to the next larger batch and then remove the padding results in the post-processing.</p><h3>Accelerating ResNet-50 at scale on IPU Hardware</h3><h4>Experiment Tracking</h4><p>During the whole project, it was important to keep track of experiments, look up the detailed parameter configurations and dive deep into the logs. Thus, we took great advantage of the experiment tracking with Weights and Biases and regularly created reports to communicate results to colleagues and compare runs to figure out any potential difference. What made the experiments challenging was that with a low epoch count, convergence gets overly sensitive to any changes. For example, cutting the accumulation count in half could easily transition a non-converging setup to a converging one. Also, there was a large variation in between experiments which required this approach to experiment tracking. We tracked more than 2600 experiments. Recently, we extended our tracking in our CNN repository. With the <em>--wandb </em>and <em>--wandb-project</em> option, results get uploaded right away instead of waiting for the final result to be uploaded.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0rTZRFPTdS_p3WlYd36ueQ.png" /><figcaption>Weights and Biases Report. Image by author.</figcaption></figure><h4>Reducing the number of epochs</h4><h4>Batch-Norm vs. Group-Norm</h4><p>In practice, people usually train their network around 90 epochs or even more. Since the MLPerf benchmark only cares about the time to train, there is a lot of tweaking happening to reduce this number. Our goal was to reduce the number to 44 epochs or even less given previous results in the competition. Our original implementation used group norm instead of batch-norm for faster processing and avoiding dependence between samples. For a discussion, see [1].</p><p>For 65 epochs or more, this normalization works perfectly. However, while fine-tuning hyperparameters, we learned that a more aggressive and costly normalization is required to get convergence in the desired small epoch count. That means the tools that we developed to enable large batch-norm were specific to the MLPerf benchmark in the sense of having to reduce the number of epochs. Without this requirement, we could have stuck to group norm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*231pRYgea5hjzVYkxaRIiQ.png" /><figcaption>Each subplot corresponds to a specific technique of normalisation characterised by its groups of components sharing the same normalisation statistics. In each subplot, the tensor of intermediate activations is constituted of the batch axis B, the channel axis C, and the spatial axes (H, W). Image by author, adapted from [7].</figcaption></figure><h4>Increasing the batch-size</h4><p>Our original setup was a pipeline over 4 IPUs (Intelligence Processing Units) with a replication factor of 4 for data parallel processing and grouped schedule. We used a batch size of 16. Changing that setting to batch-norm was a simple configuration change but still convergence improved only slightly. It got better when we fine-tuned the memory and pipeline stages to get a batch size of 24 by using memory efficient communication, limiting the code memory size for each IPU, and adjusting the length of pipeline stages. Still, we could not reach the anticipated 44 epochs. So, we ran some simulations and discovered that a batch size of 32 should be sufficient. With the simulation, we could reach convergence in 44 epochs, but the throughput was below acceptable, and we needed some tweaks as described in the “Increasing the Throughput” section.</p><h4>Small Effective Batch Size</h4><p>For getting good hardware acceleration, it is common to work with large batches of data. Using more data on a single accelerator usually increases the usage of the ALU (Arithmetic Logic Unit) and makes sure that the accelerator gets enough workload. Note that the IPU has a unique way of distributing workloads and can also accelerate workloads with small batch sizes. Additionally, communication of gradients and optimizer updates can be expensive and thus throughput can be increased by making these updates less frequent, for example by using gradient accumulation. Note that gradient accumulation gives diminishing returns with every increase though. In our experiments, we realized that larger batch-sizes require more epochs for the training as can be also seen in the MLPerf Reference Convergence Points [8] . Thus, the benefit of larger batches can outweigh the gain of speed. On the other hand, a minimum batch size of 32 is required per “replica” to make batch norm work.</p><p>For the optimal IPU-POD16 config, we ended up with an effective batch size of 3200 (16 replicas, each with a micro-batch size of 20, and a gradient accumulation count of 10) which converged in 38 epochs. <strong>For scaling to larger systems</strong>, we reduced the gradient accumulation count, respectively. Due to communication efficiency and optimizations on the larger machines, it was sufficient to keep a gradient accumulation count of 2. For the IPU-POD64 system, we used a similar effective batch size of 3840 (64 replicas, gradient accumulation count of 3) to also converge in 38 epochs. For IPU-POD128 and IPU-POD256, we used a gradient accumulation count of 2 with an effective batch size of 5120 and 10240 and respective epochs till convergence of 41 and 45. Using larger sizes would have increased the epoch counts much more and was not feasible.</p><h4>LARS Optimizer and Hyperparameter Optimization</h4><p>Whereas normal ML settings show some robustness in the choice of hyperparameters, the low epoch count in MLPerf makes the convergence sensitive to any changes. This can be best seen by the choice of optimizer. LARS is known to be a good optimizer for large batch sizes. During our project, we learned that it is also a better optimizer than Stochastic gradient descent with Momentum (SGD) in the sense of requiring less epochs till convergence. This benefit grows with increasing batch size. Note that LARS requires more memory and compute, but the minor costs exceed the major savings. One major change between our first and second MLPerf submission was to optimize memory usage and enable LARS which resulted in a great speed-up. For example, for our IPU-POD16 setup, LARS reduced the number of required epochs from 44 to 38. One key ingredient is the number of warmup epochs. Whereas SGD requires 5 epochs, LARS can work with only 2 epochs. Another sensitive hyperparameter is the weight decay parameter in LARS, reducing it by a factor of two and adjusting the learning rate, accelerates convergence by up to 2 epochs.</p><h4>Increasing the Throughput</h4><h4>Pipeline balancing</h4><p>In contrast to the natural language processing model, BERT, optimizing pipelines for RN50 can be challenging. There are a lot of different components that require different code and the size of the data in between the layers changes a lot. So, modifying the split strategy just a little bit could have a major impact on the memory profile, jumping from a lot of memory left to requiring too much memory. Also, processing load can be highly imbalanced as we discovered with the PopVision Graph Analyser. The fourth IPU took twice as long for the processing compared to the second and there was no room for balancing due to memory requirements of the data. On the other hand, IPU0 could not do too much processing because it had the largest stash of activations to store. Similarly, IPU1 and IPU2 also had some idle time which is even larger than the synchronization depicted by the green bar. Due to the imbalanced computation and memory requirements, it is hard to improve further, and we chose a different route, using a recomputation approach that can work on a single IPU and uses distributed batch norm to enable a sufficiently large batch size.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B6jlwRyA0fdFY71O2DRjdQ.png" /><figcaption>PopVision Graph Analyser excerpt of a grouped schedule pipeline over 4 IPUs with batch size 20. The third row (IPU 2) corresponds to the last, 4th stage. IPU 1 needs less than 60% of the processing cycles compared to IPU 2. Image by author.</figcaption></figure><h4>Distributed Batch Norm</h4><p>Our batch norm implementation is fused and optimized on the Poplar level. With all the memory fine-tuning we used before for the pipeline setup, we realized that we could now increase the batch size on a new setup on a single IPU from 6 to 12 without any pipelining. Now, to get to a batch size of 32 for the batch norm, we only required a distributed batch norm. For everything else, normal data parallel processing would be sufficient. We added this feature into our framework with just a few lines of code where we aggregated the statistics across multiple replicas within the normalization implementation. This approach was very efficient and did not require much more memory, especially since the vectors for the statistics are quite small. Also, the backwards pass was just an aggregation of gradients between the IPUs that contributed to the distributed batch norm and thus resulted only in a single line of code.</p><p>For the application code in TensorFlow 1.15, the interface was kept minimalistic. Only a configuration number must be set:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/43e539a66cbd8cb11efcca45bf0418b8/href">https://medium.com/media/43e539a66cbd8cb11efcca45bf0418b8/href</a></iframe><p>For a batch size of 12, we would need to aggregate data between at least 3 IPUs to reach a batch norm batch size of at least 32. To avoid communication between IPU-M2000 machines, it would have to be 4 IPUs instead.</p><h4>Recomputation</h4><p>For better throughput and less communication overhead from the distributed batch norm, we wanted to increase the batch size and limit the distributed batch norm to two IPUs. Thus, we took advantage of recomputation. Since there is no native support for this in TensorFlow 1.15, we used a programming trick. The IPU implementation of pipelining comes with a sequential schedule which can be used to shard a computational graph over multiple IPUs. Instead, we did not distribute the processing over multiple IPUs but kept them all on one IPU. Note that the code size in the on-chip memory barely increases but a lot of intermediate activations do not get stored but recomputed. Thus, we are freeing a lot of memory that can be used to increase the batch size. By setting our splits to b1/0/relu, b1/2/relu, b2/0/relu, b2/2/relu, b3/0/relu, and b3/3/relu, we increased the batch size to 20. We could have used less recomputation points by offloading some of the optimizer state to the host. We refrained from that possibility and kept everything in memory because we made better use of the communication bandwidth by bringing more data to the IPU. So, all that is required for recomputation are three ingredients:</p><ul><li>Use the IPU pipeline API at <a href="https://docs.graphcore.ai/projects/tensorflow1-user-guide/en/latest/api.html#tensorflow.python.ipu.pipelining_ops.pipeline">https://docs.graphcore.ai/projects/tensorflow1-user-guide/en/latest/api.html#tensorflow.python.ipu.pipelining_ops.pipeline</a></li><li>Set “PipelineSchedule.Sequential” as the “pipeline_schedule”</li><li>Put all computational stages onto the same device (IPU 0) by setting: <br>device_mapping = [0] * len(computational_stages) — in our case, there were 7 computational stages</li></ul><p>Balancing recomputation points requires the knowledge of memory occupation of the different stages in the computational graph. For fine-tuning, we used the memory profiles in the PopVision Graph Analyzer to make sure we make optimal use of the memory. As a result, we have more dense set of recomputation checkpoints at the initial layers and keep a longer final stage to reduce the amount of computational overhead because for the last stage the forward pass never gets recomputed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sLsKgas6Yqcx94YxyMm1GQ.png" /><figcaption>The memory utilization for 1472 tiles of a single IPU. It’s reaching the memory limit for each tile. Image by author using the PopVision Graph Analyser.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lJsQOCwEx0mAR62SFEY_bw.png" /><figcaption>Liveness report created with the PopVision Graph Analyser. We can see the forward pass excluding the last section, followed by 7 bumps for each section of recomputation of forward pass (memory increase) and backward pass (memory decrease). Image by author.</figcaption></figure><h4>Scaling Host-IPU Communication</h4><p>For the ResNet-50 benchmark data loading on preprocessing a crucial part of the workload. In one of our experiments, we tried an Intel based host with fewer CPU cores which slowed down our processing significantly compared to using an AMD host. Thus, our disaggregated host architecture is especially important to be able to choose the right host for the application at hand.</p><p>A simple approach to get more data transferred is to reduce its size. The augmented image data was transferred in Int8 format, and then normalization happened on the IPU instead of the host. The normalization was implemented as a fused operation that cast the data to Float16 and added a fourth channel with zeroes to reduce memory footprint of the subsequent convolutional layer.</p><p>Additionally, to transfer more data, the data loading needs to be parallelized using Graphcore’s PopRun framework [6]. Our analysis showed that for each set of two replicas, one host process is best. Each host was running optimally with 8 numa-aware nodes, 4 for each CPU. Thus, for IPU-POD 16, 64, 128, and 256, we used 1, 4, 8, and 16 hosts, respectively. The command looks something like:</p><p>poprun --host $HOSTS $MPI_SETTINGS --num-instances “$INSTANCES” --num-replicas “$REPLICAS” python train.py args</p><p>Numa awareness is activated, using “<em>--numa-aware 1”</em> in the $MPI_SETTINGS. Whereas the host processes are independent MPI processes, a combined representation of the computational graph over all replicas is created across multiple racks. Each host process reads a separate equally sized set of input files. Thus, any conflict in file reading is avoided. The distributed read data is kept in cache even though the effect on performance is negligible.</p><h4>Other Memory and Speed Optimizations</h4><p>Apart from recomputation, we used a few more optimization strategies:</p><ul><li>We store the batch-norm statistics to avoid recomputation.</li><li>We did all calculations in FP16 for weights, activations, and gradients. We also used stochastic rounding to improve convergence and half partials for convolutions and matrix multiplications for speed up. Stochastic rounding is supported by our hardware. Note that the weight update needs to have the same update on each replica to make sure weights stay aligned. This is achieved by synchronizing the starting seeds using horovod broadcasting <br>with tf.Graph().as_default(), tf.Session(): <br> identical_seed = hvd.broadcast( <br> identical_seed, root_rank=0, name=”broadcast_seed”).eval() <br>and ensuring that IPUs are equally seeded when the weight update is applied. This is achieved with <br>export POPLAR_ENGINE_OPTIONS=’{“target.deterministicWorkers”:”portable”}’</li><li>We fine-tuned the balance between the memory used for code and the memory used to store activations on the IPU by optimizing the “availableMemoryProportion” of the IPUConfig [5] which resulted in a value around 0.15. <br>config.convolutions.poplar_options[‘availableMemoryProportion’] = …</li></ul><p>For more ideas on memory and processing optimizations, please check out our new guide [4].</p><h3>Limitations</h3><p>All features introduced in this blog are readily available with the latest release. However, we would like to point out that whereas all these technical tricks might help in your application they do not have to. The MLPerf benchmark comes with special rules and limitations that will not hold for practical applications.</p><ul><li>The target accuracy, requested by the benchmark, is below the maximal achievable one.</li><li>For the benchmark, only 4 out of 5 runs need to converge. If there is one failure, it gets dismissed.</li><li>Only a subset of the hyperparameters can be changed and the only available optimizers are SGD with Momentum and LARS.</li><li>The target metric is time to train/convergence and not throughput. This is probably because with higher batch sizes and accumulation counts, the throughput can be increased but the time to convergence will take many more epochs.</li><li>We evaluate only every 4 epochs. A higher evaluation frequency is against the rules.</li><li>The training happens solely on ImageNet.</li><li>The learning rate schedule is a polynomial schedule with degree two and not a stepped or cosine learning rate schedule.</li></ul><p>In practice, no one would train ResNet-50 on ImageNet to 75.9% validation accuracy. It is easier to download a pretrained network with higher accuracy.</p><p>In practice one would train on a different dataset to a higher accuracy. However, when training on a different dataset, the chosen parameters for this benchmark might not transfer. Also, a larger epoch count, and a different learning rate schedule might be desired. With an increase in the number of epochs though, it might be better to go with group norm (or proxy norm) if the training happens from scratch and not with a pretrained model.</p><p>A Top1 accuracy of 75.9% is not sufficient for some applications like search. Imagine, you have a search engine on 100.000.000 images like the YFCC100M dataset and you try to search for a certain class of images like for example origami using ResNet-50 with this accuracy. Missing 24% of all relevant images might be ok for you. The challenge will be that because the other class(es) also get misclassified, you will get many more images that are not origami but misclassifications from the other images. In fact, these misclassifications will be the majority [2]. To achieve higher accuracies, it is worth considering more state-of-the-art models like EfficientNet, which benefit from some of the features mentioned in this blog. You might also want to read, <a href="https://www.graphcore.ai/posts/how-we-made-efficientnet-more-efficient">“How we made EfficientNet more efficient”</a>[3].</p><h3>Conclusion</h3><p>Combining the powerful TensorFlow 1.15 framework with Weights and Biases, Poplar, and the PopVision tools together with the powerful MK2 IPU, we managed to take full advantage of the processing speed and memory and achieved a stunning performance.</p><h3>Thanks</h3><p>A big thanks goes out to Brian Nguyen, Godfrey da Costa (and his team), and Mrinal Iyer who went above and beyond to make this work happen.</p><p>We would also like to thank Alex Cunha, Phil Brown, Stuart Cornell, <a href="https://medium.com/u/b23ad1d05ffe">Dominic Masters</a>, Adam Sanders, Håkon Sandsmark, George Pawelczak, Simon Long, <a href="https://medium.com/u/83b5c2605cec">Luke Hudlass-Galley</a>, Jorge Bellon Castro, George Matthew, and many more for their valuable contributions.</p><h3>References</h3><p>[1] A. Labatie, <a href="https://towardsdatascience.com/removing-batch-dependence-in-cnns-by-proxy-normalising-activations-bf4824eb0ba4">Removing Batch Dependence in CNNs by Proxy-Normalising Activations</a>, Towards Data Science 2021</p><p>[2] D. Ma, G. Friedland, M. M. Krell, <a href="https://arxiv.org/abs/2101.05470">OrigamiSet1.0: Two New Datasets for Origami Classification and Difficulty Estimation</a>, arxiv 2021</p><p>[3] D. Masters, <a href="https://towardsdatascience.com/how-we-made-efficientnet-more-efficient-61e1bf3f84b3">How we made EfficientNet more efficient</a>, Towards Data Science 2021</p><p>[4] <a href="https://docs.graphcore.ai/projects/memory-performance-optimisation/en/latest/index.html">Memory and Performance Optimisation Guide</a>, Graphcore</p><p>[5] <a href="https://docs.graphcore.ai/projects/available-memory/en/latest/available-memory.html">Optimising Temporary Memory Usage for Convolutions and Matmuls on the IPU</a>, Graphcore</p><p>[6] <a href="https://docs.graphcore.ai/projects/poprun-user-guide/en/latest/index.html">PopDist and PopRun: User Guide</a>, Graphcore</p><p>[7] N. Dimitriou and O. Arandjelovic, <a href="https://arxiv.org/pdf/2007.08554.pdf">A New Look at Ghost Normalization</a>, arXiv 2020</p><p>[8] <a href="https://github.com/mlcommons/logging/blob/master/mlperf_logging/rcp_checker/training_1.1.0/rcps_resnet.json">MLPerf Reference Convergence Points</a>, MLCommons</p><p>[9] MLPerf v1.1 Training Results. MLPerf ID: 1.1–2040, 1.1–2042, 1.1–2044, 1.1–2045, 1.1–2065. The MLPerf name and logo are trademarks. See <a href="http://www.mlperf.org/">www.mlperf.org</a> for more information.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2cefe43ab2b2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/accelerating-resnet-50-training-on-the-ipu-behind-our-mlperf-benchmark-2cefe43ab2b2">Accelerating ResNet-50 Training on the IPU: 
Behind our MLPerf Benchmark</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing Packed BERT for 2x Training Speed-up in Natural Language Processing]]></title>
            <link>https://medium.com/data-science/introducing-packed-bert-for-2x-faster-training-in-natural-language-processing-eadb749962b1?source=rss-24372940c2b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/eadb749962b1</guid>
            <category><![CDATA[bert]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[thoughts-and-theory]]></category>
            <dc:creator><![CDATA[Dr. Mario Michael Krell]]></dc:creator>
            <pubDate>Wed, 07 Jul 2021 21:40:01 GMT</pubDate>
            <atom:updated>2021-07-09T01:03:19.878Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/thoughts-and-theory">Thoughts and Theory</a></h4><h4><em>New BERT packing algorithm for more efficient training</em></h4><p>Authors: <a href="https://medium.com/@dr.mario/about">Dr. Mario Michael Krell</a> and <a href="https://medium.com/@matej.kosec/about">Matej Kosec</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mj8FHQ5tVXFEPnv5ab__vg.png" /><figcaption>Image by author.</figcaption></figure><p>By using a new packing algorithm, we have sped up Natural Language Processing by more than 2 times while training BERT-Large. Our new packing technique removes padding, enabling significantly more efficient computation.</p><p>We suspect this could also be applied to genomics and protein folding models and other models with skewed length distributions to make a much broader impact in different industries and applications.</p><p>We introduced Graphcore’s highly efficient Non-Negative Least Squares Histogram-Packing algorithm (or NNLSHP) as well as our BERT algorithm applied to packed sequences in a new paper [1].</p><h3>Computational Waste in NLP due to sequence padding</h3><p>We began to investigate new ways of optimising BERT training while working on our recent <a href="https://www.graphcore.ai/posts/raising-the-bar-graphcores-first-mlperf-results">benchmark submissions to MLPerf™</a>. The goal was to develop useful optimisations that could easily be adopted in real-world applications. BERT was a natural choice as one of the models to focus on for these optimisations, as it is widely used in industry and by many of our customers.</p><p>It really surprised us to learn that in our own BERT-Large training application using the Wikipedia dataset, 50% of the tokens in the dataset were padding — resulting in a lot of wasted compute.</p><p>Padding sequences to align them all to equal length is a common approach used with GPUs, but we thought it would be worth trying a different approach.</p><p>Sequences have a large variation of length for two reasons:</p><ol><li>The underlying Wikipedia data shows a large variation in document length</li><li>The BERT-pre-processing itself randomly reduces the size of extracted documents that are combined to generate a training sequence</li></ol><p>Filling up the length to the maximum length of 512 results in 50% of all tokens being padding tokens. Replacing the 50% of padding by real data could result in 50% more data being processed with the same computational effort and thus 2x speed-up under optimal conditions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9FtehUu_Chxn3o5s0O6l4w.png" /><figcaption>Figure 1: Wikipedia dataset distributions. Image by author.</figcaption></figure><p>Is this specific to Wikipedia? No.</p><p>Well, then is it specific to language? No.</p><p>In fact, skewed length distributions are found everywhere: in language, genomics and protein folding. Figures 2 and 3 show distributions for the SQuAD 1.1 dataset and GLUE datasets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/1*M3LFL3s9hpdFDRZ7q4qRPw.png" /><figcaption>Figure 2: SQuAD 1.1 BERT pre-training dataset sequence length histogram for maximum sequence length of 384. Image by author.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*8YCigtZg9nyGve5hwmW-uw.png" /><figcaption>Figure 3: GLUE dataset sequence length histograms for maximum sequence length of 128. Image by author.</figcaption></figure><p>How can we handle the different lengths while avoiding the computational waste?</p><p>Current approaches require different computational kernels for different lengths or for the engineer to programmatically remove the padding and then add it back in repeatedly for each attention block and loss calculation. Saving compute by blowing up the code and making it more complex was not appealing, so we searched for something better. Can’t we just put multiple sequences together in a pack with a maximum length and process it all together? It turns out, we can!</p><p>This approach requires three key ingredients:</p><ol><li>An efficient algorithm to decide which samples to put together to have as little remaining padding as possible</li><li>Adjusting the BERT model to process packs instead of sequences</li><li>And adjusting the hyperparameters</li></ol><h3>Packing</h3><p>At first, it seemed unlikely that you would be able to pack a large dataset such as Wikipedia very efficiently. This problem is commonly known as bin-packing. Even when packing is limited to three sequences or less, the resulting problem would be still strongly NP-complete, lacking an efficient algorithmic solution. Existing heuristics packing algorithms were not promising because they had a complexity of at least <em>O</em>(<em>n log</em>(<em>n</em>)), where <em>n</em> is the number of sequences (~16M for Wikipedia). We were interested in approaches that would scale well to millions of sequences.</p><p>Two tricks helped us to reduce the complexity drastically:</p><ol><li>Limiting the number of sequences in a pack to three (for our first solution approach)</li><li>Operating solely on the histogram of sequence length with one bin for each occurring length</li></ol><p>Our maximum sequence length was 512. So, moving to the histogram reduced the dimension and complexity from 16 million sequences to 512 length counts. Allowing a maximum of three sequences in a pack reduced the number of allowed length combinations to 22K. This already included the trick to require the sequences to be sorted by length in the pack. So why not try 4 sequences? This increased the number of combinations from 22K to 940K, which was too much for our first modelling approach. Additionally, depth 3 already achieved remarkably high packing efficiency.</p><p>Originally, we thought that using more than three sequences in a pack would increase the computational overhead and impact the convergence behaviour during training. However, to support applications such as inference, which require even faster, real-time packing, we developed the highly efficient Non-Negative Least Squares Histogram-Packing (NNLSHP) algorithm.</p><h4>Non-Negative Least Squares Histogram-Packing (NNLSHP)</h4><p>Bin packing is quite often formulated as a mathematical optimization problem. However, with 16 million sequences (or more) this is not practical. The problem variables alone would exceed most machines’ memory. The mathematical program for a histogram-based approach is quite neat. For simplicity, we decided for a least squares approach (<em>Ax=b</em>) with histogram vector <em>b</em>. We extended it by requesting the strategy vector <em>x</em> to be non-negative and adding weights to allow for minor padding.</p><p>The tricky part was the strategy matrix. Each column has a maximum sum of three and encodes which sequences get packed together to exactly match the desired total length; 512 in our case. The rows encode each of the potential combinations to reach a length the total length. The strategy vector <em>x</em> is what we were looking for, which describes how often we choose whichever of the 20k combinations. Interestingly, only around 600 combinations were selected at the end. To get an exact solution, the strategy counts in <em>x</em> would have to be positive integers, but we realised that an approximate rounded solution with just non-negative <em>x</em> was sufficient. For an approximate solution, a simple out-of-the-box solver could be used to get a result within 30 seconds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RJceuYZZJYwz1HNc5MUrug.png" /><figcaption>Figure 4: Example of a strategy matrix for sequence length 8 and packing depth 3. The rows stand for the sequences of length 1–8 that get packed together and the columns stand for all possible length combinations in a pack with no particular ordering. Image by author.</figcaption></figure><p>At the end, we had to fix some samples that did not get assigned a strategy but those were minimal. We also developed a variant solver that enforces that each sequence gets packed, potentially with padding, and has a weighting dependent on the padding. It took much longer, and the solution was not much better.</p><h4>Shortest-Pack-First Histogram Packing</h4><p>NNLSHP delivered a sufficient packing approach for us. However, we were wondering if we could theoretically get a faster online capable approach and remove the limitation of putting only 3 sequences together.</p><p>Therefore, we took some inspiration from existing packing algorithms but still focused on the histograms.</p><p>There are four ingredients for our first algorithm, Shortest-pack-first histogram-packing (SPFHP):</p><ol><li>Operate on the counts of the histogram from longest sequences to shortest</li><li>If the current sequence length does not fit into any pack, start a new set of packs</li><li>If there are multiple fits, take the pack where the sum of sequence length is the shortest and modify the counts respectively</li><li>Check again for fits of the remaining counts</li></ol><p>This approach was the most straightforward to implement and took only 0.02 seconds.</p><p>A variant was to take the largest sum of sequence length instead of the shortest and split counts to get more perfect fits. Overall, this did not change efficiency much but increased code complexity a lot.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplay.vidyard.com%2Fr2nX2Ddfx3Mbp1sHPuyFb7.html%3Fautoplay%3D0%26custom_id%3D%26embed_button%3D0%26viral_sharing%3D0%26&amp;display_name=Vidyard&amp;url=https%3A%2F%2Fshare.vidyard.com%2Fwatch%2Fr2nX2Ddfx3Mbp1sHPuyFb7&amp;image=https%3A%2F%2Fplay.vidyard.com%2Fr2nX2Ddfx3Mbp1sHPuyFb7.jpg%3F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=vidyard" width="600" height="337" frameborder="0" scrolling="no"><a href="https://medium.com/media/56d00e2f0c3aa972406a6d7ff8aad22a/href">https://medium.com/media/56d00e2f0c3aa972406a6d7ff8aad22a/href</a></iframe><h4>Wikipedia, SQuAD 1.1, GLUE packing results</h4><p>Table 1, 2 and 3 show the packing results of our two proposed algorithms. <strong>Packing depth</strong> describes the maximum number of packed sequences. Packing depth 1 is the baseline BERT implementation. The maximum occurring packing depth, wen no limit is set is denoted with an additional “max”. The <strong>number of packs</strong> describes the length of the new packed dataset. <strong>Efficiency</strong> is the percentage of real tokens in the packed dataset. The <strong>packing factor</strong> describes the resulting potential speed-up compared to packing depth 1.</p><p>We had four main observations:</p><ol><li>The more skewed a distribution, the higher the benefits of packing.</li><li>All datasets benefit from packing. Some even by more than a factor of 2.</li><li>SPFHP gets more efficient when the packing depth is not limited.</li><li>For a maximum number of 3 packed sequences, the more complex NNLSHP is, the more efficient it is (99.75 vs. 89.44).</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9PyECUvxhs9uY4JSxX3Xog.png" /><figcaption>Table 1: Key performance results of proposed packing algorithms (SPFHP and NNLSHP) on Wikipedia. Image by author.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8Q4qLbFUTV3BIxdODZjjwQ.png" /><figcaption>Table 2: Performance results of proposed packing algorithms for SQUaD 1.1 BERT pre-training. Image by author.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ccxGKUWzV_tvDrtkNNnxag.png" /><figcaption>Table 3: Performance results of proposed packing algorithms for the GLUE dataset. Only the baseline and the SPFHP packing results without limiting the packing depth are displayed. Image by author.</figcaption></figure><h3>BERT processing adjustment</h3><p>Something interesting about the BERT architecture is that most processing happens on a token level, which means it does not interfere with our packing. There are only four components that need an adjustment: the attention mask, the MLM loss, the NSP loss and the accuracy.</p><p>The key for all four approaches to handle different numbers of sequences was vectorization and using a maximum number of sequences that can be concatenated. For attention, we already had a mask to address the padding. Extending this to multiple sequences was straightforward as can be seen in the following TensorFlow pseudo code. The concept is that we made sure that attention is limited to the separate sequences and cannot extend beyond that.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/77b7bf432a0644566575a7a31cfec6eb/href">https://medium.com/media/77b7bf432a0644566575a7a31cfec6eb/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*ekCYadoMaGGP6-DoDv_1TQ.png" /><figcaption>Figure 5: Example zero-one mask</figcaption></figure><p>For the loss calculation, in principle we unpack the sequences and calculate the separate losses, eventually obtaining the average of the losses over the sequences (instead of packs).</p><p>For the MLM loss, the code looks like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/25253b6aea63d08761802509d7328ad4/href">https://medium.com/media/25253b6aea63d08761802509d7328ad4/href</a></iframe><p>For the NSP loss and the accuracy, the principle is the same. In our public examples, you can find the respective code with our in-house <a href="https://docs.graphcore.ai/projects/popart-user-guide/en/latest/intro.html">PopART framework</a>.</p><h4>Wikipedia overhead and speedup estimate</h4><p>With our modification of BERT, we had two questions:</p><ol><li>How much overhead does it bring with it?</li><li>How much does the overhead depend on the maximum number of sequences that are put together in a pack?</li></ol><p>Since data preparation in BERT can be cumbersome, we used a shortcut and compiled the code for multiple different packing depths and compared the respective (measured) cycles. The results are displayed in Table 4. With <strong>overhead</strong>, we denote the percentage decrease in throughput due to changes to the model to enable packing (such as the masking scheme for attention and the changed loss calculation). The <strong>realized speed-up</strong> is the combination of the speed-up due to packing (the <strong>packing factor</strong>) and the decrease in throughput due to the <strong>overhead</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LHtJrk3AZylZwMjVwWw4iA.png" /><figcaption>Table 4: Estimated speed-up comparison of proposed packing algorithms (SPFHP and NNLSHP) on Wikipedia. Image by author.</figcaption></figure><p>Thanks to the vectorization technique, the overhead is surprisingly small and there is no disadvantage from packing many sequences together.</p><h3>Hyperparameter-adjustments</h3><p>With packing, we are doubling the effective batch size (on average). This means we need to adjust the training hyperparameters. A simple trick is to cut the gradient accumulation count in half to keep the same effective average batch size as before training. By using a benchmark setting with pretrained checkpoints, we can see that the accuracy curves perfectly match.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o_a_FcKc4VLW8UXwDx-Nnw.png" /><figcaption>Figure 6: Comparison of learning curves for packed and unpacked processing with <strong>reduced batch size</strong> for the packed approach. Images by author.</figcaption></figure><p>The accuracy matches: the MLM training loss may be slightly different at the beginning but quickly catches up. This initial difference could come from slight adjustments of the attention layers which might have been biased towards short sequences in the previous training.</p><p>To avoid a slowdown, it sometimes helps to keep the original batch size the same and adjust the hyperparameters to the increased effective batch size (doubled). The main hyperparameters to consider are the beta parameters and the learning rates. One common approach is to double the batch size, which in our case decreased performance. Looking at the statistics of the LAMB optimizer, we could prove that raising the beta parameter to the power of the packing factor corresponds to training multiple batches consecutively to keep momentum and velocity comparable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ANQRnItajIsY9MUZ8NBUAw.png" /><figcaption>Figure 7: Comparison of learning curves for packed and unpacked processing with <strong>heuristics</strong> applied. Images by author.</figcaption></figure><p>Our experiments showed that taking beta to the power of two is a good heuristic. In this scenario, the curves are not expected to match because increasing the batch size usually decreases convergence speed in the sense of samples/epochs until a target accuracy is reached.</p><p>Now the question is if in the practical scenario, do we really get the anticipated speed-up?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HkLXFK0gRu9KPTnXYC82_A.png" /><figcaption>Figure 8: Comparison of learning curves for packed and unpacked processing in the <strong>optimized setup</strong>. Images by author.</figcaption></figure><p>Yes, we do! We gained an additional speedup because we compressed the data transfer.</p><h3>Conclusion</h3><p>Packing sentences together can save compute effort and the environment. This technique can be implemented in any framework including PyTorch and TensorFlow. We obtained a clear 2x speed-up and, along the way, we extended the state of the art in packing algorithms.</p><p>Other applications that we are curious about are genomics and protein folding where similar data distributions can be observed. Vision transformers could also be an interesting area to apply differently sized packed images. Which applications do you think would work well? We would love to hear from you!</p><p><a href="https://arxiv.org/abs/2107.02027">Read the Paper</a></p><p><a href="https://github.com/graphcore/tutorials/tree/sdk-release-2.1/blogs_code/packedBERT">Access the Code on GitHub</a></p><h3>Thank you</h3><p>Thanks to our colleagues in Graphcore’s Applications Engineering team, Sheng Fu and Mrinal Iyer, for their contributions to this work and thank you to Douglas Orr from Graphcore’s Research team for his valuable feedback.</p><h3>References</h3><p>[1] M. Kosec, S. Fu, M. M. Krell, <a href="https://arxiv.org/pdf/2107.02027.pdf">Packing: Towards 2x NLP BERT Acceleration</a> (2021), arXiv</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eadb749962b1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/introducing-packed-bert-for-2x-faster-training-in-natural-language-processing-eadb749962b1">Introducing Packed BERT for 2x Training Speed-up in Natural Language Processing</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>