<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by David Bressler on Medium]]></title>
        <description><![CDATA[Stories by David Bressler on Medium]]></description>
        <link>https://medium.com/@dwbressler?source=rss-d543b0e79d50------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*SZvL7tV4dYiEi3tS2Ed0uw.png</url>
            <title>Stories by David Bressler on Medium</title>
            <link>https://medium.com/@dwbressler?source=rss-d543b0e79d50------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 01:35:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dwbressler/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Top 10 Tips for Cutting Costs in ML Systems]]></title>
            <link>https://medium.com/@dwbressler/top-10-tips-for-cutting-costs-in-ml-systems-a72f0755ff78?source=rss-d543b0e79d50------2</link>
            <guid isPermaLink="false">https://medium.com/p/a72f0755ff78</guid>
            <category><![CDATA[cost-optimization]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[David Bressler]]></dc:creator>
            <pubDate>Mon, 24 Mar 2025 05:34:28 GMT</pubDate>
            <atom:updated>2025-03-24T05:34:28.957Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*o_Wr7xZ1FE-hS2EV.jpg" /></figure><p>Building out an ML product often feels like a whirlwind of experiments, training jobs, and quick iterations. Before you know it, you’re juggling multiple GPUs or expensive cloud instances-sometimes running idly. Suddenly, an astronomical bill arrives, pushing cost optimization to the top of your priority list.</p><p>At Eventum, we’ve seen this firsthand. We helped <a href="https://www.eventum.ai/resources/case-studies/sanas-case-study">Sanas</a>, a leading AI startup, optimize their GPU usage, implement modern MLOps practices, and drastically cut infrastructure costs-all without compromising on product innovation. Here we’ve gathered ten practical ways to keep your ML systems lean, efficient, and scalable right from the start.‍</p><h3>1. Leverage Autoscaling &amp; On-Demand Scaling (Including Serverless)</h3><p><strong>Why It Matters:</strong><br>If your infrastructure is set to run 24/7, you’re likely paying for idle resources. By autoscaling, you dynamically match compute power to real-time workloads. And for sporadic workloads or unpredictable inference traffic, serverless or on-demand scaling solutions (e.g., AWS Lambda, Azure Functions, Google Cloud Run) can drop your compute costs to near-zero during idle times.</p><p><strong>Action Steps:</strong></p><ul><li>Configure Kubernetes or cloud autoscalers to spin nodes up/down based on CPU/GPU usage.</li><li>Use serverless options for low-volume or spiky workloads, ensuring you only pay when requests come in.</li><li>Set up cost alerts (more on that later) so you know if usage unexpectedly spikes.</li></ul><h3>2. Harness Spot Instances</h3><p><strong>Why It Matters:</strong><br>Spot instances (AWS Spot, GCP Preemptible VMs, Azure Spot) can be 70–90% cheaper than on-demand. They’re perfect for training jobs or batch tasks that can handle sudden interruptions.</p><p><strong>Action Steps:</strong></p><ul><li>Implement frequent checkpointing so you can resume training if the instance is terminated.</li><li>Combine spot and on-demand instances: keep a small base of on-demand machines and scale up cheaper spot instances for extra capacity.</li><li>Use container orchestration (e.g., Kubernetes with spot-instance node pools) to automatically manage preemptions and re-deploy workloads when a spot instance is lost.</li></ul><h3>3. Right-Size Your Instances</h3><p><strong>Why It Matters:</strong><br>Running GPU-intensive tasks on an overkill instance is costly, but so is using underpowered resources that prolong training. Right-sizing matches hardware capabilities to your actual workload needs.</p><p><strong>Action Steps:</strong></p><ul><li>Monitor resource utilization (CPU, GPU, memory, I/O) during training and inference.</li><li>Experiment with different instance types/sizes to find the best cost-performance ratio.</li><li>If your pipeline is mostly CPU-bound, switch to CPU-optimized instances; if it’s memory-bound, prioritize high-memory machines.</li></ul><h3>4. Optimize Your Models (Distillation, Quantization, Pruning, LoRA)</h3><p><strong>Why It Matters:</strong><br>Bigger isn’t always better. Techniques like <strong>pruning</strong>, <strong>quantization</strong>, <strong>knowledge distillation</strong>, and <strong>LoRA (Low-Rank Adaptation)</strong> can drastically reduce model size, memory usage, and inference time.</p><p><strong>Action Steps:</strong></p><ul><li><strong>Distillation:</strong> Train a “student” model to mimic the outputs of a large “teacher” model for similar accuracy with fewer parameters.</li><li><strong>Quantization:</strong> Convert weights to lower precision (e.g., INT8) for smaller model size and faster inference.</li><li><strong>Pruning:</strong> Remove redundant weights or neurons to create a sparser network with near-identical performance.</li><li><strong>LoRA:</strong> Fine-tune only low-rank parameter matrices for large language models, slashing compute costs on each new task.</li></ul><h3>5. Choose the Right Model-Don’t Always Go Bigger</h3><p><strong>Why It Matters:</strong><br>It’s tempting to pick the latest, biggest model (like a massive language model) even when a simpler architecture might suffice. That decision can blow up your costs and complexity.</p><p><strong>Action Steps:</strong></p><ul><li>Evaluate smaller or more efficient backbone architectures (e.g., MobileNet, EfficientNet, DistilBERT).</li><li>Use transfer learning or pretrained models to avoid training from scratch.</li><li>Start with baseline experiments using modest architectures before “scaling up.” Validate the performance and only then consider more complex (and costly) models.</li></ul><h3>6. Schedule Your Training</h3><p><strong>Why It Matters:</strong><br>Manual or ad-hoc training jobs can run when nobody’s around to watch them-or worse, they can conflict with production workloads. Scheduling ensures you’re using off-peak times (when cloud spot prices might be lower) and also prevents resource contention.</p><p><strong>Action Steps:</strong></p><ul><li>Automate training pipelines with tools like Airflow, Prefect, or Dagster.</li><li>Schedule jobs during off-peak hours to possibly get cheaper spot capacity.</li><li>Avoid running large trainings during critical business hours if they share resources with production or you need immediate debugging support.</li></ul><h3>7. Efficient ETL &amp; Data Processing</h3><p><strong>Why It Matters:</strong><br>Poorly designed data pipelines can become bottlenecks, causing GPUs to sit idle waiting for data or forcing you to over-provision. Streamlining ETL (extract, transform, load) ensures maximum utilization with minimum cost.</p><p><strong>Action Steps:</strong></p><ul><li>Use parallel data loading and caching (e.g., TFRecord, RecordIO, or Parquet).</li><li>Preprocess data once and cache results (e.g., in cloud storage or a data warehouse).</li><li>Profile the end-to-end pipeline to ensure you aren’t limited by slow I/O or disorganized transformations.</li></ul><h3>8. Adopt DevOps for ML (Containerization, CI/CD, Consistent Environments)</h3><p><strong>Why It Matters:</strong><br>Modern ML needs DevOps best practices-often called MLOps-to streamline deployments, reduce manual errors, and foster reproducibility. Containerizing your environment and using continuous integration/continuous deployment (CI/CD) can drastically cut costs from wasted runs, environment inconsistencies, and debugging time.</p><p><strong>Action Steps:</strong></p><ul><li><strong>Containerize</strong> your ML applications (using Docker or similar) so developers and production environments run the same code/libraries.</li><li><strong>Set up CI/CD for ML</strong> to automate testing, linting, and partial training runs before merging code.</li><li><strong>Infrastructure as Code (IaC)</strong> with Terraform or CloudFormation ensures consistent, version-controlled infrastructure for each environment (dev, test, prod).</li></ul><h3>9. Track Experiments &amp; Use PR Environments</h3><p><strong>Why It Matters:</strong><br>If you aren’t documenting your experiments, you risk repeating the same trial-and-error, blowing through compute budget. Also, ephemeral “merge request” environments help you test code and pipelines in isolation before merging to main, preventing expensive mistakes.</p><p><strong>Action Steps:</strong></p><ul><li>Use an experiment tracking tool (Weights &amp; Biases, MLflow) to log hyperparameters, model versions, and metrics.</li><li>Create “PR environments” that spin up automatically whenever a developer opens a pull/merge request. In these temporary testbeds, you can ensure new code integrates cleanly with your data pipelines and training scripts.</li></ul><h3>10. Implement Effective Monitoring &amp; Alerts</h3><p><strong>Why It Matters:</strong><br>Without continuous visibility into resource utilization, cost metrics, and system health, you might discover issues only after you’ve racked up enormous bills. Proactive alerts reduce surprises and let you address inefficiencies quickly.</p><p><strong>Action Steps:</strong></p><ul><li>Leverage cloud billing alerts (AWS Cost Explorer, GCP Billing, Azure Cost Management) and set thresholds for your monthly or daily budgets.</li><li>Monitor CPU/GPU usage, memory, I/O, and run logs in real time (using tools like Prometheus + Grafana, or built-in cloud dashboards).</li><li>Add anomaly detection for unusual spikes in usage or cost, so you can investigate and fix issues immediately.</li></ul><h3>Conclusion</h3><p>Optimizing ML systems for cost-effectiveness, without sacrificing performance, is entirely feasible with a well-planned infrastructure approach and sound MLOps practices. These ten tips-from utilizing spot instances to enabling automated CI/CD-offer proven, concrete ways to rein in expenses. At Eventum, we’ve guided many clients toward significant cost reductions. If you’re looking to apply these optimizations or want an expert review of your current setup, <a href="https://www.eventum.ai/intake-consulting-form">schedule a meeting today</a>.</p><p>‍</p><p><em>Originally published at </em><a href="https://www.eventum.ai/resources/blog/top-10-tips-for-cutting-costs-in-ml-systems"><em>https://www.eventum.ai</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a72f0755ff78" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Use ChatGPT to Format Math for Medium]]></title>
            <link>https://medium.com/@dwbressler/how-to-use-chatgpt-to-format-math-for-medium-1e13f7612e82?source=rss-d543b0e79d50------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e13f7612e82</guid>
            <category><![CDATA[markdown]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[math]]></category>
            <category><![CDATA[medium]]></category>
            <category><![CDATA[blogging]]></category>
            <dc:creator><![CDATA[David Bressler]]></dc:creator>
            <pubDate>Thu, 06 Mar 2025 21:04:42 GMT</pubDate>
            <atom:updated>2025-03-06T21:11:02.512Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jjS8DF4LreJ1U7haSlvuQA.jpeg" /></figure><p>I recently published a math-heavy article on Medium through the Data Science Collective:</p><p><a href="https://medium.com/data-science-collective/the-beautiful-symmetry-of-forward-and-backward-passes-in-residual-neural-networks-51465d6da1a1"><strong>The Elegant Symmetry of Forward and Backward Passes in Residual Networks</strong></a></p><p>I hate to say it took me almost as much time figuring out how to properly format the blog as it did to actually write it! I’m writing this post to hopefully spare anyone else that pain.</p><h3>The Method</h3><p>Follow these simple steps:</p><p>1.<strong> Draft Your Post</strong>: Write a rough draft using your favorite text editor.</p><p>2.<strong> ChatGPT Formatting</strong>: Copy your draft into ChatGPT and use the following exact prompt:</p><p>“Format this blog post as markdown. For inline math use plain text or Unicode characters. For block equations, wrap them in $$ … $$ on separate lines, with no punctuation or Unicode characters inside those LaTeX expressions. Subscripts in LaTeX should be in standard LaTeX form (e.g., x_{l}, x_{l+1}). Avoid trailing periods or other punctuation inside the $$ … $$ blocks.”</p><p>Make further edits in ChatGPT as necessary.</p><p>3.<strong> Save as Markdown</strong>: Save ChatGPT’s output as blog.md.</p><p>4.<strong> Convert Using Pandoc</strong>: Install Pandoc and run the following command in your terminal:</p><pre>pandoc blog.md --webtex=https://latex.codecogs.com/png.latex? -o blog.html</pre><p><strong>5. Preview and Transfer</strong>: Open the newly created blog.html file in your web browser.</p><p><strong>6. Post to Medium</strong>: Copy and paste the entire content directly into Medium’s editor. Make any final edits directly on Medium.</p><p>That’s it — you’re done!</p><h3>What Didn’t Work</h3><p>Initially, I assumed that if I formatted my post with standard Markdown and LaTeX, so that it rendered properly in e.g. HackMD, it would allow for an easy copy-and-paste into Medium. Unfortunately, Medium does not support LaTeX directly.</p><p>My next approach was to download the file from HackMD as html, which does render properly when loaded locally. I was hoping that I could import the html with Medium’s “Import a story” tool. Unfortunately this approach proved to be the most time-wasting. Medium would not import the html from a local file, nor from hosted solutions including Netlify Drop, GitHub Pages, Ngrok, and even my own hosted site. Despite each method providing functional HTML, Medium consistently refused to import it — likely due to inherent conflicts with LaTeX formatting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*CmEC3cFiPiQAvTL_ImfltA.png" /></figure><p>The final solution outlined above took multiple trial-and-error iterations. ChatGPT initially made mistakes, such as inserting Unicode characters within the LaTeX blocks, but the prompt I’ve provided above reliably prevents these errors.</p><p>It’s also unfortunate that even with this approach the inline math needs to be converted to Unicode. Hopefully, Medium will address this limitation in the future!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e13f7612e82" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Elegant Symmetry of Forward and Backward Passes in Residual Networks]]></title>
            <link>https://medium.com/data-science-collective/the-beautiful-symmetry-of-forward-and-backward-passes-in-residual-neural-networks-51465d6da1a1?source=rss-d543b0e79d50------2</link>
            <guid isPermaLink="false">https://medium.com/p/51465d6da1a1</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[David Bressler]]></dc:creator>
            <pubDate>Tue, 25 Feb 2025 03:36:15 GMT</pubDate>
            <atom:updated>2025-02-25T17:50:01.716Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tyE5rLOdjeaP6tA8YWjWCQ.jpeg" /></figure><p>This post describes a delightful ‘aha’ moment I experienced, when I figured out that the backward pass of information in a residual neural network is actually just as simple as it’s forward pass.</p><p>As a reminder, most modern architectures (most notably <a href="https://www.eventum.ai/resources/blog/three-breakthroughs-that-shaped-the-modern-transformer-architecture">Transformers</a> used for LLMs) make use of a simple ‘residual’ trick:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/134/0*KrrbRyzAMlxOSSQx" /></figure><p>On the forward pass, each “residual block” f<em>ₗ</em> adds its output to the existing input x<em>ₗ. </em>This assures that the activation reaching the final layer, is actually just a summation of the outputs from all the layers (plus the original input). What took me much longer to realize, is that the gradient that is passed backward and reaches the first layer, is actually just a summation of each layer’s gradient contribution (plus the original gradient).</p><p>This blog post demonstrates the <strong>beautiful symmetry</strong> at play: the forward pass and backward pass each follow a nearly identical, three-step routine at every layer. And since we’ve all learned backpropagation as a method grounded in differential calculus, I’ll also show how the simple accumulation I describe here aligns perfectly with the formal chain rule used in backpropagation.</p><h3>A Simple Three-Step Algorithm for Forward and Backward Passes</h3><p>Below, I illustrate how, in residual architectures, the forward pass and backward pass each follow a nearly identical three-step routine at every layer.</p><h3>Forward Pass</h3><ol><li>The residual stream activation x<em>ₗ</em> enters the sublayer.</li><li>The sublayer processes this input, producing f<em>ₗ</em>(x<em>ₗ</em>).</li><li>The output is added back to the residual stream:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/134/0*7zyePZo9aF1OxxBb" /></figure><h3>Backward Pass</h3><ol><li>The residual stream gradient g<em>ₗ</em>₊₁ flows into the sublayer.</li><li>The sublayer applies its local derivative, multiplying g<em>ₗ</em>₊₁ by the sublayer’s gradient (g_local).</li><li>This result is then added back to the residual stream:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/192/0*WEqOtqgZ1I7khd2J" /></figure><h3>Two-Layer Illustration</h3><p>To demonstrate this symmetry in more detail, consider a simple two-layer network.</p><h4>Forward Pass Steps</h4><p>1. <strong>Start</strong> with an input x<em>₁</em>.</p><p>2.<strong> Layer 1</strong> adds its function f₁ to x₁:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/127/0*sOMvopoO5IlmyUcv" /></figure><p>3.<strong> Layer 2</strong> adds its function f₂ to x₂:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/127/0*S7uc2fKFYh1xFq0G" /></figure><p>4. The final output is o = x₃.</p><h3>Backward Pass Steps</h3><p>1. <strong>Start</strong> with ∂L/∂x₃ at the network output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/73/0*TBHHHhfpHv0yqW3a" /></figure><p>2.<strong> Layer 2</strong> transforms ∂L/∂x₃ by ∂f₂/∂x₂, then <strong>adds </strong>this sublayer output to the residual stream:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/178/0*Rpp0JCNB9LWR8R78" /></figure><p>or, more precisely:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/182/0*gx4reTs6ABhUI6sE" /></figure><p>3.<strong> Layer 1</strong> takes that gradient, transforms it by ∂f₁/∂x₁, and <strong>again </strong>adds it to the residual:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/159/0*Xxsmh79G4CN1goD5" /></figure><p>or, more precisely:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/397/0*1Cg6cMLX5K8CJTXu" /></figure><p>In expanded form:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/0*et6sWLO52w06w1fl" /></figure><p>Notice how the backward pass “accumulates” gradients in the same additive way the forward pass accumulates activations.</p><h3>Using the Chain Rule</h3><p>If we derive ∂L/∂x₁ via the chain rule, we get the same result. Defining</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/357/0*rih7UB4fmiGuTtBp" /></figure><p>the chain rule expansions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/84/0*6JO6TZ_ymEzADFbz" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/305/0*QslHkwBeUjzN57Kc" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/305/0*UJlACRCZw0lRRSM2" /></figure><p>Combining the terms:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/259/0*Y4qqERH1vvJ2usxe" /></figure><p>When fully multiplied out, this expansion <strong>exactly</strong> matches the step-by-step gradient accumulation from above:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/0*HWKn4pOewDZ1I89x" /></figure><h3>Conclusion</h3><p>Residual connections ensure robust gradient flow throughout the network, helping to avoid vanishing or exploding gradients and making it easier to train very deep modern architectures like <a href="https://www.eventum.ai/resources/blog/three-breakthroughs-that-shaped-the-modern-transformer-architecture">Transformers</a>. Understanding that residual gradients flow via simple additions — mirroring the forward pass additions — demystifies backpropagation and unveils the <strong>beautiful symmetry</strong> at play.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=51465d6da1a1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/the-beautiful-symmetry-of-forward-and-backward-passes-in-residual-neural-networks-51465d6da1a1">The Elegant Symmetry of Forward and Backward Passes in Residual Networks</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Transformers Evolved: The Three Breakthroughs That Made Transformers Scalable and Powerful]]></title>
            <link>https://medium.com/data-science-collective/three-breakthroughs-that-shaped-the-modern-transformer-architecture-2313abfe47ad?source=rss-d543b0e79d50------2</link>
            <guid isPermaLink="false">https://medium.com/p/2313abfe47ad</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[transformers]]></category>
            <dc:creator><![CDATA[David Bressler]]></dc:creator>
            <pubDate>Tue, 18 Feb 2025 19:16:32 GMT</pubDate>
            <atom:updated>2025-02-25T17:48:54.966Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hjCKZhUecrBItFws4RX0aQ.jpeg" /></figure><p>Perhaps the most surprising aspect of the evolution of the Transformer architecture for language modeling is how consistent it has remained over the past near-decade. The most cutting-edge OpenAI or DeepSeek model would still look quite familiar to an NLP engineer from 2017, when the seminal paper “Attention Is All You Need” introduced the core design [Vaswani et al., 2017]. Nevertheless, a few refinements to the Transformer architecture have proven to be truly significant, improving model quality, stability, and efficiency. This blog covers three of the most important of those architectural improvements.</p><p><strong>Scope Clarification<br></strong> This post focuses on advancements to the Transformer’s architecture that specifically enhance model performance. Many other innovations exist that primarily focus on efficiency or memory reduction-sometimes at the cost of accuracy-and I will not delve into those here. Examples of such efficiency-focused improvements include the following:</p><ul><li><strong>Reformer</strong> [Kitaev et al., 2020] (uses locality-sensitive hashing for attention)</li><li><strong>Linformer</strong> [Wang et al., 2020] (linearizing attention)</li><li><strong>Performer</strong> [Choromanski et al., 2021] (random feature maps for linear attention)</li><li><strong>Big Bird</strong> [Zaheer et al., 2020] (sparse attention over large contexts)</li></ul><p>I also do not discuss training procedure innovations, including the important step of <em>alignment</em>, which typically involves reinforcement learning from human feedback (RLHF) [Stiennon et al., 2020; Ouyang et al., 2022], as well as other training advances such as large-batch training [You et al., 2019], mixed-precision training [Micikevicius et al., 2018], or parameter-efficient fine-tuning methods like LoRA [Hu et al., 2022]. These training-related improvements are critically important for building today’s massive language models, but they lie outside the scope of the architecture-centered focus here.</p><h3>A QUICK REVIEW OF TRANSFORMERS</h3><p>The Transformer architecture introduced by Vaswani et al. [2017] radically changed the landscape of NLP by discarding recurrence and convolutions in favor of <em>self-attention</em>. At a high level, a Transformer for language modeling has these key components (see Figure from paper):</p><ol><li><strong>Embedding Layer<br></strong>Each word unit in the input sequence is converted to a continuous vector representation.</li><li><strong>Positional Encoding<br></strong>The original Transformer used a sinusoidal function to encode position information, which is then added to the input embeddings. This positional signal allows the self-attention mechanism to differentiate between tokens at different positions.</li><li><strong>Multi-Head Self-Attention<br></strong>Self-attention gathers and combines information across all positions in the sequence. The token at each position produces its own set of Queries, Keys, and Values vectors. For each token’s Query, attention weights are computed via a dot product with Keys from all tokens, then normalized (via softmax) and used to weight the Values vectors before summation. Multi-head attention means the process is repeated multiple times in parallel with different learned projections, allowing the model to attend to multiple representation subspaces.</li><li><strong>Feed-Forward Network (FFN)<br></strong>After self-attention, the position-wise feed-forward network applies the same MLP transformation to each token individually (i.e., independently of other positions), often with two linear layers with a nonlinearity in between (such as GELU). This allows each token to refine its own features in isolation before the model once again combines information across positions.</li><li><strong>Residual Connections &amp; Layer Normalization<br></strong>Each sublayer (attention or FFN) is wrapped in a residual connection, plus a layer normalization (LN). In the original (“post-norm”) Transformer, layer normalization is applied after the residual sum.</li><li><strong>Stacked Layers<br></strong>The above (multi-head attention + FFN + normalization) block is repeated <em>L</em> times, forming a deep network. The final output can then be projected to vocabulary space for next-token prediction (language modeling) or used for classification, etc.</li></ol><p>Despite its simplicity, the Transformer’s self-attention mechanism enables learning long-range dependencies efficiently across sequences, and the architecture scales well when combined with large amounts of data and distributed training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/227/0*xPizLQOPJQRNZb6p.png" /><figcaption>Vaswani et al., 2017</figcaption></figure><h3>1) MOVING NORMALIZATION OUTSIDE THE RESIDUAL STREAM (“PRE-NORM”)‍</h3><h4>Background: Layer Normalization</h4><p>Layer normalization (LN) [Ba et al., 2016] normalizes the features of each sample so that they have zero mean and unit variance. In Transformers, LN helps stabilize training by controlling the distribution of activations inside each layer.</p><h4>Post-Norm vs. Pre-Norm</h4><p>The original Transformer applied LN in a “post-norm” fashion [Vaswani et al., 2017]: after each sublayer’s output was added to the residual stream, the <em>combined</em> sum was normalized. Formally (for a sublayer):</p><p>y = sublayer(x)</p><p>z = x + y</p><p>o = LN(z)</p><p>x = o‍</p><p>Subsequent research [Wang et al., 2019; Xiong et al., 2020] showed that switching to “pre-norm” (i.e., applying LN <em>before</em> the sublayer) yields better training stability and allows scaling to deeper networks. In “pre-norm,” the sublayer sees a normalized input, while the residual stream itself remains an unnormalized sum of all outputs (see Figure from Xiong et al., 2020):</p><p>x’ = LN(x)</p><p>y = sublayer(x’)</p><p>o = x + y</p><p>x = o</p><h4>Why Does Pre-Norm Help?</h4><p>By keeping the residual pathway <em>clean</em> (just a direct sum across layers), gradients can backpropagate more directly to earlier layers. In post-norm, the gradient to earlier layers must pass through a normalization of (x + y). Because LN’s derivative is not simply an identity function, it can dampen or amplify signals in ways that are more unpredictable over many layers. Pre-norm avoids that complication for the residual stream, improving gradient flow and training stability-especially critical for very deep Transformers.</p><p>Moreover, in pre-norm:</p><ul><li>Each sublayer consistently sees a well-conditioned input (via LN).</li><li>The residual sum can grow in magnitude, but that is not problematic because each next sublayer again normalizes its input before processing.</li><li>Many implementations also add small scaling factors to the residual connections to keep forward and backward signals in a stable range.</li></ul><p>As a result, most modern Transformer-based architectures (e.g., GPT-3, PaLM, LLaMA) adopt pre-norm by default [Brown et al., 2020; Chowdhery et al., 2022; Touvron et al., 2023].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/0*rZrY2uFMOh7Xy3Bl.png" /><figcaption>Xiong et al., 2020</figcaption></figure><h3>‍2) ROTARY POSITIONAL ENCODINGS (RoPE)</h3><h4>‍Why Positional Embeddings?</h4><p>Self-attention is inherently order-invariant if all tokens only differ by their embeddings. Positional encodings or embeddings break this symmetry by informing the model about where each token occurs within a sequence.</p><p>Originally, Transformers used <em>absolute</em> position embeddings [Vaswani et al., 2017], adding a sinusoidal or learned vector to each token’s input. Later, <em>relative</em> position embeddings [Shaw et al., 2018] were proposed, encoding the <em>difference</em> between query and key positions within the attention mechanism.</p><h4>From Relative Position Embeddings to RoPE</h4><p>Rotary Positional Encodings (RoPE) [Su et al., 2021/2023] extend relative position embeddings by applying a rotation operation to the Query and Key vectors that encodes their relative positions. Conceptually:</p><p><strong>1. Pair the Dimensions<br></strong>For a given token’s Query or Key vector of dimension d, split it into d/2 pairs. The pairs are indexed by ‘i’ (i.e. ‘i’ ranges from 1 to d/2).</p><p><strong>2. Rotate Each Pair<br></strong>Each pair (u, v) is treated like a 2D point or a complex number with magnitude sqrt(u² + v²)​. The pair is rotated by an angle θ via:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/412/0*_1cEk3K2SpRI1kuv.png" /></figure><p><strong>3. Position-Dependent Frequencies<br></strong>The angle θ is chosen based on the pair index ‘i’, and the token position ‘p’. Each pair index ‘i’ has a different “frequency” that determines <em>how fast</em> the 2D point is rotated, whereas each token position ‘p’ determines <em>how much</em> the 2D point is rotated. By rotating each pair of each vector by a reliable amount depending on position and pair index, this effectively encodes relative distances between positions in the phase differences across multiple frequencies.</p><p><strong>4. Compute Attention<br></strong>Once each pair of a given Query or Key has been rotated, the pairs are recombined back into a single vector of dimension d, and the standard dot-product attention is computed between the Queries and the Keys. The angles align in a way that the dot product depends on relative position rather than solely on absolute positions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/0*OBLCnIJlGOWpxow-.png" /><figcaption>Su et al., 2023</figcaption></figure><h4>Benefits</h4><ul><li><strong>Smooth Relative Encoding</strong>: RoPE encodes relative position information continuously, without needing a huge table of embeddings for all possible offsets.</li><li><strong>Multi-Scale Awareness</strong>: The chosen frequencies for the pair indices are usually spaced exponentially, which captures both local and long-distance positional relationships.</li><li><strong>Easy Extension to Long Contexts</strong>: With RoPE, you can extend context length by adjusting the rotation range, without having to learn new position embeddings.</li><li><strong>Simplicity in Implementation</strong>: Only a rotation of Qs and Ks is required.</li></ul><p>As a result, RoPE (and related approaches) has been adopted in many state-of-the-art LLMs that target long context capabilities, such as LLaMA [Touvron et al., 2023], DeepSeek, and several others.</p><h3>‍3) MIXTURE OF EXPERTS (MoE)‍</h3><h4>MoE in a Nutshell</h4><p>A Mixture of Experts (MoE) layer replaces the standard feed-forward sublayer of the Transformer with multiple “expert” sub-networks. Each expert is itself typically a feed-forward network with the same shape as the original FFN. A <em>gate</em> decides which expert(s) each token will route to, usually based on the dot product between the token representation and some gating parameters [Shazeer et al., 2017; Lepikhin et al., 2021; Fedus et al., 2022].</p><h4>How It Works</h4><ol><li><strong>Token-Level Routing</strong>: For each token, compute the dot product with each expert’s “centroid” vector to generate affinity scores.</li><li><strong>Top-K Selection</strong>: Choose the top-K experts (often K=1 or 2) according to the largest affinity scores.</li><li><strong>Expert Processing</strong>: Each selected expert is a separate FFN that transforms the token independently.</li><li><strong>Combine Outputs</strong>: If K&gt;1, combine expert outputs (often via a weighted sum using the gating scores).</li></ol><p>By routing tokens to only a fraction of the total experts, MoE can dramatically increase the model’s overall parameter count (capacity) <em>without</em> increasing the per-token compute cost by the same factor. It also allows the model to learn more specialized sub-networks that focus on certain token patterns or language phenomena.</p><h4>DeepSeekMoE (2024) Advances</h4><p>In “DeepSeekMoE” [Dai et al., 2024], the number of experts per MoE layer is increased, while the size of each individual expert is reduced proportionally. Key additional ideas:</p><ul><li><strong>Fine-Grained Expert Segmentation</strong>: Many smaller experts rather than fewer large ones.</li><li><strong>Higher K</strong>: More experts can be selected simultaneously to leverage more specialized transformations.</li><li><strong>Shared Expert Isolation</strong>: A subset of “shared” experts is used for all tokens, handling broadly useful knowledge, while the rest can specialize in narrower linguistic or domain-specific patterns.</li></ul><p>This “fine-grained plus shared” approach maintains constant total compute while improving the potential for specialization. Each token picks from a broader pool of specialized experts, and the shared experts capture universal functions in a single place.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/809/0*VGf_8yywRYmxHgFz.png" /><figcaption>Dai et al., 2024</figcaption></figure><h3>CONCLUSION</h3><p>From the original post-norm Transformer to today’s cutting-edge language models, most of the <em>core</em> architecture of Transformers has remained intact: self-attention, residual connections, layer norms, and feed-forward layers. However, three architectural refinements have proven particularly impactful:‍</p><ol><li><strong>Pre-Norm Layer Normalization</strong> — Simplifies gradient flow and stabilizes deeper models.</li><li><strong>Rotary Positional Encodings</strong> — Elegantly incorporate relative positional awareness at multiple scales.</li><li><strong>Mixture of Experts</strong> — Increases model capacity and specialization without linear increases in compute.‍</li></ol><p>These innovations, combined with large-scale data, clever training strategies, and efficiency improvements, have powered the remarkable performance leaps in large language models.</p><p>‍</p><p><em>Originally published at </em><a href="https://www.eventum.ai/resources/blog/three-breakthroughs-that-shaped-the-modern-transformer-architecture"><em>https://www.eventum.ai</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2313abfe47ad" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/three-breakthroughs-that-shaped-the-modern-transformer-architecture-2313abfe47ad">How Transformers Evolved: The Three Breakthroughs That Made Transformers Scalable and Powerful</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a convolutional neural network for natural language processing]]></title>
            <link>https://medium.com/data-science/how-to-build-a-gated-convolutional-neural-network-gcnn-for-natural-language-processing-nlp-5ba3ee730bfb?source=rss-d543b0e79d50------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ba3ee730bfb</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[David Bressler]]></dc:creator>
            <pubDate>Tue, 11 Dec 2018 06:19:56 GMT</pubDate>
            <atom:updated>2019-04-01T07:07:02.726Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to build a gated convolutional neural network (GCNN) from scratch, with Pytorch implementation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gl7hmFe-bq6OyDNY" /></figure><p>Recurrent neural networks (RNNs) with LSTM or GRU units are the go-to tools for NLP researchers, and provide state-of-the-art results on many different NLP tasks, including language modeling (LM), neural machine translation (NMT), sentiment analysis, and so on. However, a major drawback of RNNs is that they are notoriously slow to train, and so much of the effort over the past few years has been focused on trying to speed them up. There have been a variety of approaches to do this, including <a href="https://thegradient.pub/nlp-imagenet/">using pretrained models</a>, using a <a href="https://towardsdatascience.com/speed-up-your-deep-learning-language-model-up-to-1000-with-the-adaptive-softmax-part-1-e7cc1f89fcc9">faster softmax</a>, and using different architectures such as convolutional neural networks (CNNs), quasi-recurrent neural networks (<a href="https://arxiv.org/abs/1611.01576">QRNNs</a>), or transformer architectures. In this blog post, I’ll tell you about recent research from <a href="https://research.fb.com/category/facebook-ai-research/">Facebook’s AI Research (FAIR) group</a>, who for the first time ever are able to get state-of-the-art results on a language modeling task using a CNN. I’ll explain their approach, and then walk you step by step through a <a href="https://github.com/DavidWBressler/GCNN">Pytorch implementation</a> I wrote.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*ojEw8m2SUS0YvdDafduTgQ.png" /><figcaption>RNNs are inherently sequential. Image from <a href="https://vimeo.com/238222385]">https://vimeo.com/238222385</a></figcaption></figure><p>As you probably know, language modeling involves predicting the next word in a sequence, when given a sequence of words as an input. A major reason why RNNs are slow is that each word in that input string must be processed sequentially: in the sentence “Parallelization is fun”, the words ‘parallelization’ and ‘is’ must be processed before the word ‘fun’. In contrast, all the elements in a CNN are processed simultaneously, which can speed up processing immensely. Previous attempts at using a CNN for language modeling significantly underperform results from RNNs, but in a recent paper called <a href="https://arxiv.org/abs/1612.08083">“Language Modeling with Gated Convolutional Networks”</a>, Dauphin et al. (2016) build a convolutional language model that produces results competitive with RNNs, and can drastically cut down computational cost.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/549/1*jnwJCAM5qZPARX3k_sKQiw.png" /><figcaption>CNNs can process information in parallel. Image from <a href="https://vimeo.com/238222385">https://vimeo.com/238222385</a></figcaption></figure><p>A quick reminder of how information flows through a RNN language model will be helpful. The input to the model is a sequence of words represented by word embeddings <em>X </em>of size [<em>seq_length, emb_sz</em>], where <em>seq_length</em> is the sequence length, and <em>emb_sz </em>is the dimensionality of your embeddings. After <em>X</em> is passed through multiple LSTM layers, the output of the final layer is a hidden state representation <em>H</em> of size [<em>seq_length, c_out</em>], where <em>c_out </em>is the dimensionality of that final layer. Then, a softmax step produces predictions over the entire vocabulary, for each step in the sequence.</p><p>Dauphin et al.’s CNN similarly takes embedding activations of size [<em>seq_length, emb_sz</em>] as input, but then uses multiple layers of gated convolutions to produce the final hidden state output <em>H</em> (also of size [<em>seq_length, c_out</em>]). Each layer is composed of 1) a convolutional block that produces two separate convolutional outputs, and 2) a gating block that uses one convolutional output to gate the other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/381/1*sdkRXexo4EuizrNa_DlXNQ.png" /><figcaption>Figure 1 from Dauphin, et al. (2016), showing GCNN architecture.</figcaption></figure><p>The convolutional block performs “causal convolutions” on the input (which for the first layer will be size [<em>seq_length, emb_sz</em>]). Whereas a normal convolution has a window of width <em>k</em> that is centered on the current timestep (and therefore includes inputs from both future and past timesteps), a causal convolution has a window that overlaps only the current and previous timesteps (see figure below). This is accomplished by simply computing a normal convolution on an input that has been zero-padded with <em>k</em>-1 elements on the left. Causal convolutions are necessary because it would be cheating if the CNN was able to “see” information from the future timesteps that it is trying to predict.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*t-PnRtIFL8oKaQqnc1HVPw.png" /><figcaption>TOP: Normal convolution. The convolutional window (width k=3) is centered on the current timestep. BOTTOM: Causal convolution. The input is zero-padded on the left with k-1=2 elements.</figcaption></figure><p>As mentioned, the convolutional block will actually produce two separate convolutional outputs, <em>A</em> and <em>B</em>, one of which will gate the other through element-wise multiplication. The first convolutional output is computed as <em>A</em>= <em>X*W+b</em>. Since <em>X</em> is a tensor of size [<em>seq_length, emb_sz</em>], after left-padding with zeroes it will be of size [<em>seq_length+k-1, emb_sz</em>]. W is a convolutional filter of size [<em>k, in_c, out_c</em>], where <em>in_c </em>is the number of input channels (here equal to <em>emb_sz </em>for the first layer), and <em>out_c </em>is the number of output channels. <em>b</em> is a bias vector of length <em>out_c</em>. After applying the convolutions, the output<em> A</em> will be size [<em>seq_length, out_c</em>]. Finally, you do the same steps with <em>B= X*V+c</em>… i.e. produce a second output <em>B</em> using a different convolutional filter <em>V </em>and bias vector<em> c</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/246/1*zPaT0Z5fnF8sO4YBb3q5eA.png" /><figcaption>Gated Linear Unit (GLU), with residual skip connection. A convolutional block with window k=3 produces two convolutional outputs, A and B. A is element-wise multiplied with sigmoid(B), and the residual is added to the output. Image modified from <a href="https://vimeo.com/238222385">https://vimeo.com/238222385</a></figcaption></figure><p>The two outputs of the convolutional block, <em>A</em> and <em>B</em>, are then passed to the gating block. At this point its helpful to be reminded of what happens in CNNs for image processing… the convolutional output A would likely be put through an activation mechanism such as ReLU: <em>ReLU(A)</em>, a.k.a. <em>ReLU(X*W+b)</em>. For the GCNN’s gating block however, Dauphin et al. use a mechanism they call a “gated linear unit” (GLU), which involves element-wise multiplying <em>A</em> by <em>sigmoid(B</em>):</p><p><em>A </em>⊗<em> sigmoid(B)</em></p><p>or equivalently,</p><p><em>(X*W+b) </em>⊗<em> sigmoid(X*V+c)</em></p><p>Here, <em>B</em> contains the ‘gates’ that control what information from A is passed up to the next layer in the hierarchy. Conceptually, the gating mechanism is important because it allows selection of the words or features that are important for predicting the next word, and provides a mechanism to learn and pass along just the relevant info. Similar to ReLU, the gating mechanism also provides the layer with non-linear capabilities, while providing a linear path for the gradient during backpropagation (thereby diminishing the vanishing gradient problem).</p><p>Dauphin et al. also add a residual skip connection to each layer (see figure above), similar to the <a href="https://arxiv.org/abs/1512.03385">ResNet architecture</a>. Residual connections minimize the vanishing gradient problem, allowing networks to be <a href="https://www.coursera.org/learn/convolutional-neural-networks/lecture/XAKNO/why-resnets-work">built much deeper</a> (more layers). The input to the layer (<em>X</em>) is added to convolutional output <em>A</em>, after it is gated by convolutional output <em>B.</em></p><p><em>X + (A </em>⊗<em> sigmoid(B) )</em></p><p>or equivalently,</p><p><em>X + ( (X*W+b) </em>⊗<em> sigmoid(X*V+c) )</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/206/1*JR3JP4kDQdFmXzQdKIttAw.png" /><figcaption>Bottleneck structure of a visual processing ResNet layer [Figure 5 from He, et al. (2016)]. A k=1 convolution reduces dimensionality from 256 to 64, then a k=3 convolution is performed in the lower dimensional space, then a k=1 convolution increases the dimensionality back to 256.</figcaption></figure><p>Dauphin et al. sometimes use a bottleneck structure within a layer, also similar to the <a href="https://arxiv.org/abs/1512.03385">ResNet architecture</a>. The purpose of a bottleneck structure is to reduce the computational cost of a convolutional operation by first reducing the dimensionality of the data. Suppose you’d like to perform a <em>k</em>=4 convolution on an input of size [<em>seq_length</em>=70, <em>in_c</em>=600], with 600 output channels. This would normally require a convolutional weights matrix of size [4, 600, 600]. In contrast, a bottleneck layer first reduces the dimensionality of the data to e.g. [70, 30], then performs the <em>k</em>=4 convolution in the lower-dimensional space, then finally increase the dimensionality of the data back to [70,600]. This approach requires a convolutional weights matrix size of only [4, 30, 30], which is much more computationally efficient. The bottleneck increases and decreases dimensionality through the use of k=1 convolutions (see Pytorch implementation below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/280/1*FLMq9sk8fYQcxliV_PuOAQ.png" /><figcaption>Stacking multiple layers, and left-padding the output of each layer. Image from <a href="https://vimeo.com/238222385">https://vimeo.com/238222385</a></figcaption></figure><p>Regardless of whether a bottleneck structure is used within a layer, the GCNN layers can be stacked, making sure at every layer to left-pad the outputs of the previous layer. The input to each layer will be size [<em>seq_length+k-</em>1<em>, in_c</em>] and the output of each layer will be size [<em>seq_length, out_c</em>]. Dauphin et al. stack at least 5 of these layers on top each other, to produce the final hidden state output H. They then use the adaptive softmax to choose which word to select, from a huge list of words in the language model’s vocabulary (check out my <a href="https://towardsdatascience.com/speed-up-your-deep-learning-language-model-up-to-1000-with-the-adaptive-softmax-part-1-e7cc1f89fcc9">blog post</a> on how to speed up the time-consuming softmax step by up to 1000%).</p><h3>Pytorch implementation</h3><p>After some searching, I was not able to find a working Pytorch implementation of the paper, so I decided to build one from scratch. Check out the accompanying <a href="https://github.com/DavidWBressler/GCNN/blob/master/GCNN.ipynb">Jupyter notebook</a> to follow along. For preprocessing you will need <a href="https://docs.fast.ai/">fastai</a>, a deep learning library that runs on top of Pytorch that simplifies training neural networks. [For those who want to learn state-of-the-art deep learning techniques, I highly recommend Jeremy Howard’s <a href="https://course.fast.ai/">fast.ai course</a>, which is available online for free]. I decided to use the <a href="https://einstein.ai/research/blog/the-wikitext-long-term-dependency-language-modeling-dataset">Wikitext-2 dataset</a>, which is a relatively small dataset that contains ~2 million tokens and ~33 thousand words in the vocabulary, and is a subset of the larger Wikitext-103 dataset used in the paper.</p><p>As in the paper, I divide the data into paragraphs. For the sake of having a somewhat standardized sequence length, I remove paragraphs that have less than 10 or more than 300 words. Once the data has been properly formatted in csv files, fastai makes it easy to quickly tokenize and numericalize it. I created customized dataset, sampler, and collating functions, which properly format the data into data loaders. Each minibatch randomly samples paragraphs of approximately the same sequence length. I also downloaded <a href="https://nlp.stanford.edu/projects/glove/">GloVe pre-trained word vectors</a> that are used for the models’ word embeddings. Finally, I created a modeler class that handles training. During training, gradient clipping allows the use of a large learning rate (lr=1), which along with Nesterov’s momentum and weight normalization helps training to converge quickly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*ieBv2K1vcokaB59lKS6ubA.png" /><figcaption>The first few layers of the architecture.</figcaption></figure><p>As mentioned above, the sequence length differs between minibatches, but just assume for the rest of this post that seq_length equals 70. After the embeddings and resizing, the input x to the first layer is size [<em>bs=50, emb_sz=</em>300<em>, seq_length=</em>70, 1], where <em>bs </em>is batch size, and <em>emb_sz </em>is the embedding dimensionality. I use a convolutional kernel size <em>k</em> of 4, so after left-padding the input with <em>k</em>-1=3 elements, the input size is [50, 300, 73, 1].</p><p>I use a bottleneck structure in each layer of my architecture. In the first layer, (called with <em>self.inlayer = GLUblock(k, emb_sz, nh, downbot)</em> ), I produce two convolutional outputs, <em>x1</em> and <em>x2</em>. To produce <em>x1</em>, the first k=1 convolution decreases the dimensionality of input <em>x</em> from [50, 300, 73, 1] to [50, 15, 73, 1]. Then the <em>k</em>=4 convolution is performed in this lower dimensional space, to produce output of size [50, 15, 70, 1]. Finally, a <em>k</em>=1 convolution increases the dimensionality of <em>x1</em> to size [50, 600, 70, 1]. The same three steps are used to produce <em>x2</em>, which is then passed through a sigmoid and used to gate <em>x1</em> through element-wise multiplication, to produce output <em>x </em>(also of size [50, 600, 70, 1]). The <em>residual</em> (which was set equal to <em>x</em> at the beginning of the block), is transformed from size [50, 300, 70, 1] to size [50, 600, 70, 1] through the use of a <em>k</em>=1 convolution, and added to the output <em>x</em>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/456ffbee01362c291eb135fb23acf481/href">https://medium.com/media/456ffbee01362c291eb135fb23acf481/href</a></iframe><p>I next pass the data x through 4 gated convolutional layers. Each layer has an input of size [50, 600, 70, 1], and produces an output of the same size. Since the dimensionality of the input and the output is the same (i.e. <em>in_c==out_c</em>), the <em>residual</em> can simply be copied and set aside until the end of the block. As described above, the input is left-padded to be size [50, 600, 73, 1], bottleneck convolutional outputs <em>x1 </em>and <em>x2 </em>are produced (both of size [50, 600, 70, 1]), and <em>x1</em> is gated by <em>sigmoid(x2) </em>before the <em>residual</em> is added to produce the layer’s output. After the 4 convolutional layers, the final output is then passed through the adaptive softmax to produce the model’s predictions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*EThN-Lbqzn3L5Lvv5nRXfw.png" /><figcaption>Perplexity=76.6 on epoch 49.</figcaption></figure><p>I ran this network on the Wikitext-2 dataset for 50 epochs, which produced a minimum loss of 4.34 (perplexity=76.6) on epoch 49. This compares pretty well with the results from Dauphin et al., who achieve a final perplexity of 37.2 on the larger Wikitext-103 dataset. My model’s performance would likely be closer to that found in the paper by increasing the model capacity (more layers, more hidden units per layer), by training for much longer, by using a less aggressive dimensionality decrease in the bottleneck layers, and by using a larger dataset (e.g. Wikitext-103).</p><p><a href="https://vimeo.com/238222385">Language Modeling with Gated Convolutional Networks --- Yann Dauphin, Angela Fan, Michael Auli, David Grangier</a></p><p>Dauphin et al.’s results are competitive those of state-of-the-art models, either nearly matching (Google Billion Word dataset) or surpassing (Wikitext-103) the best published results from LSTM-based models. Moreover, their approach greatly reduces computational cost. Their use of the <a href="https://towardsdatascience.com/speed-up-your-deep-learning-language-model-up-to-1000-with-the-adaptive-softmax-part-1-e7cc1f89fcc9">adaptive softmax</a> speeds training by ~500% compared to LSTM models with the full softmax, so that it requires significantly fewer operations to reach a given perplexity value. Since the convolutional approach is parallelizable over both sequences and the tokens within a sequence, the GCNN model reduces the latency to process a single sentence by an order of magnitude*.</p><p>Dauphin et al.’s paper is one of many recent publications that completely abandon RNNs for NLP tasks, relying instead on either purely <a href="https://arxiv.org/abs/1706.03762">attentional</a> or <a href="https://arxiv.org/abs/1705.03122">convolutional</a> structures. Other approaches such as the quasi-recurrent neural network (<a href="https://arxiv.org/abs/1611.01576">QRNN</a>), partially abandon recurrence and speed training by an order of magnitude. It’s exciting to see these new approaches reach state-of-the-art results while drastically reducing computational costs!</p><p>*However, the authors note that since the GPU accelerated functionality of cuDNN is optimized for an LSTM but not for the type of convolutions needed for the GCNN, the overall time required to process a corpus of text is still similar between the RNN and GCNN approaches. Improvements to cuDNN’s convolution implementation could potentially lead to much better computational performance for the GCNN.</p><p><strong>References:</strong></p><p><a href="https://arxiv.org/abs/1611.01576">Bradbury, James, et al. “Quasi-recurrent neural networks.” arXiv preprint arXiv:1611.01576 (2016).</a></p><p><a href="https://arxiv.org/abs/1612.08083">Dauphin, Yann N., et al. “Language modeling with gated convolutional networks.” arXiv preprint arXiv:1612.08083 (2016).</a></p><p><a href="https://arxiv.org/abs/1705.03122">Gehring, Jonas, et al. “Convolutional sequence to sequence learning.” arXiv preprint arXiv:1705.03122 (2017).</a></p><p><a href="https://arxiv.org/abs/1512.03385">He, Kaiming, et al. “Deep residual learning for image recognition.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.</a></p><p><a href="https://arxiv.org/abs/1706.03762">Vaswani, Ashish, et al. “Attention is all you need.” Advances in Neural Information Processing Systems. 2017.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ba3ee730bfb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-build-a-gated-convolutional-neural-network-gcnn-for-natural-language-processing-nlp-5ba3ee730bfb">Building a convolutional neural network for natural language processing</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Speed up your deep learning language model up to 1000% with the adaptive softmax, Part 2: Pytorch…]]></title>
            <link>https://medium.com/data-science/speed-up-your-deep-learning-language-model-up-to-1000-with-the-adaptive-softmax-part-2-pytorch-d47fe9a56152?source=rss-d543b0e79d50------2</link>
            <guid isPermaLink="false">https://medium.com/p/d47fe9a56152</guid>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[David Bressler]]></dc:creator>
            <pubDate>Wed, 14 Nov 2018 08:54:48 GMT</pubDate>
            <atom:updated>2018-11-16T00:22:08.122Z</atom:updated>
            <content:encoded><![CDATA[<h3>Speed up your deep learning language model up to 1000% with the adaptive softmax, Part 2: Pytorch implementation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dmEhey3rLz_tWpmX" /></figure><p>In <a href="https://medium.com/@dwbressler/speed-up-your-deep-learning-language-model-up-to-1000-with-the-adaptive-softmax-part-1-e7cc1f89fcc9">Part 1</a> of this blog post, I explained how the adaptive softmax works, and how it can speed up your language model by up to 1000%. Here in Part 2, I’ll walk you step by step through a Pytorch implementation (with an accompanying <a href="https://github.com/DavidWBressler/bressler_public_code/blob/master/Adaptive_softmax_example.ipynb">Jupyter notebook</a>), which uses Pytorch’s built-in <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/adaptive.html#AdaptiveLogSoftmaxWithLoss">AdaptiveLogSoftmaxWithLoss</a> function.</p><p>For preprocessing you will need fastai (see <a href="https://docs.fast.ai/">https://docs.fast.ai/</a>), a deep learning library that runs on top of Pytorch that simplifies training neural networks. [For those who want to learn state-of-the-art deep learning techniques, I highly recommend Jeremy Howard’s fast.ai course, which is available online for free: <a href="https://course.fast.ai/">https://course.fast.ai/</a>]. I decided to use the <a href="https://einstein.ai/research/blog/the-wikitext-long-term-dependency-language-modeling-dataset">Wikitext-2 dataset</a>, which is a relatively small dataset that contains ~2 million tokens and ~33 thousand words in the vocabulary. Once the data has been downloaded and properly formatted in csv files, fastai makes it easy to quickly tokenize, numericalize, and create a data-loader that will be used for training. I also downloaded <a href="https://nlp.stanford.edu/projects/glove/">GloVe pre-trained word vectors</a> that are used for the models’ word embeddings. Finally, I created a modeler class that handles training.</p><p>For the model, I created a simple version of a quasi-recurrent neural network (QRNN), which has better predictive accuracy than traditional recurrent neural networks (RNNs), and is up to 16 times faster (see Bradbury et al., 2016). The network I built is relatively small, with only 4 layers and 300 hidden units per layer… My purpose was to create a network that can reach a decent (if not state-of-the-art) result relatively quickly. The use of gradient clipping allows the use of a large learning rate (<em>lr</em>=1), which along with <a href="http://cs231n.github.io/neural-networks-3/#sgd">Nesterov’s momentum</a> helps training to converge quickly.</p><p><a href="https://medium.com/@dwbressler/speed-up-your-deep-learning-language-model-up-to-1000-with-the-adaptive-softmax-part-1-e7cc1f89fcc9">Part 1</a> describes how the data moves through the network. The output <em>x</em> of the last QRNN layer is of size <em>[seq_length=70, bs=50, nh=300]</em>, where <em>seq_length</em> is the length of the sequence, <em>bs</em> is batch size, and <em>nh</em> is the dimensionality of the final layer.</p><pre>x, new_h = self.rnn(x, self.hidden)</pre><h3>Regular Softmax</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0040e73c4428e6adf4d0e1ae7dc57e98/href">https://medium.com/media/0040e73c4428e6adf4d0e1ae7dc57e98/href</a></iframe><p>For the regular softmax, the next step after the last QRNN layer is a fully-connected linear layer (here called <em>fc1</em>), that transforms the output to have a number of output features equal to the vocabulary size (so going from size <em>[seq_length, bs, nh]</em> to size <em>[seq_length, bs, vs]</em>, where <em>vs</em> is the size of your vocabulary). For most language models, the compute time is dominated by this operation.</p><pre>#self.fc1 = nn.Linear(nh, vs)<br>x= self.fc1(x)</pre><p>The model then outputs the log-softmax of those values, and negative-log-likelihood is used to compute the loss.</p><pre>return F.log_softmax(x.view(-1,vs),dim=1)<br>...<br>loss = self.criterion(output, target)</pre><h3>Adaptive Softmax</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d763904be133c5d9fcd1102417356afa/href">https://medium.com/media/d763904be133c5d9fcd1102417356afa/href</a></iframe><p>The approach of the adaptive softmax is somewhat different. The output <em>x</em> of the last QRNN layer is still of size <em>[seq_length=70, bs=50, nh=300]</em>. The output is then resized to <em>[seq_length*bs, nh]</em>, and fed to the <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/adaptive.html#AdaptiveLogSoftmaxWithLoss">AdaptiveLogSoftmaxWithLoss</a> function, which computes the loss for you.</p><pre>x,new_h = self.rnn(x, self.hidden)<br>x=x.view(-1,x.size()[2])<br>#self.out=nn.AdaptiveLogSoftmaxWithLoss(nh, vs, cutoffs=[round(vs/15), 3*round(vs/15)], div_value=4)<br>return self.out(x,target)<br>...<br>loss= output.loss</pre><p>Let’s see what that AdaptiveLogSoftmaxWithLoss function does. You can see that I specified cutoffs = [round(vs/15), 3*round(vs/15)]. This means that I am splitting my vocabulary of 25520 words into 3 clusters. The head cluster contains the 1701 most common words (i.e. 1/15 of the vocabulary), plus two ‘words’ to indicate likelihood of the two tail clusters, for a total of 1703 words. The first tail cluster contains the next 3402 most common words (i.e. 2/15 of the vocabulary), and the second tail cluster contains the remaining 20417 least common words (i.e. 12/15 of the vocabulary). For this to work, I needed to make sure that my vocabulary (here called <em>itos</em>) was organized in order of the most common to the least common words.</p><p>As I mentioned in <a href="https://medium.com/@dwbressler/speed-up-your-deep-learning-language-model-up-to-1000-with-the-adaptive-softmax-part-1-e7cc1f89fcc9">Part 1</a>, each of these clusters is given a different capacity, the relative sizes of which are specified by the <em>div_value</em> given to the AdaptiveLogSoftmaxWithLoss function. A <em>div_value</em> of 4 means that the capacity of each additional cluster will be 4x smaller than the previous cluster. In this case, the head cluster has a capacity of 300, the first tail cluster has a capacity of 75, and the second tail cluster has a capacity of only 18. This means for the most common words in the head cluster, it will directly transform the dimensionality from 300 to 1703 output features (since 1703 is the number of vocabulary words in the head). For the first tail cluster however, an initial linear layer will first reduce the dimensionality down to 75 features, before a second linear layer transforms the dimensionality to 3402 output features. The initial reduction in the dimensionality speeds processing time (compared to a direct linear transformation from 300 input features to 3402 output features). A similar process occurs for the least common words in the last tail cluster.</p><pre>(out): AdaptiveLogSoftmaxWithLoss(<br>    (head): Linear(in_features=300, out_features=1703, bias=False)<br>    (tail): ModuleList(<br>      (0): Sequential(<br>        (0): Linear(in_features=300, out_features=75, bias=False)<br>        (1): Linear(in_features=75, out_features=3402, bias=False)<br>      )<br>      (1): Sequential(<br>        (0): Linear(in_features=300, out_features=18, bias=False)<br>        (1): Linear(in_features=18, out_features=20417, bias=False)<br>      )<br>    )</pre><h3>Results</h3><p>The traditional softmax network has a final loss of 4.84 (perplexity = 127), whereas the adaptive softmax network has a final loss of 4.88 (perplexity = 132). That is a less than 1% difference in final loss (and a 4% difference in perplexity). On the other hand, the adaptive softmax network finished computation at 7.80 minutes (7 minutes, 48 seconds), whereas the traditional softmax network completed at 24.55 minutes. Thus the outcomes of the two approaches are very similar in terms of accuracy, but the adaptive softmax network runs 3 times faster! These results match Grave et al.’s experiment on the Text8 dataset (which is of similar size to the Wikitext-2 dataset that I use)… a near identical final loss with an almost 3x speedup compared to the full softmax.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/1*BWixHrGnGaKIU7jd51yAVA.png" /><figcaption>Plotting time vs loss for the two approaches. Even though the final result is nearly the same, the adaptive softmax network completes training 3x quicker than an identical traditional softmax network.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/408/1*LcqR3Mg_Kqcg5AigZCG7SQ.png" /><figcaption>Plotting epoch vs loss for the traditional and adaptive softmax approaches. Training proceeds almost identically between the two approaches, although the traditional softmax network completes with a 1% lower loss than the adaptive softmax network.</figcaption></figure><p>So how good is the adaptive softmax compared to the traditional softmax? If I had decided to run training for more epochs, it’s quite possible that the final loss would have diverged even more. However, Grave et al. (2017) and other NLP researchers (e.g. Dauphin et al., 2016) show that they are in fact able to use the adaptive softmax to achieve state-of-the-art language modeling results on a variety of datasets. Therefore you should definitely consider adding the adaptive softmax to your language modeling toolbox, since it provides a massive speed-up over the full softmax with minimal cost on accuracy. See <a href="https://medium.com/@dwbressler/speed-up-your-deep-learning-language-model-up-to-1000-with-the-adaptive-softmax-part-1-e7cc1f89fcc9">Part 1</a> for a full explanation of how the adaptive softmax works.</p><h3>References:</h3><p><a href="https://arxiv.org/abs/1611.01576">Bradbury, James, et al. “Quasi-recurrent neural networks.” arXiv preprint arXiv:1611.01576 (2016).</a></p><p><a href="https://arxiv.org/abs/1612.08083">Dauphin, Yann N., et al. “Language modeling with gated convolutional networks.” arXiv preprint arXiv:1612.08083 (2016).</a></p><p><a href="https://arxiv.org/abs/1609.04309">Grave, Edouard, et al. “Efficient softmax approximation for GPUs.” arXiv preprint arXiv:1609.04309v3 (2017).</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d47fe9a56152" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/speed-up-your-deep-learning-language-model-up-to-1000-with-the-adaptive-softmax-part-2-pytorch-d47fe9a56152">Speed up your deep learning language model up to 1000% with the adaptive softmax, Part 2: Pytorch…</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Speed up your deep learning language model up to 1000% with the adaptive softmax, Part 1]]></title>
            <link>https://medium.com/data-science/speed-up-your-deep-learning-language-model-up-to-1000-with-the-adaptive-softmax-part-1-e7cc1f89fcc9?source=rss-d543b0e79d50------2</link>
            <guid isPermaLink="false">https://medium.com/p/e7cc1f89fcc9</guid>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[pytorch]]></category>
            <dc:creator><![CDATA[David Bressler]]></dc:creator>
            <pubDate>Wed, 14 Nov 2018 05:52:48 GMT</pubDate>
            <atom:updated>2018-12-12T01:51:14.775Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VQLmkf8jDQPZChYy" /></figure><p>How would you like to speed up your language modeling (LM) tasks by 1000%, with nearly no drop in accuracy? A recent paper from <a href="https://research.fb.com/category/facebook-ai-research/">Facebook’s AI Research group (FAIR)</a>, by <a href="https://arxiv.org/abs/1609.04309">Grave et al. (2017)</a>, called “Efficient softmax approximation for GPUs”, shows how you can gain a massive speedup in one of the most time-consuming aspects of language-modeling, the computation-heavy softmax step, through their “adaptive softmax”. The giant speedup from using the adaptive softmax comes with only minimal costs in accuracy, so anyone who is doing language modeling should definitely consider using it. Here in Part 1 of this blog post, I’ll fully explain the adaptive softmax, then in <a href="https://medium.com/@dwbressler/speed-up-your-deep-learning-language-model-up-to-1000-with-the-adaptive-softmax-part-2-pytorch-d47fe9a56152">Part 2</a> I’ll walk you step by step through a Pytorch implementation (with an accompanying <a href="https://github.com/DavidWBressler/bressler_public_code/blob/master/Adaptive_softmax_example.ipynb">Jupyter notebook</a>), which uses Pytorch’s built-in <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/adaptive.html#AdaptiveLogSoftmaxWithLoss">AdaptiveLogSoftmaxWithLoss</a> function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/1*BWixHrGnGaKIU7jd51yAVA.png" /><figcaption>In my experiment (see <a href="https://medium.com/@dwbressler/speed-up-your-deep-learning-language-model-up-to-1000-with-the-adaptive-softmax-part-2-pytorch-d47fe9a56152">Part 2</a> of this blog), the adaptive softmax version of my model trained 3x faster than the same model using a traditional softmax. Final accuracy was nearly the same.</figcaption></figure><p>As you probably know, language modeling is one of the most basic and important tasks in natural language processing (NLP). It involves creating a model that can predict the next word in a sequence, when given a sequence of words as an input (this is what enables the predictive text functionality on your phone’s keyboard). In order to do this, the last computational step in a language model involves choosing which word to select, from a huge list of words in the language model’s vocabulary. Since the vocabulary usually contains 10’s or 100’s of thousands of words, this last step is very computation- and time- intensive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hRfdoo3BE5YL9OSU" /><figcaption>Predictive texting… Did you mean “auto cucumber” or “autocorrect”?</figcaption></figure><p>So how does the language model select a single word from the vocabulary as its prediction? After the tokenization / numericalization preprocessing steps, a typical input to your model might be a tensor of size <em>[seq_length, bs]</em>, and the target would also be a tensor of size <em>[seq_length, bs]</em> (which is actually just the input shifted one time-point into the future), where <em>bs</em> is the number of examples in a given minibatch, and <em>seq_length</em> is the length of any given example. After accessing the embeddings (data will now be size <em>[seq_length, bs, emb_sz]</em>), usually the data is put through multiple layers of a recurrent neural network (RNN) using LSTM or GRU units, for a final hidden-state output of size <em>[seq_length, bs, nh]</em>, where <em>nh</em> is the dimensionality of your final layer. The goal is to compute <em>p(w|h)</em>… i.e. the probability of word <em>w</em> given final hidden state <em>h</em> (here, <em>h</em> is the final hidden-state output for a given example and timepoint, a vector of length <em>nh</em>).</p><p>The softmax solution to getting a prediction from that hidden-state output involves first transforming the output with a fully-connected linear layer, with the number of output features equal to the vocabulary size (so going from size <em>[seq_length, bs, nh]</em> to size <em>[seq_length, bs, vs]</em>, where vs is the size of your vocabulary). Then the softmax formula <em>exp(x[i]) / sum(exp(x))</em> is applied across all the words in the vocabulary, producing a probability value (between 0 and 1) for each word. Finally, you usually use negative log-likelihood (NLL) loss to compare these predictions to the target.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/537/1*Jn60jFG1g8qUPzsgTR8J1g.png" /><figcaption>Model design for a small RNN-network, with vs=25520 and nh=300. Notice that the last step is a fully-connected linear layer, that transforms the dimensionality from 300 to 25520.</figcaption></figure><p>That step of transforming the output from size <em>[seq_length, bs, nh]</em> to size <em>[seq_length, bs, vs]</em> is massive! The computational cost for this last step is linear with the size of your vocabulary, which as mentioned previously will be quite large if you want a decent language model. For typical values of <em>nh</em> and <em>vs</em>, the weight matrix for this transformation could be size [1,000 x 200,000], which is applied to every single timestep of every example in your dataset. This step dominates the computation, both at training and at test time, and is a major time-suck for any language model.</p><p>Grave et al.’s adaptive softmax is a pretty elegant solution to this problem. To understand it, you first need to understand the hierarchical softmax, which is a previous attempt to speed up the softmax (e.g. <a href="https://www.researchgate.net/publication/228348202_Hierarchical_probabilistic_neural_network_language_model">Morin &amp; Bengio, 2005</a>). You begin by assigning each word w to a unique cluster <em>C(w)</em>. The hierarchical softmax then breaks the softmax into two stages: you first predict the cluster <em>C</em>, and next you do a softmax over all the words for just that cluster. You can factorize the conditional probability of word w given final hidden state <em>h</em> as <em>p(w|h) = p</em>₂<em>(w | C(w), h) * p</em>₁<em>( C(w) | h)</em>… i.e. the probability of word <em>w</em> given hidden-state <em>h</em>, is the probability of cluster <em>C(w)</em> given <em>h,</em> times the probability of the word given both cluster <em>C(w)</em> and <em>h</em>. In simpler terms, first just predict the cluster, then predict which word in the cluster. This should reduce computation time from being linear with the size of the vocabulary, to being on the order of the square-root of the size of the vocabulary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/733/1*MHRvmqDtO_54ggrfiErcFQ.png" /><figcaption>Grave et al., 2017</figcaption></figure><p>However, as explained in Grave et al.’s paper, the hierarchical softmax actually doesn’t work that well on GPU’s (the explanation for this is beyond the scope of this blog post). Their adaptive softmax is a simple variant of the hierarchical softmax that is tailored for GPUs. It takes advantage of Zipf’s law… the observation that in any corpus, most of the probability mass of the distribution of words is covered by only a small fraction of the vocabulary. For example, in the <a href="https://catalog.ldc.upenn.edu/ldc99t42">Penn Treebank dataset</a>, 87% of the words in the document are covered by only 20% of the vocabulary. The adaptive softmax exploits this information, by assigning words in the vocabulary into clusters based on how common the words are.</p><p>To understand this approach, it’s best to start with the simplest form of the adaptive softmax, in which the vocabulary is partitioned into just two clusters, Vᴴᴱᴬᴰ and Vᵀᴬᴵᴸ. A smaller number of the most frequently occurring words in your corpus should go into Vᴴᴱᴬᴰ, and Vᵀᴬᴵᴸ should contain the remaining infrequent words. So in your corpus, its much more likely that a word will occur from Vᴴᴱᴬᴰ than from Vᵀᴬᴵᴸ (i.e. p(Vᴴᴱᴬᴰ) &gt; p(Vᵀᴬᴵᴸ) ), but there should be many more words in Vᵀᴬᴵᴸ than in Vᴴᴱᴬᴰ.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FBohFX3Hg9Fc%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DBohFX3Hg9Fc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FBohFX3Hg9Fc%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/6aae337695baef9c32f1ec71c8cfb93d/href">https://medium.com/media/6aae337695baef9c32f1ec71c8cfb93d/href</a></iframe><p>The first step in this 2-cluster example is to compute a softmax over all the words in the head, plus 1 extra ‘word’ that corresponds to selecting the tail-cluster. That is, first do a softmax over Vᴴᴱᴬᴰ + 1 words, where the extra ‘word’ is the probability it belongs to the tail-cluster. For words in Vᴴᴱᴬᴰ, <em>p(w|h)</em> is simply <em>pᴴᴱᴬᴰ(w|h)</em>… i.e. just compute the probability of the word in Vᴴᴱᴬᴰ given <em>h</em>. If the ‘word’ corresponding to the tail category is selected, only then do you need to do an additional softmax (over all the words in Vᵀᴬᴵᴸ). For these words, <em>p(w|h) is pᴴᴱᴬᴰ(tail|h) * pᵀᴬᴵᴸ(w|h)</em>… i.e. if <em>pᴴᴱᴬᴰ(w|h)</em> indicates ‘tail-cluster’, then take the probability of ‘tail-cluster’ in the head, multiplied by the probability of the selected word in the tail. For just using this 2-cluster approach, Grave et al. observe a 5x speedup over the full softmax!</p><p>However, you can get even further gains by using more clusters… the sweet spot seems to be 2–5 clusters. The approach is the same as for the 2-cluster example, except the first softmax will be over all the words in the head, plus however many additional ‘words’ to represent the tail clusters. For example, for a 5-cluster implementation, the initial softmax would be over Vᴴᴱᴬᴰ + 4 words. If one of the tail-cluster ‘words’ were selected in the initial softmax, then an additional softmax would be computed over just the words in the selected tail. Using the same principle as above, the most frequent words are assigned to the smallest clusters… The head and the first tails will have more frequent words but have a smaller vocabulary, and the last tails will have less frequent words and a larger vocabulary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/1*jlI9Q86RDBQLQjKLIpVwtA.png" /><figcaption>Figure 3 from Grave et al., 2017. This figure shows a clustering with 4 clusters: a head and 3 tails. The head contains a small number of the most frequent words, whereas the tails contain larger numbers of less frequent words.</figcaption></figure><p>There’s one more wrinkle that boosts speed at minimal cost to accuracy, which is that each cluster is given a different capacity. Above, <em>nh</em> is a measure of the capacity of the final hidden-state output. You need high capacity for the most frequently occurring words to be predicted correctly. However, rare words by definition only appear a few times and therefore cannot be learned that well by your model… It would be wasteful to use high capacity for these words. Therefore, Grave et al. reduce the dimensionality of the final hidden state input for the tail clusters’ softmax, through applying a projection matrix that reduces the dimensionality for each additional tail cluster by a factor of 4 (this will become clearer once we go through an example implementation in <a href="https://medium.com/@dwbressler/speed-up-your-deep-learning-language-model-up-to-1000-with-the-adaptive-softmax-part-2-pytorch-d47fe9a56152">Part 2</a> of this blog post).</p><p>And that’s it! You can experiment with the number of clusters and the distribution of words between the clusters to find out what words best for your dataset, using the principles outlined above (or through using dynamic programming). Overall, the adaptive softmax provides a massive speed-up (up to 10x!) over the full softmax, with minimal cost on accuracy. [Check out my <a href="https://towardsdatascience.com/how-to-build-a-gated-convolutional-neural-network-gcnn-for-natural-language-processing-nlp-5ba3ee730bfb">blog post</a> on CNN’s that can also speed training for NLP]. The adaptive softmax is going to be helpful any time you need to select from a huge number of possible categories, some of which are more common than others. This is true not only for language modeling, but for many other NLP tasks as well (neural machine translation (NMT), speech-to-text, etc.), and probably for many non-language applications too. Check out <a href="https://medium.com/@dwbressler/speed-up-your-deep-learning-language-model-up-to-1000-with-the-adaptive-softmax-part-2-pytorch-d47fe9a56152">Part 2</a> of this blog post to see a step by step <a href="https://github.com/DavidWBressler/bressler_public_code/blob/master/Adaptive_softmax_example.ipynb">Pytorch implementation</a> of adaptive softmax, which uses Pytorch’s built-in <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/adaptive.html#AdaptiveLogSoftmaxWithLoss">AdaptiveLogSoftmaxWithLoss</a> function.</p><h3>References:</h3><p><a href="https://arxiv.org/abs/1609.04309">Grave, Edouard, et al. “Efficient softmax approximation for GPUs.” arXiv preprint arXiv:1609.04309v3 (2017).</a></p><p><a href="https://www.researchgate.net/publication/228348202_Hierarchical_probabilistic_neural_network_language_model">Morin, Frederic, and Yoshua Bengio. “Hierarchical probabilistic neural network language model.” Aistats. Vol. 5. 2005.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e7cc1f89fcc9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/speed-up-your-deep-learning-language-model-up-to-1000-with-the-adaptive-softmax-part-1-e7cc1f89fcc9">Speed up your deep learning language model up to 1000% with the adaptive softmax, Part 1</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>