<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Aniket Dwivedi on Medium]]></title>
        <description><![CDATA[Stories by Aniket Dwivedi on Medium]]></description>
        <link>https://medium.com/@aniket.iitism?source=rss-7b647b198010------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*wUS5gJPIlgtWktkrT6g5Eg.jpeg</url>
            <title>Stories by Aniket Dwivedi on Medium</title>
            <link>https://medium.com/@aniket.iitism?source=rss-7b647b198010------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 17:34:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aniket.iitism/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building a Small Language Model (SLM) from Scratch: A Deep Dive into Transformers with PyTorch]]></title>
            <link>https://medium.com/@aniket.iitism/building-a-small-language-model-slm-from-scratch-a-deep-dive-into-transformers-with-pytorch-5c29c949db1e?source=rss-7b647b198010------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c29c949db1e</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[llm-vs-slm]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Aniket Dwivedi]]></dc:creator>
            <pubDate>Fri, 26 Sep 2025 13:28:09 GMT</pubDate>
            <atom:updated>2025-09-26T13:30:08.010Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/474/0*XSHu09Uqk6C1doNC" /><figcaption>Source: imageflip.com</figcaption></figure><h3>Introduction</h3><p>Large Language Models (LLMs) like GPT and LLAMA have revolutionized the field of Natural Language Processing (NLP). However, training and running these models require immense resources, making them impractical for many applications, especially those with limited computational budgets. This is where<strong> Small Language Models</strong> (SLMs) come into play.</p><h3>Problem Statement</h3><p>Why build an SLM?</p><ul><li>Resource Constraints: Not every organization or developer can afford to train or deploy gigantic models. SLMs are lightweight and can run on consumer-grade hardware.</li><li>Faster Inference: SLMs offer lower latency, making them suitable for real-time applications.</li><li>Customization: Building your own SLM lets you tailor the architecture to your dataset or application, rather than relying on pre-trained, general-purpose models.</li></ul><p><strong>But how do you actually build one from scratch, and what are the important architectural choices?</strong></p><p>In this article, we’ll walk through the implementation of an SLM based on the transformer architecture, using PyTorch. We’ll cover the why and how of each component, offering code insights and practical notes for aspiring ML practitioners.</p><h3>The Transformer Backbone</h3><p>The transformer architecture, introduced in <a href="https://arxiv.org/abs/1706.03762">Attention is All You Need (Vaswani et al., 2017)</a>, has become the de facto standard for NLP. Its core innovations — self-attention and positional encodings — allow models to capture rich relationships in text. This blog is not to explain transformer decoder block rather implementing it for the different use cases. In this blog, Tokenizer is <strong>Byte Pair Encoding</strong> used by GPT2 model.<br>We will drill down the following steps in this blog which is required to build SLM.</p><ol><li>Casual Self Attention</li><li>Layer Normalization</li><li>Multi-Layer Perceptron (Fast Feed Neural Network)</li><li>Attention Block</li><li>Model Consolidation Block</li><li>Rotatory Positional Encoding</li></ol><h3>What is Causal Self-Attention?</h3><p>Self-attention is the mechanism that lets every token in a sequence “look at” (i.e., attend to) other tokens, computing a weighted average of all tokens’ representations.<br>Causal self-attention adds a crucial twist: it ensures that, for language modeling, a token at position <em>i</em> can only attend to itself and tokens before it (never future tokens). This is essential for autoregressive generation and prevents “data leakage” from future words.</p><h3>Step-by-Step Explanation</h3><ol><li><strong>Initialization (__init__)</strong></li></ol><p>a) Linear Projection for QKV</p><pre>self.c_attn = nn.Linear(config.n_embd, 3 * config.n_embd, bias=config.bias)</pre><ul><li>Purpose:<br>Projects the input tensor into three separate tensors: queries (Q), keys (K), and values (V), all needed for attention.</li><li>How it works:<br>A single linear layer outputs a tensor with three times the hidden size. This will later be split into Q, K, and V.</li></ul><p>b) Output Projection: Transforms the concatenated attention outputs back to the model’s embedding size.</p><pre>self.c_proj = nn.Linear(config.n_embd, config.n_embd, bias=config.bias)</pre><p>c) Dropout Layers: Dropout is used to regularize the attention and projection outputs, helping to prevent overfitting.</p><pre>self.attn_dropout = nn.Dropout(config.dropout)<br>self.resid_dropout = nn.Dropout(config.dropout)</pre><p>d) Multi Head Setup: Stores how many attention heads and the hidden size per head.</p><pre>self.n_head = config.n_head<br>self.n_embd = config.n_embd</pre><p>2. Forward Pass (forward)</p><pre>B, T, C = x.size()<br>q, k, v = self.c_attn(x).split(self.n_embd, dim=2)<br>k = k.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)<br>v = v.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)<br>q = q.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)<br>q = apply_rope(q, seq_dim=2)<br>k = apply_rope(k, seq_dim=2)<br><br>y = F.scaled_dot_product_attention(q, k, v, attn_mask=None, dropout_p=..., is_causal=True)<br><br># if Flash attention is not there<br>att = (q @ k.transpose(-2, -1)) * (1.0 / math.sqrt(k.size(-1)))<br>causal_mask = torch.tril(torch.ones((T, T), device=x.device))<br>att = att.masked_fill(causal_mask == 0, float(&#39;-inf&#39;))<br>att = F.softmax(att, dim=-1)<br>att = self.attn_dropout(att)<br>y = att @ v<br><br>y = y.transpose(1, 2).contiguous().view(B, T, C)<br>y = self.resid_dropout(self.c_proj(y))</pre><ul><li>B: Batch size</li><li>T: Sequence length</li><li>C: Embedding dimension (model size)</li><li>Input tensor is projected and split into Q, K, and V tensors, each of shape [B, T, C].</li><li>Rearranges tensors so each attention head processes its own slice:<br>Shape becomes [B, n_head, T, head_dim], where head_dim = C // n_head.</li><li>Infuses position information directly into Q and K, so attention is sequence aware.</li><li>Recombine all heads by transposing and reshaping back to [B, T, C].</li><li>Pass through the output projection and apply final dropout.</li></ul><p><strong>Full Code for Initialization</strong></p><pre>class CausalSelfAttention(nn.Module):<br>  def __init__(self, config):<br>    super().__init__()<br>    self.c_attn = nn.Linear(config.n_embd, 3 * config.n_embd, bias=config.bias)<br>    self.c_proj = nn.Linear(config.n_embd, config.n_embd, bias=config.bias)<br>    self.attn_dropout = nn.Dropout(config.dropout)<br>    self.resid_dropout = nn.Dropout(config.dropout)<br>    self.n_head = config.n_head<br>    self.n_embd = config.n_embd<br>    self.flash = hasattr(torch.nn.functional, &#39;scaled_dot_product_attention&#39;)<br>    if not self.flash:<br>      self.register_buffer(&quot;bias&quot;, torch.tril(torch.ones(config.block_size, config.block_size))<br>                                       .view(1, 1, config.block_size, config.block_size))<br>      <br>  def forward(self, x, layer_past=None):<br>    B, T, C = x.size()<br>    q, k, v = self.c_attn(x).split(self.n_embd, dim=2)<br>    k = k.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)<br>    v = v.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)<br>    q = q.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)<br><br>    q = apply_rope(q, seq_dim=2)  # (B, n_head, T, head_dim)<br>    k = apply_rope(k, seq_dim=2)<br><br>    if self.flash:<br>      y = F.scaled_dot_product_attention(q, k, v, attn_mask=None, dropout_p=self.attn_dropout.p if self.training else 0.0, is_causal=True)<br>    else:<br>      att = (q @ k.transpose(-2, -1)) * (1.0 / math.sqrt(k.size(-1)))<br>      causal_mask = torch.tril(torch.ones((T, T), device=x.device))<br>      att = att.masked_fill(causal_mask == 0, float(&#39;-inf&#39;))<br>      att = F.softmax(att, dim=-1)<br>      att = self.attn_dropout(att)<br>      y = att @ v<br>    <br>    y = y.transpose(1, 2).contiguous().view(B, T, C)<br>    y = self.resid_dropout(self.c_proj(y))<br>    return y </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/1*GemVHcLRtjaqr9s_Hq8IXg.png" /><figcaption>Summary Table for Casual Attention Layer</figcaption></figure><h3>Layer Normalization</h3><p>Layer Normalization (LayerNorm) is a normalization technique used in deep learning models, especially transformers. Unlike batch normalization, which normalizes across a batch, LayerNorm normalizes across the features (embedding dimensions) of each individual data point — in this case, each token in the sequence.</p><pre>class LayerNorm(nn.Module):<br>  def __init__(self, ndim, bias):<br>    super().__init__()<br>    self.weight = nn.Parameter(torch.ones(ndim))<br>    self.bias = nn.Parameter(torch.zeros(ndim)) if bias else None<br><br>  def forward(self, x):<br>    x=F.layer_norm(x, self.weight.shape, self.weight, self.bias, 1e-5)<br>    return x</pre><ul><li>Stabilizes Training: Deep models (like transformers) can suffer from exploding or vanishing gradients. LayerNorm helps control this by ensuring each token’s features have mean zero and variance one, making training more stable and faster.</li><li>Sequence Independence: Since it normalizes each token independently, it works well for variable-length sequences — perfect for NLP tasks.</li><li>ndim: The number of features (should be equal to n_embd, the embedding size).</li><li>weight: Learnable scaling parameter (one for each feature). Initialized to 1.</li><li>bias: Learnable shifting parameter (one for each feature). Initialized to 0 and only created if bias=True.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/488/1*ngNUqR1itqya9Q87fQxFvA.png" /><figcaption>Summary Table for Layer Normalization</figcaption></figure><h3>Multi-Layer Perceptron (Fast Feed Neural Network)</h3><p>The mlp class implements the position-wise feed-forward network used in transformer architectures.<br>In transformer blocks, this comes right after the attention layer and is sometimes called the FFN (Feed Forward Network) sublayer.</p><h4>Why is a Feed-Forward Network Needed?</h4><ul><li>Nonlinearity: While self-attention mixes information between sequence positions, a feed-forward network allows the model to apply nonlinear transformations to each position independently.</li><li>Expressivity: The two-layer MLP increases the model’s ability to capture complex relationships and patterns in the data.</li><li>self.c_fc: A linear layer projecting input from size n_embd to 4 * n_embd. This expansion allows the network to learn a richer set of features (common in transformers).</li></ul><pre>class mlp(nn.Module):<br>  def __init__(self, config):<br>    super().__init__()<br>    self.c_fc = nn.Linear(config.n_embd, 4 * config.n_embd, bias=config.bias)<br>    self.gelu=nn.GELU()<br>    self.c_proj = nn.Linear(4 * config.n_embd, config.n_embd, bias=config.bias)<br>    self.dropout = nn.Dropout(config.dropout)<br><br>  def forward(self, x):<br>    x = self.c_fc(x)<br>    x = self.gelu(x)<br>    x = self.c_proj(x)<br>    x = self.dropout(x)<br>    return x</pre><ul><li>self.gelu: The GELU activation function introduces nonlinearity. GELU is smoother than ReLU and is widely used in transformers for its empirical performance.</li><li>self.c_proj: Projects the expanded features back to the original size (n_embd).</li><li>self.dropout: Randomly zeroes some values for regularization, helping prevent overfitting.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/519/1*LX0xbkE86x7wJeNJ5rdfPg.png" /><figcaption>Summary Table for MLP</figcaption></figure><h3>Attention Block</h3><p>The Block class represents a single transformer block, combining layer normalization, causal self-attention, and a feed-forward MLP with residual connections. It processes input sequences by first attending to previous tokens, then applying a nonlinear transformation, enabling deep contextual understanding and feature extraction at each layer.</p><pre>class Block(nn.Module):<br>  def __init__(self, config):<br>    super().__init__()<br>    self.ln1 = LayerNorm(config.n_embd, bias=config.bias)<br>    self.attn = CausalSelfAttention(config)<br>    self.ln2 = LayerNorm(config.n_embd, bias=config.bias)<br>    self.mlp = mlp(config)<br>  <br>  def forward(self, x):<br>    x = x + self.attn(self.ln1(x))<br>    x = x + self.mlp(self.ln2(x))<br>    return x</pre><h3>Model Consolidate Block</h3><ol><li>ModelConfig (@dataclass)</li></ol><p>This class simply packages all the configuration hyperparameters needed to define your model (like block_size, vocab_size, n_layer, n_head, n_embd, dropout, and bias).<br>Purpose: Makes model setup concise and error-free by passing a single config object.</p><pre>@dataclass<br>class ModelConfig:<br>  block_size: int<br>  vocab_size: int<br>  n_layer: int<br>  n_head: int<br>  n_embd: int<br>  dropout: float = 0.0<br>  bias: bool = True<br><br>class modelSLM(nn.Module):<br>  def __init__(self, config):<br>    super().__init__()<br>    self.config=config<br>    self.transformer = nn.ModuleDict(dict(<br>      wte=nn.Embedding(config.vocab_size, config.n_embd),<br>      wpe=nn.Embedding(config.block_size, config.n_embd),<br>      drop=nn.Dropout(config.dropout),<br>      h=nn.ModuleList([Block(config) for _ in range(config.n_layer)]),<br>      ln_f=LayerNorm(config.n_embd, bias=config.bias),<br>    ))<br>    self.lm_head = nn.Linear(config.n_embd, config.vocab_size, bias=False)<br>    self.lm_head.weight = self.transformer.wte.weight<br><br>    self.apply(self._init_weights)<br><br>    for pn, p in self.named_parameters():<br>      if pn.endswith(&#39;c_proj.weight&#39;):<br>        nn.init.normal_(p, mean=0.0, std=0.02 / math.sqrt(2 * config.n_layer))   #GPT-2 paper suggest 2*n_layer<br><br>  def _init_weights(self, module):<br>    if isinstance(module, nn.Linear):<br>      nn.init.normal_(module.weight, mean=0.0, std=0.02)<br>      if module.bias is not None:<br>        nn.init.zeros_(module.bias)<br>    elif isinstance(module, nn.Embedding):<br>      nn.init.normal_(module.weight, mean=0.0, std=0.02)    # std = 0.02 empirical chosen constant in gpt papers<br><br>  def forward(self, idx, target=None, return_last_token_only=False):<br>    device=idx.device<br>    b, t = idx.size()<br>    assert t &lt;= self.config.block_size, f&quot;Cannot forward sequence of length {t}&quot;<br>    pos=torch.arange(0, t, dtype=torch.long, device=device)<br>    tok_emb=self.transformer.wte(idx)<br>    pos_emb=self.transformer.wpe(pos)<br>    x=tok_emb+pos_emb<br>    x=self.transformer.drop(x)<br>    for block in self.transformer.h:<br>      x=block(x)<br>    x=self.transformer.ln_f(x)<br>    logits = self.lm_head(x)<br>    if return_last_token_only:<br>        logits = logits[:, -1:, :]  # (batch, 1, vocab_size)<br>    if target is not None:<br>        loss = F.cross_entropy(logits.view(-1, logits.size(-1)), target.view(-1), ignore_index=0)<br>        return logits, loss<br>    else:<br>        return logits, None</pre><ul><li>Weight tying: self.lm_head.weight = self.transformer.wte.weight.This “ties” the output projection weights to the input embedding weights, a trick that reduces model parameters and often improves performance (as shown in various language modeling papers).</li></ul><h4>In model.forward()</h4><ul><li>idx: Tensor of token indices (shape [batch, seq_len]).</li><li>target: (Optional) Target token indices for supervised training.</li><li>Option to return only last token’s logits: If return_last_token_only is True, only the last time-step (last token) logits are returned. This is useful for fast autoregressive generation.</li><li>Loss Calculation: If target is provided, computes cross-entropy loss between predicted logits and target tokens (ignores padding index 0). If target is not provided, returns logits and None for loss.</li></ul><h3>Loading Model</h3><pre>model_config = modelConfig(<br>    vocab_size=50000,    <br>    block_size=4096,<br>    n_layer=12,<br>    n_head=6,<br>    n_embd=384,<br>    dropout=0.1,<br>    bias=True<br>)<br><br>model=model(model_config)</pre><h3>Conclusion</h3><p>Building a Small Language Model (SLM) from scratch is not only a rewarding exercise for understanding the inner workings of transformers, but also a practical approach for deploying language models in resource-constrained environments. Throughout this article, we dissected each component of a transformer-based SLM — covering attention mechanisms, layer normalization, feed-forward networks, and the overall architecture — emphasizing how sequences are processed from input to output.</p><h4>Popular SLMs in Use</h4><p>While giants like GPT-3 and Llama 2 dominate headlines, there are several compact language models widely used in industry and research, such as:</p><ul><li>DistilGPT-2: A smaller, distilled version of GPT-2, optimized for speed and efficiency.</li><li>ALBERT: A light-weight variant of BERT that reduces parameters via factorized embeddings.</li><li>TinyBERT: A distilled BERT model tailored for on-device NLP.</li><li>MobileBERT: Designed specifically for mobile and edge devices.</li><li>MiniLM: Lightweight and fast, yet competitive in many NLP benchmarks.</li></ul><h4>How Can you use your SLM</h4><p>Your custom SLM can be used in a variety of scenarios, including:</p><ul><li>On-Device Inference: Deploy your model on mobile phones, IoT devices, or embedded systems for private, low-latency NLP tasks.</li><li>Chatbots and Virtual Assistants: Power conversational agents for customer support or in-app assistance without needing cloud resources.</li><li>Text Generation: Generate summaries, headlines, or creative text autonomously in applications where GPU/TPU resources are limited.</li><li>Personalized Models: Fine-tune your SLM on domain-specific data — like legal, medical, or technical corpora — for specialized use cases.</li><li>Edge Computing: Integrate your SLM into edge devices for real-time, local decision-making, such as smart home automation or automotive voice control.</li><li>Education and Research: Use your SLM as a teaching tool or research baseline to experiment with new architectures, optimization techniques, or data pipelines.</li></ul><p><em>Thank you for reading this article!</em><br>Stay tuned for more deep dives and practical guides on building and deploying intelligent systems from scratch.</p><h3>Rotary Positional Encoding (RoPE) — Optional/Advanced</h3><p>Transformers need a way to know the position of each token in a sequence, since self-attention alone is “orderless.”<br>Rotary Positional Encoding (RoPE) is a modern technique for adding this position information directly into the attention mechanism.</p><p>How it works (intuitively):</p><ul><li>RoPE rotates each token’s vector in a special way using sine and cosine functions.</li><li>This “rotation” is different for each token position, so the model can tell apart, for example, the 3rd word from the 10th word in a sentence.</li><li>It allows the model to better capture the relationship between token positions, especially for long sequences.</li></ul><p>Why use RoPE?</p><ul><li>More efficient and flexible than classic positional encodings, especially for longer texts.</li><li>Used in newer language models like GPT-NeoX and Llama.</li></ul><p>You can easily get the RoPE code on the internet. Please comment if you are looking for the full RoPE code</p><h4>References</h4><ul><li>Vaswani, A., et al. (2017). <a href="https://arxiv.org/abs/1706.03762">“Attention is All You Need”</a></li><li>GPT-2 Paper: <a href="https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf">“Language Models are Unsupervised Multitask Learners”</a></li><li>Customization: Building your own SLM lets you tailor the architecture to your dataset or application, rather than relying on pre-trained, general-purpose models.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c29c949db1e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimizers]]></title>
            <link>https://medium.com/@aniket.iitism/optimizers-40b5803d7c32?source=rss-7b647b198010------2</link>
            <guid isPermaLink="false">https://medium.com/p/40b5803d7c32</guid>
            <dc:creator><![CDATA[Aniket Dwivedi]]></dc:creator>
            <pubDate>Mon, 24 Apr 2023 18:44:09 GMT</pubDate>
            <atom:updated>2025-08-31T18:35:41.656Z</atom:updated>
            <content:encoded><![CDATA[<h3>Gradient Descent — Minimizing the Cost Function</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*i4rXK2RMkuTDbz8totUeOQ.png" /><figcaption>Gradient Descent Image Source : Wikipedia</figcaption></figure><p>The goal of any Machine Learning Algorithm is to minimize the Cost Function.</p><p><strong>Let’s climb to the top of the mountain as an example.</strong></p><p>Suppose we are located on the mountainside of a mountain, without a map, and do not know how to reach the top of the mountain. So I decided to take one step, that is, to go further in the direction of the steepest and most easy to go up the current position, and then continue to take a small step in the steepest direction of the next position. This step by step, go all the way to think that we have reached the top of the mountain. Here the direction of the mountain through the steepest path is the gradient.</p><h3>Gradient Descent</h3><p><em>Gradient descent is a first-order iterative optimization algorithm used to minimize a function L, commonly used in machine learning</em> and <em>deep learning.</em></p><p>The idea behind the first-order derivative is that the algorithm takes the first order derivates to update the parameters or weight of the neural nets. In every iterations, we update the parameters in the opposite direction of the gradient computed for the Loss Function (L) w.r.t to the <strong>weight</strong> (w) and <strong>bias</strong> (b). The size of the update is determined by the step size called <strong>learning rate α.</strong></p><blockquote><strong><em>w = w — alpha * dL/dw</em></strong></blockquote><blockquote><strong><em>b = b — alpha * dL/db</em></strong></blockquote><p>Here, the learning rate (alpha) is constant but for the better convergence we generally use Adagrad, Adadelta, RMSprop optimezers which we are not going to cover in this article.</p><h3><strong>Types Of Gradient Descent</strong></h3><blockquote><strong><em>Batch gradient descent (BGD)</em></strong></blockquote><h4><strong>Gradient update rule:</strong></h4><p>BGD uses the data of the entire training set to calculate the gradient of the cost function to the parameters:</p><h4><strong>Disadvantages:</strong></h4><p>— Because this method calculates <strong>the gradient for the entire data set in one update, the calculation is very slow</strong>, it will be very tricky to encounter a large number of data sets, and you cannot invest in new data to update the model in real time.</p><p><strong>—</strong> BGD can converge to a global minimum for convex functions and to a local minimum for non-convex functions</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fcAYKx6_vio-BZZmjui9cg.png" /><figcaption>Local and Global Minima in BGD Image Source: Wikipedia</figcaption></figure><blockquote><strong><em>Stochastic gradient descent (SGD)</em></strong></blockquote><h4><strong>Gradient update rule:</strong></h4><p>Compared with BGD’s calculation of gradients with all data at one time, SGD updates the gradient of each sample with each update.</p><p>x += — learning_rate * dx</p><p>where x is a parameter, dx is the gradient and learning rate is constant</p><h4><strong>Disadvantages:</strong></h4><p>— However, because SGD is updated more frequently, the cost function will have severe oscillations.</p><p>— BGD can converge to a local minimum, of course, the oscillation of SGD may jump to a better local minimum.</p><p>— When we decrease the learning rate slightly, the convergence of SGD and BGD is the same.</p><blockquote><strong><em>Mini-batch gradient descent (MGBD)</em></strong></blockquote><h4><strong>Gradient update rule:</strong></h4><p>MBGD uses a small batch of samples, that is, n samples to calculate each time. In this way, it can reduce the variance when the parameters are updated, and the convergence is more stable. It can make full use of the highly optimized matrix operations in the deep learning library for more efficient gradient calculations.</p><p>The difference from SGD is that each cycle does not act on each sample, but a batch with n samples.</p><p>Setting value of hyper-parameters:</p><p>n Generally value is 50 ～ 256</p><h4><strong>A:dvantage:</strong></h4><p>— Mini-batch gradient descent does not guarantee good convergence,</p><p>— If the learning rate is too small, the convergence rate will be slow. If it is too large, the loss function will oscillate or even deviate at the minimum value.</p><p>— However, the setting of learning rate needs to be written in advance adapt to the characteristics of the data set.</p><blockquote><strong><em>Conclusion</em></strong></blockquote><p>In BGD, SGD and Mini Batch gradient descent <strong>same learning rate</strong> is applied to all parameter updates. If our data is sparse ( data having more 0 in the sets), we would prefer to update the features with lower frequency.</p><p>In addition, for <strong>non-convex functions</strong>, it is also necessary to avoid trapping at the local minimum or saddle point, because the error around the saddle point is the same, the gradients of all dimensions are close to 0, and SGD is easily trapped here.</p><p><strong><em>Saddle points </em></strong><em>are the curves, surfaces, or hypersurfaces of a saddle point neighborhood of a smooth function are located on different sides of a tangent to this point.</em></p><p><strong><em>Comparison</em></strong>: If we compare all three optimizer, then every optimizer has its own advantages and disadvantages ,we can’t come to conclusions which optimizer is best, it totally depends on datasets.</p><p>But this diagram may clear your overall confusion</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/527/1*XqlJ0oHoesK_GWiAa0jslQ.jpeg" /></figure><p>One disadvantage of the SGD method is that its update direction depends entirely on the current batch, so its update is very unstable. A simple way to solve this problem is to introduce <strong>momentum </strong>which we will discuss in the next article. Stay Tuned !!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=40b5803d7c32" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>