<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Aarav Dabral on Medium]]></title>
        <description><![CDATA[Stories by Aarav Dabral on Medium]]></description>
        <link>https://medium.com/@aarav.dabral?source=rss-292bc66f6ebd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*C7O-P8F4xMlvY86poOcnRw.jpeg</url>
            <title>Stories by Aarav Dabral on Medium</title>
            <link>https://medium.com/@aarav.dabral?source=rss-292bc66f6ebd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 10:27:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aarav.dabral/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[HOW A LANGUAGE MODEL THINKS: LLMS ARCHITECTURE FIRST PRINCIPLES]]></title>
            <link>https://medium.com/@aarav.dabral/how-a-language-model-thinks-llms-architecture-first-principles-2e124b210008?source=rss-292bc66f6ebd------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e124b210008</guid>
            <category><![CDATA[llms-reasoning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Aarav Dabral]]></dc:creator>
            <pubDate>Wed, 04 Nov 2026 14:35:53 GMT</pubDate>
            <atom:updated>2026-11-04T14:35:53.710Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*4-kGGOvRxFkdxPLpmCViqQ.jpeg" /><figcaption>I’m the supreme leader</figcaption></figure><p>So, let us divide into topics to understand the intuition about , the main motive to write this blog/article whatever you want to call it out is to understand LLM’s topics from the first principles with a bit of philosophical(something exciting to relate with so your brain asks WHY?).</p><p>but but(yeah I ain’t exaggerating , I just wanna make sure you get hooked by the first paragraph itself so you don’t get bored by the further transformer blocks)</p><h3>SIMPLE UNDERSTANDING DEFINATION</h3><p>Every large language model(LLM), from the smallest open-weights toy to the frontier giants, is built on one almost insultingly simple bet: <strong>given everything so far, predict the next token.</strong> That is the entire training objective. No grammar module, no knowledge base, no reasoning engine bolted on.</p><p>The philosophical wrinkle is that to predict the next token <em>well</em>, you must model the world that produced the text. To finish a detective story, you need some grip on motive. To finish a proof, you need some grip on logic. Compression, it turns out, is a quiet cousin of understanding. This post walks through the machinery that makes that bet pay off, from first principles upward.</p><p>TOPICS TO BE COVERED:</p><ul><li>Attention Mechanism(explaining GQA , MHA and attention itself yeah obviously will be taking reference to when Attention Mechanism was first stated out in the world ie “Attention is all you Need”).</li><li>Token Embeddings , Tokens , Context Length , the main formula which involves:- Query , Values and Keys(Q,K and V not in order-wise but who cares) , also it involves the length of the model(d_model).</li><li>few hints about KV-Cache , MOE(Mixture of Experts) and the activation functions used in Llama ,Qwen and obviously GPT.</li></ul><h3>TOKENS(TOKENIZATION):</h3><p>A neural network eats numbers, not text. So the first step is to chop text into pieces and map each piece to an integer ID. Those pieces are <strong>tokens</strong>.</p><p>Tokens are not words and not characters. They are something in between, learned from data by an algorithm like <strong>Byte-Pair Encoding (BPE) especially used in GPT</strong>: start with raw bytes, repeatedly merge the most frequent adjacent pair, and stop when you reach a target vocabulary size (typically 32k to 200k entries)</p><p>Let’s take an example to understand this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*fVLw0PYpAVhVI7hn9diamQ.png" /></figure><p>This is Plato’s cave in silicon. The model has never seen a cat, only the shadows words cast, and shadows that have been further sliced into sub word fragments. And yet, as Wittgenstein suggested, meaning may live in <em>use</em> rather than in the thing itself. A model that has seen how a word is used ten billion times knows something real about it.</p><h3><strong>TOKEN EMBEDDINGS(GEOMETRICAL MEANING)</strong></h3><p>These vectors start random and are shaped by training. Tokens used in similar contexts get pushed toward similar vectors. That is the <strong>distributional hypothesis</strong>: a word is characterized by the company it keeps. Direction and distance in this space come to encode relationships: tense, plurality, topic, sentiment, formality.</p><p>now , let me give a example to understand this better</p><p>So , lets take an IntegerID which has no general meaning at all , ID 4821 is not “closer” to 4822 than to 90,000. So the next step is a giant lookup table, the <strong>embedding matrix</strong> E of shape [vocab_size, d_model]. Token ID i simply selects row i: a vector of, say, 4096 learned numbers.</p><h4>Let me take an initiative into <strong>RoPE( Rotationary Positional Embeddings)</strong> which comes in the form of theta(a special symbol can’t “type” it out here) wile using sine and cosine computation which rotate query and key vectors by an angle that depends on position, so attention scores naturally depend on relative distance.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*6S0QBXYPp2h23Fas-dH1WA.png" /><figcaption>simple discussion on the RoPe functionalites occuring</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*IbKmCy25SxuSKDo1ylkP6A.png" /><figcaption>LINEARRRRR ALGEEBRRAAA(speaking in statquest “intro vid style”)</figcaption></figure><pre># Compute the angles<br>    angles = positions[:, None] * inv_freq[None, :]  # Shape: (context_length, head_dim // 2)<br><br>    # Expand angles to match the head_dim<br>    angles = torch.cat([angles, angles], dim=1)  # Shape: (context_length, head_dim)<br><br>    # Precompute sine and cosine<br>    cos = torch.cos(angles)<br>    sin = torch.sin(angles)<br><br>    return cos, sin<br><br><br>def apply_rope(x, cos, sin):<br>    # x: (batch_size, num_heads, seq_len, head_dim)<br>    batch_size, num_heads, seq_len, head_dim = x.shape<br>    assert head_dim % 2 == 0, &quot;Head dimension must be even&quot;<br><br>    # Split x into first half and second half<br>    x1 = x[..., : head_dim // 2]  # First half<br>    x2 = x[..., head_dim // 2 :]  # Second half<br><br>    # Adjust sin and cos shapes<br>    cos = cos[:seq_len, :].unsqueeze(0).unsqueeze(0)  # Shape: (1, 1, seq_len, head_dim)<br>    sin = sin[:seq_len, :].unsqueeze(0).unsqueeze(0)<br><br>    # Apply the rotary transformation<br>    rotated = torch.cat((-x2, x1), dim=-1)<br>    x_rotated = (x * cos) + (rotated * sin)</pre><h4>Philosophical aside: Learned absolute positions are Newton’s absolute space: every token has an address on a fixed grid. RoPE is Leibniz’s relational space: there is no grid, only distances between things.</h4><h3>ATTENTION(NOT YOUR INSTA TIME SPAN)</h3><p>So , firstly let me spread out the Transformer Block for you in order for you to understand the working mechanism of Attention whether its the masking(casual masking) or the simple attention head(MHA).</p><p>YAYA more hint into the casual masking one , as it is also used in fine tuning a model also(recently self-learned myself) , A <strong>causal mask</strong> ensures token <em>t</em> can only look at tokens ≤ <em>t</em>. The model may never peek at the future, which is exactly what makes next-token training honest. So , we use this while comprehending with max_length we set the default to be None(during the if-else tree)</p><p>A decoder-only Transformer is the same block stacked dozens of times. Picture a <strong>residual stream</strong>: a running vector per token, like a whiteboard passed down a line of specialists. Each layer reads the board, computes something, and <em>adds</em> its contribution back (x = x + layer(norm(x))). Nothing is overwritten; everything accumulates.</p><p>It is split into two parts</p><ul><li>Attention: Tokens look at other tokens and move information between positions.</li><li><strong>MLP (feed-forward network)</strong>: each position independently transforms its own vector. This is where much of the model’s stored “knowledge” is believed to live.</li></ul><h4>ATTENTION FROM FIRST PRINCIPLES-</h4><p>So , firstly we dedicate this solely to the goated paper in recent times involving Transformers or even for the introduction of transformers(“ATTENTION IS ALL YOU NEED”)</p><p>so firstly it is the formulated code of the attention</p><pre>import torch, math<br><br>def attention(Q, K, V):           # each: [seq, d_head]<br>    scores = Q @ K.T / math.sqrt(Q.shape[-1])<br>    mask = torch.triu(torch.ones_like(scores), diagonal=1).bool()<br>    scores = scores.masked_fill(mask, float(&quot;-inf&quot;))   # causal mask<br>    return torch.softmax(scores, dim=-1) @ V</pre><p>For each token vector x, the model computes three projections:</p><ul><li><strong>Query (Q):</strong> “what am I looking for?”</li><li><strong>Key (K):</strong> “what do I contain?”</li><li><strong>Value (V):</strong> “what will I hand over if you pick me?”</li></ul><p>A token’s query is compared against every earlier token’s key (a dot product); softmax turns the scores into weights that sum to 1; and the output is the weighted average of the values. The √d division keeps dot products from growing with dimension, which would push softmax into saturation and starve gradients.</p><p>It is a soft, differentiable library lookup: instead of fetching one book, you fetch a blend of all books, weighted by relevance.</p><h3>Topics which are real important to keep in mind while studying LLMS:</h3><h4>Multi Head Attention(MHA):multiple things to look after-</h4><p>Language has many simultaneous relations: syntax (which verb belongs to which subject), coreference (what “it” points to), position, topic. So we split the model dimension into <strong>H heads</strong>, each with its own W_Q, W_K, W_V(weight vectors), each attending in a smaller subspace (d_head = d_model / H). Outputs are concatenated and mixed by an output projection. In <strong>vanilla MHA(like the basic to basic MHA)</strong>, every query head has its own key head and value head. Different heads empirically specialise: some track the previous token, some match repeated patterns, some bind pronouns to nouns.</p><h4>MQA AND GQA(and where are they used and the WHY?, they just share keys and values):</h4><p>The observation behind <strong>Multi-Query Attention (MQA)</strong>: keep many query heads but let them all share <strong>one</strong> K and V head. The cache shrinks by a factor of H. The price: some quality loss and training instability.</p><p><strong>Grouped-Query Attention (GQA)</strong> is the sensible middle path. Split the query heads into G groups; each group shares one K/V head.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/503/1*-fbytGQ6XihWKhV7Y-0Xfw.png" /><figcaption>it just means in GQA , queries can be same but they can have multiple keys,values</figcaption></figure><p>Lets take Llama-3b for teaching purpose(feynman technique ifykyk)</p><p>It uses 32 query heads with 8 KV heads, a 4× reduction: roughly 128 KB per token instead of 512 KB, so an 8k-token context needs about 1 GB rather than 4 GB. Quality stays close to full MHA, which is why GQA became the default in most open models.</p><pre>class Attention(nn.Module):<br>    # n_kv_heads == n_heads -&gt; MHA | 1 &lt; n_kv_heads &lt; n_heads -&gt; GQA | n_kv_heads == 1 -&gt; MQA<br>    def __init__(self, d, n_heads, n_kv_heads):<br>        super().__init__()<br>        self.h, self.kv, self.hd = n_heads, n_kv_heads, d // n_heads<br>        self.q = nn.Linear(d, n_heads * self.hd, bias=False)<br>        self.k = nn.Linear(d, n_kv_heads * self.hd, bias=False)   # smaller under GQA<br>        self.v = nn.Linear(d, n_kv_heads * self.hd, bias=False)   # smaller under GQA<br><br>    def forward(self, x, cache):<br>        B, T, _ = x.shape<br>        q = self.q(x).view(B, T, self.h,  self.hd).transpose(1, 2)<br>        k = self.k(x).view(B, T, self.kv, self.hd).transpose(1, 2)<br>        v = self.v(x).view(B, T, self.kv, self.hd).transpose(1, 2)<br>        # ...append k, v to the KV-cache HERE, while they still have only kv heads...<br>        rep = self.h // self.kv<br>        k = k.repeat_interleave(rep, dim=1)   # each KV head serves `rep` query heads<br>        v = v.repeat_interleave(rep, dim=1)<br>        # ...then ordinary causal attention over q, k, v...</pre><h4>Now , lets talk about context length(The Model’s Working Memory)</h4><p>The <strong>context window</strong> is the maximum number of tokens the model can attend to at once: prompt, history, and generated output together. Everything outside it does not exist for the model.</p><h3>Now lets talk about few inference/optm things of LLMs</h3><h4>KV-Cache(m also learning this along the rasbt/LLMS-from-scratch repo, :-)</h4><p>So , the main motive is to remembering things so you don’t recompute it later on. Generation is autoregressive: produce one token, append it, repeat. Naively, each new token would force recomputing keys and values for the entire prefix. But past keys and values never change (thanks to the causal mask). So we store them: the <strong>KV-cache</strong>.</p><p>With it, each step computes Q, K, V only for the <em>new</em> token, appends K and V to the cache, and attends over the stored history. Compute per step drops sharply, but <strong>memory becomes the bottleneck</strong></p><pre>KV bytes per token = 2 (K and V) × layers × kv_heads × head_dim × bytes_per_value</pre><h4>MoE(Mixture of Experts) scale without paying for all of it:</h4><p>In a dense model, every token passes through every parameter. MoE breaks that link. Replace the single MLP in each (or every other) block with <strong>N expert MLPs</strong> plus a small <strong>router</strong>. For each token the router scores the experts and sends the token to only the <strong>top-k sampling</strong>(often k = 1 or 2, plus sometimes a shared expert) then combines their outputs weighted by router scores.</p><h4>GeLu and SiLu(activation fns used in GPT, Llama and Qwen)</h4><p>The MLP in every block needs a nonlinearity, otherwise stacked linear layers collapse into one big linear layer. GPT-style models use <strong>GELU(Gaussian Error Linear Unit)</strong>; Llama and Qwen use <strong>SiLU(Sigmoid Linear Unit) </strong>inside a gated unit called <strong>SwiGLU</strong>.</p><p>Now , firstly we’ll look into the why? part of ReLu not available be an activation fn. ReLU is a hard door: negatives are shut out, positives pass untouched, it is a switcher on the other hand GeLu and SiLu are dimmer switches and also it prevents the dying neural problem , if we use the other two activation fns(yea correlate it with the dying star thingy)</p><pre>def gelu(x): return 0.5 * x * (1 + torch.erf(x / math.sqrt(2))) # x * Phi(x), Phi = Gaussian CDF<br>----------------------------------------<br>def silu(x): return x * torch.sigmoid(x) # x * sigmoid(x), also called Swish</pre><h3>For the final act of life(I’ll give it everything)</h3><ol><li><strong>Tokenise</strong> the text into IDs.</li><li><strong>Embed</strong> each ID into a d_model vector (position enters via RoPE inside attention).</li><li>For each of <em>L</em> layers: <strong>normalise → attention (GQA, causal, reading and writing the KV-cache) → add to the residual stream → normalise → MLP or MoE → add to the residual stream.</strong></li><li><strong>Final norm</strong>, then project through the (often tied) embedding matrix to get a score for every token in the vocabulary.</li><li><strong>Softmax and sample</strong> the next token. Append it. Repeat, using the cache.</li></ol><p><strong>There is something quietly humbling in the design. Nothing in the architecture mentions grammar, facts, or logic. We supply a tokenizer, a geometry of vectors, a mechanism for tokens to consult one another, and a vast pile of text, and structure emerges because predicting the next token demands it. Whether that amounts to understanding is a question the architecture cannot answer. But it does sharpen the question beautifully.</strong></p><p>hope you love the blog , the next one will be a real detail one but after my GPT scratch py project(for muscle memory). It’ll be on the comparision of GPT, Llama and Qwen architectures.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/364/1*QeMuaFVwMq7vBfXlgKrd8A.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e124b210008" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[INTUITION BEHIND “LSTM”.]]></title>
            <link>https://medium.com/@aarav.dabral/intuition-behind-lstm-17de6e15e5e2?source=rss-292bc66f6ebd------2</link>
            <guid isPermaLink="false">https://medium.com/p/17de6e15e5e2</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[rnn-architecture]]></category>
            <dc:creator><![CDATA[Aarav Dabral]]></dc:creator>
            <pubDate>Thu, 29 Oct 2026 07:35:53 GMT</pubDate>
            <atom:updated>2026-10-29T07:35:53.169Z</atom:updated>
            <content:encoded><![CDATA[<p>“forgetting is not the enemy of memory , its just the precondition”.</p><p>(please get familiar with my short forms and notations , thanks)</p><p>So , let me cut straight to the point , its just going to be simple understanding basis and probably you should know few ml(machine learning) notations and dl(deep learning) notations to get the intuition , but I’ll obviously try my best to give some easy examples which may/may not(depends on your IQ) be interpreted by you.</p><p>So LSTM comes under RNN(which is Recurrent Neural Networks) it’s more of a sequential thingy in the state of time being(sounds quite of a philosophical time this “state of time being”)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*-zUdyGspjkuqnJAd67zTIA.jpeg" /><figcaption>me to you guys</figcaption></figure><p>So , firstly lemme just list down the topics we going to learn and I would share my inputs into(funny enough to comprehend these topics into such intuition but yea)</p><h3><strong>BOOM SHAKA LAKA(I’m rather too excited to write this piece of work , just feynman inspo things)</strong></h3><ul><li>RNN’s(quite obvious isn’t it??) , also I’m including the BPTT(backpropagation through time, yes m a backprop ninja) thing here only.</li><li>DropConnect(related to dropout? yea we’ll discuss it later on).</li><li>SGD(especially ASGD sub part of it but yes we’ll discuss that).</li><li>AdamW(includes the weight decay factor in it) and the other optimizers used while making especially AWD-LSTM(making a project also but yeah there’s already a paper written on this topic so , if you’re keen enough for more learning just read the paper) <a href="https://arxiv.org/abs/1708.02182">https://arxiv.org/abs/1708.02182</a>.</li><li>AR and TAR(don’t get scared by the short forms its just regularization terms) obvious I’ll explain it.</li></ul><h3><strong>RNNs: a mind that carries a summary</strong></h3><p>A feed-forward network lives in the present: it sees an input, answers, and forgets. A recurrent neural network (RNN) adds the one thing a reader needs, a past. It feeds its own state back into itself</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/1*kbFaEbsEg2yy8Bq84d27uw.png" /><figcaption>x(t) is the input at step t , h(t) is the hidden state , b are the biases and W is the weight matrix , softmax is the activation function used here(please learn activation yourself as this is all about LSTM(ie RNN’s)</figcaption></figure><p>BPTT is an ordinary backpropagation on the RNN unrolled over T steps, plus one rule. Since every step reuses the same weights, a shared weight’s gradient is the sum of its gradients from all steps.</p><p>Here is the key idea and how it is presentable into three mainly steps carried forward to implement BPTT:</p><ol><li><strong>Unroll{} :</strong>Treat the loop as a chain of (t) copies of the same cell, linked by the hidden state. The total loss is the sum of the per-step losses(summation of loss per step in mathematical language) , L(t){very well its a cost/loss fn(here we use cross entropy loss) so yeah} , is the cross-entropy of the prediction at step “t”. In the forward pass we run the chain left to right and <em>store</em> every intermediate value (gates, cell states, hidden states), because the backward pass will need them.</li></ol><p><strong>2. Send blame backwards, step by step{} </strong>A hidden state h(t) , influenced two things: the prediction made at its own step, and every later state that was computed from it. Its gradient is therefore the sum of a direct term and an inherited term.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*lA1oTY8BB7JSjFnVVPJYgA.png" /><figcaption>its the mathematical gradient forms written from the statement provided.</figcaption></figure><p>3.<strong> Accumulate for shared weights{} </strong>The matrix W(h) was used at <em>every</em> step, so every step has a claim on it. Its gradient is the sum of the local contributions</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/733/1*vNZaYxNdees8fAsR0fFkXg.png" /><figcaption>also , we have the grad clipping norm here also but its in truncated BPTT.</figcaption></figure><h3>DropConnect(no single wire may be indispensable)</h3><p>Dropout silences neurons. <strong>DropConnect</strong> silences individual <em>connections</em>, the weights themselves. The AWD-LSTM applies it to exactly one place, the hidden-to-hidden matrix W(h), with probability 0.5(reference to my project).</p><p>The idea is <strong>redundancy as a virtue</strong>. If any weight might vanish, no weight can become a single point of failure, and knowledge must be spread across many pathways rather than stored in one fragile one. The choice of where is deliberate too: the recurrent matrix is the most overfitting-prone part of an LSTM, and dropping <em>weights</em> rather than <em>activations</em> leaves the carried state, the memory itself, untouched. The model’s recollection stays intact while its wiring is made robust.</p><p><strong><em>{this is the main computational which we follow while implying DropConnect}</em></strong></p><p>Two implementation facts follow from the philosophy. The mask is drawn once per forward/backward pass, so the same damaged network runs across the whole window, keeping its internal dynamics consistent. And only the surviving weights receive gradient, because a connection that was absent cannot be blamed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*fJlb7AhINQJHqyyKUdwVRw.png" /><figcaption><a href="https://arxiv.org/html/2502.19948v1">https://arxiv.org/html/2502.19948v1</a> refer this article for more intel about dropout and its further implementations.</figcaption></figure><p><strong>SGD(OH MY.. THIS ONE TOPIC THOSE WHO’RE INTO ML/DL THEY KNOW ITS IMPORTANCE QUITE A LOT)</strong></p><p>Ladies and Gentlemen , I present to you Stochastic Gradient Descent(aka SGD) , this incredibly amusing , astonishing any AAA’S validation you wanna give it its justified believe me(you should as m batmen). Plain SGD estimates the direction of improvement from a small batch, which is a noisy, honest-but-incomplete glimpse of the truth. That noise is a feature: it keeps the search wandering and stops it from locking onto the first comfortable hollow it finds. We run it at a startlingly large learning rate of 30(basically giving reference to a project , I’m personally making which is on AWD-LSTM trained on WikiTest-2(via huggingface). This only works because of the clipping(gradient clipping) above. A bold stride is permitted <em>because</em> a hard cap exists on how far any single step may go. Window lengths are also randomized (usually the base length, sometimes half, plus Gaussian jitter), so the model’s learning does not synchronize with an arbitrary rhythm in the data, and the learning rate is scaled by (seq_len / bptt) to keep short windows from being over-trusted.</p><p>So , in this AWD-LSTM , we mainly use two further complexities of SGD</p><ol><li>. ASGD(Average Stochastic Gradient Descent):</li></ol><p>The cleanest way to understand ASGD is to notice that a training run suffers from <strong>two different kinds of error</strong>, and they need opposite medicine.</p><ul><li><strong>Distance error (bias).</strong> Early on, the weights are simply far from a good solution. The cure is to <em>move</em>: take big, bold steps downhill.</li><li><strong>Jitter error (variance).</strong> Later, the weights have arrived in the right valley, but they will not sit still. Each step is computed from a small noisy batch, and with a large learning rate every step kicks the weights a little off-centre. The iterates orbit the valley floor instead of resting on it. The cure here is not to move more. It is to <em>stop trusting any single step(just like you shouldn’t trust your exes).</em></li></ul><p><strong><em>(WELL THERE’S A THING CALLED BIAS AND VARIANCE TRADEOFF , WOULD LIKE TO RECCOMMEND YOU TO STUDY THE COMPUTATION VIA STATSQUEST(he’s good honestly , real good))</em></strong></p><p>Averaging is the cure for jitter. <strong>ASGD</strong> keeps a running mean of its recent iterates and evaluates with that mean rather than the latest weights(well in the below picture its just a further more computed/mathematical understanding of the term)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*mSCulAz4O1VQXxw2_9E-3g.png" /></figure><p>also those who don’t know Polyak-Ruppert Averaging: Averaging a stochastic method that forms estimators from the average of past iterates, offering improved convergence and variance reduction(as its based on the sequential data forms that’s why we use this averaging).</p><p>NT-ASGD(well quite a long form of words , Non-Monotonous Truncating Average Stochastic Gradient Descent):</p><p>Everything in NT-ASGD exists to answer one question: <strong>has the dominant error switched from distance to jitter yet?</strong> Start averaging too early and you smear a still-moving trajectory. Start too late and you waste epochs bouncing around a valley you already found. The best switch point differs for every model and dataset, and the paper’s contribution is to stop guessing it and have the training run <em>notice</em> it.</p><p>Let me introduce a example for you to get a clear glimpse of what is “exactly NT-ASGD”.</p><p><em>Imagine walking to a campsite through thick fog. For the first part of the trip you are </em><strong><em>travelling</em></strong><em>: every stretch of walking brings you closer. Eventually you </em><strong><em>arrive</em></strong><em>, but you cannot see that you have, and your steps are noisy, so you keep shuffling around the camp instead of standing on it. Once you have arrived, the best guess for the centre of the camp is not wherever your foot happens to land. It is the </em><strong><em>average of all your shuffling spots</em></strong><em>.</em></p><p><em>That is exactly the ASGD idea, and the whole difficulty is one question: how do I know I have arrived? Start averaging too early and your average is dragged back by the early walking spots. Start too late and you waste time shuffling around a camp you already found. The right moment differs for every model and dataset, so NT-ASGD stops guessing it and lets the training run notice.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*uYcaAk__l4WofkQvrpY1uA.png" /></figure><h3>“In plain words: <em>I am worse than anything I had already achieved a week ago.</em> One bad day cannot trigger it, because the comparison point is deliberately old”.</h3><h3>AdamW(separating how you learn from how much you forget):</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*jfR_njMFu6JZ7tBP0jKcQA.png" /><figcaption>mathematical expression for AdamW optm</figcaption></figure><p>Its made up of from Adam optm only , recurrent weights get gradients summed over (T) steps, which are large and spiky, and rare-word embeddings get tiny, sparse gradients. Adam’s per-parameter step sizes help both. But AdamW differs because of having a plain L2 regln inside it , so the decay is divided by (root v(moment RMSprop grad), so weights with big gradients (like W{hh}) are decayed <em>less</em>. AdamW applies (lambda*theta) directly, so every weight shrinks at the same relative rate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/327/1*zKYQe-dbSJhl9sgk3hfdAQ.png" /><figcaption>this if for weight decay</figcaption></figure><p><strong>Weight Decay</strong>: Let’s first know why does it matter in RNN’S , so as we have W(hh) a recurrent matrix as it is applied into every step , so its scale decides to whether the dynamics should blow up or fade away. Decay keeps it from growing, which also keeps gates out of saturation.</p><p><strong><em>AR AND TAR(Activation Regularization and Temporary Activation Regularization)</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/938/1*YIJbFrqjQ-iaDNQ1X5t_rQ.png" /><figcaption>Don’t let the output of the network grow large. In the loss</figcaption></figure><p>So , lets first take interest into AR(assault rifle yay) , AR keeps the activations in the middle, responsive part of the curve. Think of it as keeping the volume at a level where you can still turn the knob.</p><p>Here’s another data thingy which you can see to learn by seeing how it changes as we know the range of hyberbolic tangent(tanh) is -1 to 1 , so it’ll try to squeeze numbers in b/w that range given of the activation function</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*4azB7P7WvcB9ZDq5CiHxoA.png" /><figcaption>Tan(h) &gt;&gt;&gt;&gt; any other activation fn to exist.</figcaption></figure><p>Now , lets get our attention(my next blog will be on this) into TAR(don’t jerk the wheel): Language changes slowly. The topic of a paragraph, the tense of a sentence and who the subject is don’t flip from word to word. So the memory should change <em>gradually</em> too</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*eoNoWOBWNceF5OHlsILPbg.png" /><figcaption>well beta and alpha are just hyperparams used for better scaling of the activation fn</figcaption></figure><p>AR wants (h) to be <strong>small</strong>. TAR wants (h) to be <strong>steady</strong>. Taken alone, each has a lazy solution: set (h = 0) forever, which is both small and steady. But then the model predicts nothing. The main loss, cross-entropy, pulls the other way because it needs informative states. The knobs <strong>(alpha)</strong> and <strong>(beta)</strong> set the balance: <strong><em>be informative, but calm and smooth.(THIS IS THE MAIN DIFFERENCE B/W THE TWO ALSO HERE’S THE CODE DIFFERENCE)</em></strong></p><pre>total += c.alpha * float(np.mean(xd ** 2))              # AR<br>total += c.beta  * float(np.mean((raw[1:] - raw[:-1])**2))  # TAR</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=17de6e15e5e2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MATHEMATICS FOR MACHINE LEARNING]]></title>
            <link>https://medium.com/@aarav.dabral/mathematics-for-machine-learning-f4157733aa34?source=rss-292bc66f6ebd------2</link>
            <guid isPermaLink="false">https://medium.com/p/f4157733aa34</guid>
            <category><![CDATA[mathematics]]></category>
            <dc:creator><![CDATA[Aarav Dabral]]></dc:creator>
            <pubDate>Fri, 17 Apr 2026 17:53:58 GMT</pubDate>
            <atom:updated>2026-04-17T17:53:58.209Z</atom:updated>
            <content:encoded><![CDATA[<p>So , this blog is all about the necessity of mathematics and its various domains which are used in machine learning industry(I’ll use ml for the short basis).</p><p>“every thing has math in it polymath/philomath”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/699/1*CGqwbIvXwys8BPK9DXlMQg.jpeg" /></figure><p>In ml mainly these specific domains of mathematics is used</p><ul><li>Linear Algebra(requires focus + grind)</li><li>Calculus( it has two sub-parts , I’ll discuss both separately)</li><li>Probability(just remember bayes , also develop an intuition)</li><li>Statistics(for huge data’s in ml)</li></ul><h4>LINEAR ALGEBRA</h4><p>This is really something which requires to revise the formula’s(yes lots of them) , daily practice and that’s it you’re well set to boom linear algebra.</p><p>Resources(which I personally preferred and really the best)</p><ul><li>Gilbert Strang Lectures(MIT 18.06)-available on youtube</li><li>Book(Gilbert Strang’s Application of Linear Algebra)</li><li>a bonus tip(do make notes and learn eigen values , vectors , Single Value Decomposition ie SVD and 4 fundamental subspaces)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*Lk4VIjVjyycxVcVzGF__6A.jpeg" /><figcaption>do the competition of this asap</figcaption></figure><h4>Calculus</h4><p>Single Variable Cal- Basically revolves around differentiation and integration(basic ones which you have already done in your high school) , its easy. Newton Aproxx , Riemann Sums and dealing with simple calculus that’s it.</p><p>Resources</p><p>For Video Lectures-MIT 18.01 available free on yt.</p><p>For Book Solving(may or may not be needed)- Thomas Calculus.</p><p>Multi Variable Calculus- now this is something you’ll really need to focus on for the calculus based part of ml. Give good amount of intuitive understanding towards the line integral and the area integral concepts and you’re cool enough.</p><p>Resources</p><ul><li>For Video Lectures+ Concept Clarity: MIT 18.02</li><li>For Book Solving(indeed needed according to me)-Thomas Calculus</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*pz_PWaQAleSZEckkHXQrZg.jpeg" /><figcaption>therapist.</figcaption></figure><h4>Probability</h4><p>“Probability theory is nothing but common sense reduced to calculation”. Also marvel is lately using all the concept of probability which you’ll be learning for ml on your long journey obviously. Make sure the concept of permutations and combinations , also bayes axioms stays with you it’ll help you on the long term.</p><p>Resources</p><ul><li>You’ll according to me just need to understand the concept of seriously huge gambling that’s it(jk) MIT 6.041 is the course for probability.</li><li>or , statquest(never tried those but heard they good too)</li></ul><h4>Statistics</h4><p>“a gambler is always having statistical plans in his sleeves” — anon</p><p>This subject of mathematics is really about life lessons too , how many times the frequencies of your tries will increase the volume of your success. So , you obviously have to do it for ml no choice for that but its also a good thing to learn about life lessons along the way ain’t it??.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*X2AMmJ5LCNc5BRPlSA17UQ.jpeg" /><figcaption>the grammar of science.</figcaption></figure><h4>CONCLUSION</h4><p>I personally while writing this blog has completed calculus and linear algebra but I know all of these because while learning all the topics , I can vouch that you’ll be damn curious about mathematics in general and be a cool guy with mathematics. Hope you got a basic brief understanding of mathematics used in machine learning. Also , just try seeing mathematics more as a tool for life and less as a subject.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f4157733aa34" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[FREAK/WEIRDO]]></title>
            <link>https://medium.com/@aarav.dabral/freak-weirdo-1afa70b44438?source=rss-292bc66f6ebd------2</link>
            <guid isPermaLink="false">https://medium.com/p/1afa70b44438</guid>
            <category><![CDATA[philosphy]]></category>
            <category><![CDATA[nerd-culture]]></category>
            <dc:creator><![CDATA[Aarav Dabral]]></dc:creator>
            <pubDate>Mon, 29 Dec 2025 11:00:07 GMT</pubDate>
            <atom:updated>2025-12-29T11:05:22.086Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/735/1*tVNmljpy9TG81jgEQyOy3A.jpeg" /><figcaption>Even people say while grinding you’re a mental freak</figcaption></figure><p>Alright , so this blog is kind of related to weirdness , to a person who’s a freak. I’ll categorize this word into several different perspectives with respect to whatever I have learnt about , studied about and most importantly experienced about. Also , just for making you quite indulging into this I have used some players from sports field , also some physics terminologies(I guess I haven’t really used them) which ain’t quite difficult to interpret according its just basic modern physics.</p><ol><li>THE NERD KID: Alright the best category or the best persona any person could have according to me. Now , here is the fact why some random person would point you out and say “Oh , look out the freak is coming”. This tells you’re basically on a path to either suffocate in your life further on(So , this is from the reference from the book “The Monk who sold his Ferrari” where even the nerdiest lawyer suffered an heart attack so yes suffocation will still be there) or you’re going to outshine everyone who comes ahead as an obstacle. Basically , you have so many neurons kicking inside your brain so whenever you meet a person , you just speak about anything which looks fascinating in your “eyes” not from the point of view of the secondary person in this case.</li><li>MAMBA MENTALITY/EVIL MIND: Alright here comes all the references I was talking about at the starting of this blog. When a person heard about the term mamba mentality he/she quite remembers Kobe Bryant yes unfortunately he died in 2020 but his mental game was so strong that even when his ankle got twisted he still did those two free-throws and won it for the Lakers(a franchise in NBA). For the cricket fans obviously it would be Steven Smith from the Ashes which were in England where Jofra Archer was bowling quite good with raw pace but still Smith stood up and just did what the cricket experts say “Australian Mindset”.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*Zw7P_s0NJR1ie36hF8l_Tw.jpeg" /></figure><p>Now for the Devil Mindset its actually Mikhail Tal(my favourite player of all time) , I would be talking about here , He always used think all the unfavourable outcomes , you know when people say be delusional and all , that multiverse is such a cool thing and all kinds of curious thoughts starts popping inside your mind. That person just used to outplay his opponents by being potentially the “Satan”. So basically He and all the above athletes mentioned above are just freaks , they have built themselves like someone who literally thinks all the wrong outcomes , who just really don’t care so that declares themselves as the mentality freak???. All you want is to feel either entering the time dilation or the sixth dimension checking those multiple paradoxes occuring. That’s suppose to be freak of the god know’s what type of mentality or nerdiness</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/465/1*COK4tfcpWol2f9nP5vfaVw.jpeg" /></figure><p>3. CREEPS: Honestly , while just thinking about this also while writing and jotting all the points I could list down. I realized its really about communication between a girl and a boy(no experiences will be showcasing in this one). Its really about the book “Men are from Mars and Women are from Venus” both of their thinking sense is quite different , but when someone says you’re a creep or a weird person either you’re lying in those above mentioned categories or you really are a douchebag kind of a person.</p><p>I know 2025 is ending , everybody is hoping out there in their respective residencies that 2026 will be my peak year , that 2026 will be my comeback year. Ladies and Gentlemen , honestly speaking , I have also planned the same stuff as you guys have done but it’ll be about either you become a freak(the nerd kid) or you become a douchebag</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*TY7oxuX-kDDwFRuoOKVp8g.jpeg" /></figure><p>So , at last I really want to be Dr.Richard Feynman(that type of freak) or obviously after hitting the seasons of the batman grind , I’ll potentially turn out to be someone but not a douchebag.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1afa70b44438" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[WHO AM I]]></title>
            <link>https://medium.com/@aarav.dabral/who-am-i-f81e984cded9?source=rss-292bc66f6ebd------2</link>
            <guid isPermaLink="false">https://medium.com/p/f81e984cded9</guid>
            <category><![CDATA[life-lessons]]></category>
            <dc:creator><![CDATA[Aarav Dabral]]></dc:creator>
            <pubDate>Thu, 25 Dec 2025 14:13:03 GMT</pubDate>
            <atom:updated>2025-12-25T14:13:03.395Z</atom:updated>
            <content:encoded><![CDATA[<h3>WHO AM I?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*JM9IEpXoL8cac-FgIVVSsg.jpeg" /></figure><h3>Alright so yes basically this blog is about my life changes shall I say?. Like normally you’ll see those transformation reels , shorts and those movies. But , it happened to me in real life I was just another guy you can honestly say the desperate , the wanna-be one. I was in my school life had less confidence especially my last two years where I chose PCM as a stream. Things just didn’t work out.</h3><p>11TH</p><p>So, yes chose PCM(only because your lad watched interstellar back then in his young prime). But I didn’t had the confidence because yes my crush dumped because no good looks no boyfriend scheme I guess , yes so after that moment probably in June 2023(yes somehow luckily I remember the month). Everything changed , I did cardio and strength training. Then I realized that confidence really comes after you achieve something in any aspect , whether its academic , your physique or it could any form of sports. The latter half of 11th was literally the prime form of mine , academically , physically , emotionally , its just everything. Yes learnt some hard shit too , that people don’t care until you tell the truth them. It’s basically like the book “The Subtle Art Of Not Giving A F**k”. When you actually tell them the truth about their own life , people go mad at you(abusive words , its common here).</p><p>12TH</p><p>THE YEAR WHERE EVERYTHING AGAIN WENT WRONG</p><p>So , the worst things which for every PCM student is not getting selected for IIT. Same thing happened to me messed up advance quite literally I guess. Result came , cried a lot(seriously I cried for I don’t how much hours shall I say). That day I realized maybe I didn’t put up such efforts that’s why God decided to do more hard work. But , It was like I sacrificed my social life , my friends , my body once more on the verge of being lazy.</p><p>PRESENT TIME</p><p>So here we are writing this on 25th on Christmas. Your guy has his last of his first end semesters exam tomorrow. But it’s been a truly roller-coaster ride for me and since the day I joined X its even more metaphorical. At beginning I used to see , people doing 100 days of code and things like that. So , I was like I won’t be in the typical mindset of only doing code , I would like to be unique I suppose so yes we started Project100 where I would do my gym , code , reading books and blogs , also watching and learning about new things from You tube(Internet). During those roughly around 60 days(2 Months approximate) I met some cool people , some very helpful seniors and obviously “CHAUKDI” recently. This taught me about many things , from being a complete stupid , idiot looking guy to just able to reshaped himself(ain’t at my best till now…). So if you see me depressed or something probably I would be in the memories of my past , because my present self is more optimistic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*BSYtnKAX0rCV9A40tj_nPQ.png" /></figure><p>At last whenever in life you’ll be reading this , I hope you end up well in your life ahead or in your current lifetime.</p><p>Cheer!.</p><p>Lakshay.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f81e984cded9" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>