<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vinayak on Medium]]></title>
        <description><![CDATA[Stories by Vinayak on Medium]]></description>
        <link>https://medium.com/@vinayak1672006?source=rss-768eaa045335------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*rQ0aKc6Oq2oolQ-lzF1Naw.png</url>
            <title>Stories by Vinayak on Medium</title>
            <link>https://medium.com/@vinayak1672006?source=rss-768eaa045335------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 01:10:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vinayak1672006/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building an LLM From Scratch: The Mechanism That Changed AI Forever, Implemented From Zero]]></title>
            <link>https://pub.towardsai.net/building-an-llm-from-scratch-the-mechanism-that-changed-ai-forever-implemented-from-zero-15cb1127ed27?source=rss-768eaa045335------2</link>
            <guid isPermaLink="false">https://medium.com/p/15cb1127ed27</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Vinayak]]></dc:creator>
            <pubDate>Wed, 10 Jun 2026 16:01:02 GMT</pubDate>
            <atom:updated>2026-06-10T16:01:02.062Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JEzxcHMyH8TYAfdJypoW0w.png" /><figcaption>Attention</figcaption></figure><h4>After training the embeddings in the previous part, now comes the most important part of LLMs that shifted how the entire field thinks about sequence modeling.</h4><h3>Self Attention (What and Why)</h3><p>Building this from character level tokenization all the way to self attention has taught me much in depth about the mathematics used in LLMs, particularly calculus and linear algebra.</p><p>Basic idea of self attention was that to make the computer know which word is relevant to which word. Let’s take the most common example:<br>“River bank flows” and “Money bank grows”. In these two, bank is the common word but very different meaning.</p><p>So, if a traditional prediction method is used. it would basically never be able to distinguish between these 2 words. So what was done was surprisingly simple in hindsight.</p><p>We check each word’s dependency to the preceding words (not only the single previous word like this example but sequence of words like previous 16 words), and see how is the target word relevant to the preceding words and what’s the most relevant word to it. And since machines don’t understand language, we represent all of this as numbers.</p><h3>How Self Attention works</h3><p>So, coming to the most important part, how to implement it and make the machines understand the relevance of words with each other.</p><p>Since we know that words are represented in the form of embeddings, it becomes easier for processing. So, instead of representing the words as their own embeddings we can represent as each word’s relevance with other words, for example:</p><p>Instead of<br>“River = Embeddings of river” and “Bank = Embeddings of bank”</p><p>We can represent like:<br>“River = 0.8(River)E + 0.2(Bank)E” and “Bank = 0.4(River)E + 0.6(Bank)E”.</p><p>This shows how each word is relevant to each other where 0.8 and 0.2 are the attention weights showing how much each word contributes to the final representation.</p><p>Now the question comes how to make the word embedding like these.<br>As we know that to find the relevance or similarity between 2 embeddings best way is to do dot product between 2 embeddings.</p><p>So the formula becomes like:<br>(River)E = (river . (river)T)River + (river . (bank)T)Bank &amp;<br>(Bank)E = (bank . (river)T)River + (bank . (bank)T)Bank</p><p>(Here T is transpose of the matrix)</p><p>To normalize these weights so that we can see in the scale of 0 to 1 as how each embedding is contributing to the other embedding, we pass these to the SoftMax function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fqc2WeklTg3CTVVC--CVQQ.png" /><figcaption>A simple diagram for intuition.</figcaption></figure><p>If we see this diagram, we can see that a single word plays 3 different role.</p><p>Let’s take a word money, we take the dot product of the embeddings of money with all the other words, where money is the QUERY and the other words play the role of KEYS. Here QUERY vector asks the KEY vector for how relevant the are to each other. We pass these scores through SoftMax to get the attention weights, then multiply them with the VALUE vectors to get the final representation. Here, we are finding the final VALUE of the embedding so it’s named as VALUE</p><p>Now we can categorize each words into three vectors QUERY- Q, KEY- K and VALUE- V, this also introduced learnable parameters: W_Q, W_K and W_V which the model trains through forward and backward propagation.</p><p>Summarizing all this mathematics into one formula we get is:</p><blockquote>Attention(Q, K, V) = SoftMax(QKᵀ)V</blockquote><p>But wait that’s not the correct formula according to that Attention is All You Need. But in the actual paper, it something like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/1*MugPnWx-tXz3NQpp2kiyXw.png" /></figure><p>The difference is the division of root of the dimensions of the embeddings, let’s understand why.</p><h3>Significance of √dₖ</h3><p>SoftMax is best function to convert group of numbers into percentages, but it has one drawback.</p><p>While applying SoftMax, we have to make sure that the variance of the numbers is the lowest to get the best and optimal result.<br>If the variance is too high or even high, the problem of vanishing gradients arises as the large scores saturate toward 1 where as while smaller scores collapse toward 0.</p><p>So, before adding SoftMax function to the product of embeddings we need to scale them first.</p><p>We know that as the dimensions of embeddings increases, the variance between them increases too. So, we need to scale them, but why only √dₖ?<br>This question arises.</p><p>There is a simple mathematics rule about variance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/914/1*7ogp7cMn-v1Tf6q9ia2LsA.png" /></figure><p>In our case, the dot product of Q and K scales the variance by d_k. So dividing by √d_k brings it back to 1 and keeps the softmax inputs in a stable range.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QFpF53xiXg_Xe-P2BAJDvg.png" /><figcaption>Generated</figcaption></figure><h3>Code Implementation</h3><p>Starting with the code implementation, we start with Forward propagation first where the model computes all the dot products without knowing the actual answer.</p><h4>Forward Pass</h4><pre>def forward(contextToken):<br>    X = input_embeddings[contextToken]<br># each word is now projected into 3 diff vectors i.e. Query, Key, Value.<br>    Q = X @ W_Q # (8, 32) @ (32, 32) = (8, 32) (Query vector)<br>    K = X @ W_K #      &quot; (Key vector)<br>    V = X @ W_V #      &quot; (Value vector)<br>    scores = Q @ K.T / np.sqrt(d_k) # K.T is just the transpose of K (8, 32)@(32, 8) = (8,8)<br>    mask = np.triu(np.ones_like(scores), k=1)<br>    scores = np.where(mask, -1e9, scores)  # this is used to mask the scores of the words that are not in the context.<br># it makes the matrix of perfect size of no of context tokens x no of context tokens and Cell [i, j] = dot product of Q_i and K_j = &quot;how relevant is word_j to word_i?&quot;<br>    attn_w = softmax(scores) # (8,8)<br>    context = attn_w @ V # Value of each words to the embeddings i.e. [8,8] @ [8,32] = [8,32]<br>    last_context = context[-1] # we need only the last context from the context vector to predict the next word (32,)<br>    logits = last_context @ W_O # logits are basically the product of context representaton against each word&#39;s learned output weight vector (32,) @ (32, vocab size) = (vocab size,)<br>    probs = softmax_vectors(logits) # it gives most probable next word.<br><br>    cache = dict(X = X, Q = Q, K = K, V = V,<br>    scores = scores, attn_w = attn_w, context = context, last_context = last_context, logits = logits, probs = probs)<br>    return probs, cache</pre><p>Here we applied causal masking so that the model only sees the words before the current position, not the ones after it. Without this, word 1 could attend to word 8 which hasn’t been generated yet. That would be cheating.</p><p>The scores matrix is an 8x8 grid where cell [i, j] answers the question: how relevant is word j to word i? We divide by √d_k before SoftMax to keep the variance stable, as explained above.</p><p>The cache stores every intermediate value from the forward pass. We need all of them during backpropagation to compute the gradients.</p><h4>Backpropagation</h4><pre>def backward(cache, target_idx, lr, ctx_tokens):<br>    global W_Q, W_K, W_V, W_O<br>    probs = cache[&#39;probs&#39;]<br>    logits = cache[&#39;logits&#39;]<br>    last_context = cache[&#39;last_context&#39;]<br>    context = cache[&#39;context&#39;]<br>    attn_w = cache[&#39;attn_w&#39;]<br>    scores = cache[&#39;scores&#39;]<br>    V = cache[&#39;V&#39;]<br>    K = cache[&#39;K&#39;]<br>    Q = cache[&#39;Q&#39;]<br>    X = cache[&#39;X&#39;]<br><br># Starting from right to left as backpropogation is just the opposite of forward pass<br>    d_logits = probs.copy()  # d_logits is just the derivative of logits w.r.t. target_index which basically how wrong the predicted vector is<br>    d_logits[target_idx] -= 1.0 <br><br>    last_ctx = context[-1]<br><br>    dW_O = np.outer(last_ctx, d_logits)<br><br>    d_last_ctx = W_O @ d_logits<br><br>    d_context = np.zeros_like(context)<br>    d_context[-1] = d_last_ctx<br><br>    d_attn_w = d_context @ V.T<br>    d_V = attn_w.T @ d_context<br><br>    d_scores = np.zeros_like(attn_w)<br>    for i in range(seq_len):<br>        a = attn_w[i]<br>        jac = np.diag(a) - np.outer(a,a)<br>        d_scores[i] = jac @ d_attn_w[i]<br><br>    d_scores /= np.sqrt(d_k)<br><br>    d_Q = d_scores @ K<br>    d_K = d_scores.T @ Q<br><br>    dW_Q = X.T @ d_Q<br>    dW_K = X.T @ d_K<br>    dW_V = X.T @ d_V<br><br>    d_X = d_Q @ W_Q.T + d_K @ W_K.T + d_V @ W_V.T<br>    W_Q -= lr * dW_Q<br>    W_K -= lr * dW_K<br>    W_V -= lr * dW_V<br>    W_O -= lr * dW_O<br><br>    # train embeddings too <br>    global input_embeddings<br>    for j, tok_idx in enumerate(ctx_tokens):<br>        input_embeddings[int(tok_idx)] -= lr * d_X[j] # here we convert the token index to int because it is a numpy array and we need to subtract the gradient from the embedding matrix.</pre><p>Backpropagation here follows one simple rule. For any operation Y = A @ B, the gradient flows as dA = dY @ B.T and dB = A.T @ dY. We apply this rule in reverse order through every step of the forward pass.</p><p>The hardest part is the SoftMax gradient. SoftMax is not a simple element-wise function so its gradient is a full Jacobian matrix.</p><p>Softmax takes a vector and outputs a vector, so its gradient is a matrix showing how every output changes when every input changes. This matrix is called the Jacobian. For softmax it works out to diag(a) - outer(a, a), where a is the attention weight vector for that row.</p><p>Everything else is just matrix multiplication rules applied in reverse.</p><p>One important detail: we compute the embedding gradient before updating W_Q, W_K, W_V. If you update the weights first and then compute the gradient, you are using the wrong weights. The gradient must use the same weights that were active during the forward pass.</p><pre># Training loop<br>Epochs = 15<br>try:<br>    for epoch in range(Epochs):<br>        random.shuffle(sequences)<br>        total_loss = 0.0<br><br>        for i, (context, target_idx) in enumerate(sequences):<br>            lr = 0.01 / (1 + 0.0001 * i)<br>            probs, cache = forward(context)<br>            loss = -np.log(probs[target_idx] + 1e-10)<br>            total_loss += loss<br>            backward(cache, target_idx, lr, context)<br><br>            if i % 20_000 == 0:<br>                print(f&quot;Epoch {epoch+1} | step {i:&gt;6}/{len(sequences)} | &quot;<br>                      f&quot;loss {total_loss / (i + 1):.4f}&quot;)<br><br>        avg_loss = total_loss / len(sequences)<br>        print(f&quot;Epoch {epoch+1} complete. Avg loss: {avg_loss:.4f}&quot;)<br><br>except KeyboardInterrupt:<br>    print(&quot;\nTraining interrupted — saving weights...&quot;)<br><br>finally:<br>    np.save(&quot;W_Q.npy&quot;, W_Q)<br>    np.save(&quot;W_K.npy&quot;, W_K)<br>    np.save(&quot;W_V.npy&quot;, W_V)<br>    np.save(&quot;W_O.npy&quot;, W_O)<br>    print(&quot;Weights saved.&quot;)<br><br>def predict_next(words_in, top_k=5):<br>    ctx = [word_to_idx[w] for w in words_in if w in word_to_idx]<br>    ctx = ctx[-seq_len:]          # take last SEQ_LEN tokens<br>    if len(ctx) &lt; seq_len:<br>        ctx = [0] * (seq_len - len(ctx)) + ctx   # pad with index 0<br> <br>    probs, _ = forward(ctx)<br>    top_indices = np.argsort(probs)[::-1][:top_k]<br>    print(f&quot;\nContext : {words_in}&quot;)<br>    print(f&quot;Top-{top_k} predictions:&quot;)<br>    for idx in top_indices:<br>        print(f&quot;  &#39;{idx_to_word[idx]}&#39;  p={probs[idx]:.4f}&quot;)</pre><p>We also train the embeddings here, not just the attention weights. Every token in the context window gets its embedding nudged based on how much it contributed to the wrong prediction.</p><h3>Outputs After Training</h3><p>After 15 epochs on 969,000 sequences the loss dropped from 7.6 to 5.4. To put that in perspective, random guessing on a 24,000 word vocabulary gives a loss of around 10.1. The model is roughly 100x better than random chance.</p><p>To test what the model actually learned, I ran random next-word prediction tests across the corpus.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*bj-2cmwtb1DKITxYuE7BwQ.png" /><figcaption>Terminal</figcaption></figure><p>The hit rate varies between 1 and 5 out of 10 depending on the batch. The misses are mostly unpredictable words like ‘jealousy’ or ‘pestle’ that even a human would struggle to guess from 8 words of context. The hits show the model has learned grammatical structure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZsWMpScNQ5W2NJ6NNt9Kaw.png" /><figcaption>Terminal</figcaption></figure><p>To understand what attention is actually doing internally, I built a visualization that tracks how much each word’s embedding changes during gradient updates for a specific prediction.<br>Each gradient step means one full forward pass followed by one backpropagation update across all embeddings in the context window.</p><p>The context given was ‘above all don’t lie to yourself the man’ with the target word being ‘who’.</p><p>The L2 drift column shows how much each word’s embedding vector moved during 50 gradient steps. The results are striking:</p><ul><li><em>“‘man’ drifted 0.508 — by far the largest update”</em></li><li><em>“‘yourself’ drifted 0.021”</em></li><li><em>“‘lie’ drifted 0.021”</em></li><li><em>“‘above’, ‘to’, ‘the’ drifted essentially zero”</em></li></ul><p>The cosine similarity of ‘man’ dropped to 0.9925 meaning its direction in embedding space actually shifted, while every other word stayed at 1.0000 meaning they barely moved at all.</p><p>This confirms that attention is working exactly as needed and all the steps of forward pass, backpropagation, updates, etc are working properly.</p><p>The model cannot yet predict with high accuracy because the transformer block is incomplete. It is currently missing positional encoding, residual connections, a feed-forward layer, and multi-head attention. These are not optional extras, they are what makes a transformer a transformer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uSXbSiO2lcLDPITOEfskpQ.png" /><figcaption>Generated</figcaption></figure><p>In the next part I will revisit some earlier decisions like embedding dimensions and sequence length, and build the complete transformer block.</p><p>Part 4 is next. Positional encoding, residual connections, feed forward layer, and multi head attention. That is the full transformer block.<br>If this was useful, follow along. Every part has working code, honest documentation of what broke, and no shortcuts.</p><p><a href="https://github.com/07Codex07">Github</a><br><a href="https://www.linkedin.com/in/vinayak-sahu-8999a9259">Linkedin</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=15cb1127ed27" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/building-an-llm-from-scratch-the-mechanism-that-changed-ai-forever-implemented-from-zero-15cb1127ed27">Building an LLM From Scratch: The Mechanism That Changed AI Forever, Implemented From Zero</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an LLM From Scratch: I Trained Word Embeddings on Dostoevsky. Here’s What I Found.]]></title>
            <link>https://pub.towardsai.net/building-an-llm-from-scratch-i-trained-word-embeddings-on-dostoevsky-heres-what-i-found-b3169c1ae674?source=rss-768eaa045335------2</link>
            <guid isPermaLink="false">https://medium.com/p/b3169c1ae674</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Vinayak]]></dc:creator>
            <pubDate>Wed, 13 May 2026 18:01:00 GMT</pubDate>
            <atom:updated>2026-05-13T18:01:00.752Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5DN0p1VBpFk6XRR6OCrjsg.png" /></figure><p>In my past article I wrote about how I implemented Character Level Tokenization over a very small corpus and understood the most basic and initial phases of NLP and base of LLMs.</p><p>This time I implemented the next step towards Modern NLP and LLM system “Embeddings” and implemented it from scratch and trained it over my laptop which took 30+ hours.</p><p>Let’s dive into the concept, mathematics and code that i understood in my process of building LLM from scratch and implementing the embeddings to my corpus of nearly 1 million words.</p><h3>What are Embeddings?</h3><p>Embeddings are just the vector representation or format of each word to make the machine or model understand the word instead of making it memorize it with a unique number given to each unique words in a corpus.</p><p>For simple breakdown of concept of embeddings, let’s take an example:</p><pre>Text = &quot;dog cat&quot;<br>dog = (0.01, 0.04, 0.2)<br>cat = (0.01, 0.03, 0.2)<br># dog and cat are given similar vectors because they appear in similar context</pre><p>So, now these two words can easily be represented in a 3D space in the form of vector, and these dense vectors allow neural networks to capture semantic relationships between words instead of treating every word as an isolated symbol to make them related to each other.</p><blockquote>It transforms words into a format like:<br>king − man + woman ≈ queen</blockquote><p>Not because anyone programmed it but semantic relationships emerged geometrically.</p><p>Those shocked researchers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EKnIKEy51tE7frKCmKv_2Q.png" /></figure><h3>Why Early NLP Models Needed Embeddings ?</h3><p><strong>So, first and most important question “WHY”<br></strong>Statistical language models only stored how frequently consecutive words appeared together.<br>They had no understanding of what those words actually meant.</p><p>So, if the Tokenizer model learns ‘dog barks’,<br>it learnt nothing about ‘cat meows’.</p><p>That’s just a single stated problem which is huge because without solving this, nothing could have been proceeded in AI world as context is the main concern till now.</p><p>But for more problems were like data sparsity as it knows only what has been trained or what was there in training dataset, if anything outside of that came to the model, it predicted very close to 0.</p><p>Explosion of memory as n increased in n-gram models.</p><p><strong>Curse of One-Hot-Encodings</strong>:<br>If vocabulary size = 50,000:<br>1) Each word becomes 50,000-dimensional vectors.<br>2) Since one-hot vectors are orthogonal to each other, the model cannot naturally represent semantic similarity between related words like ‘dog’ and ‘cat’.</p><p>All these problems were needed to be solved to make the NLP applicable in real life.</p><p>Then came a major breakthrough in 2003 Bengio, Neural Language Models. where they used neural networks to learn dense vectors.</p><h3>After Tokenizer and Before Embeddings</h3><p>After implementing the tokenizer from scratch, what I had was a system that tokenizes each character from the corpus and predicts what should come next with the help of bigram and SoftMax function.</p><p>But now I had to increase the corpus size vastly and characters don&#39;t carry semantic meaning and character-level modeling also required much longer sequences and made semantic learning slower, which pushed me toward word-level representations.</p><p>Being a Dostoevsky devotee, I scraped his 5 novels which were:<br>1. <em>The Brothers Karamazov<br></em>2. The Idiot<br>3. Short Stories by Dostoevsky<br>4. <em>The Possessed (The Devils)<br>5. Notes From the Underground</em></p><p>These gave me a corpus of nearly 1Million words for training the embeddings.</p><h3>How Do Embeddings Learn Meaning?</h3><p>This all starts with corpus of text that we have and extracting the unique words from it to build the vocabulary.</p><p><strong>Deciding dimensions of each vector.<br></strong>Embedding dimensions are a hyperparameter and a rough starting point is the 4th root of your vocabulary size but that gives you just 15 dimensions for 50,000 words which is too small to capture anything meaningful. In practice 100–300 dimensions is what most implementations use. Higher dimensions capture more context but take longer to train and can overfit on smaller datasets.</p><p>In my case i used 32 because of hardware constraints, not because it was the ideal number.</p><p>Then moving forward, once the dimensions are decided, each word is initialized to a random vector which are later trained so that words appearing in similar contexts get closer in vector space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QOq7iMqhWYxsrScFAS7pCg.png" /><figcaption>General representation of embeddings table</figcaption></figure><p><strong>Context Window for each word<br></strong>So, once the dimensions are fixed, we decide the max context size for each word and make two separate tables for context word’s embeddings and center word’s embeddings, the reason for two separate tables will be covered in the code section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cD6gRApNAPJLuKtDU9Yswg.png" /></figure><p><strong>Negative Sampling</strong><br>I added 5 fake samples for each real pair of context and center word by choosing the context word randomly for the same center word.<br>Something with no real relationship to the center word, like {Crime, Banana}.</p><p><strong>Training the Embeddings<br></strong>We pass these pairs for training where their untrained embeddings are fetched.</p><p>Dot products are taken to measure similarity and alignment.</p><pre>score = np.dot(center_vec, context_vec)</pre><p>If this score is high, the model thinks these words belong together. If low or negative, the model thinks they are unrelated.</p><p>Then the scores are passed through sigmoid function which makes them in the range of (0,1)<br>A very negative score maps close to 0, score = 0 becomes 0.5 and positive maps closer to 1.</p><p><strong>Loss Function</strong><br>The loss function penalizes the model when real word-context pairs receive low probabilities or when randomly sampled negative pairs receive high probabilities.</p><pre>Losspositive​ = −log(σ(Vc​ ⋅ Vo​)) # Model wants this probability close to 1.<br><br>Lossnegative ​= −log(1−σ(Vc​ ⋅ Vn​)) # Model wants this probability close to 0.</pre><p><strong>Gradients</strong><br>A gradient tells the model how much each number inside the vectors contributed to the prediction error and in which direction it should be adjusted to reduce the loss</p><p>Mathematically, gradients are the derivative of loss with respect to the score i.e. Prediction — Truth (this is only for sigmoid activation and binary cross entropy loss)</p><p><strong>Backpropagation<br></strong>It just understands error reduction and asks “Which number inside the vector caused the bad prediction”<br>i.e. How the gradients actually flow backwards to update and optimize the vectors.</p><p>How vectors actually change using learning rate is:</p><pre>vector = vector - learning_rate * gradient</pre><p>This is how actual learning happens, where vectors physically move in space.</p><p><strong>Training Loop:</strong><br>how many times we pass through the data and how loss changes over epochs. So, one epoch is basically going through this whole process over every pair of vectors for once.</p><p>Over multiple epochs Loss gradually decreases and the vector drifts towards the meaningful position in space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*CmxOG7wFdC-HSlMBAmonrA.png" /><figcaption>Loss over each epoch</figcaption></figure><p>I trained the model for over 20 epochs and with 5 negative samples for each pair which took 30+ hours to train, and the results were actually remarkable.<br>I was not expecting many changes because in actual Google’s research paper where they trained the Skip-Gram model with negative sampling had a massive amount of data and trained for much longer time which was not even comparable with mine.</p><p>Below are the vector representations of words from the corpus, before and after training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J4PoKUOw5IeWMuy_qV5Xcw.png" /><figcaption>These are the representation of vectors before training</figcaption></figure><p>Just observe the vector of murder and crime how far they are before training which would change in the next image which is representation of vectors after training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JW3YCFKPyQEi84bhcc93bw.png" /><figcaption>vector representation after training</figcaption></figure><p>After 20 epochs: School, university, and monastery cluster together, and murder moves close to crime, punishment, and death</p><h3>Why Skip-Gram not CBOW?</h3><p>So, there are two popular architectures for training word embeddings, Skip-Gram and CBOW (Continuous Bag of Words) and they are essentially inverses of each other.</p><p>CBOW takes multiple context words and predicts the center word. Skip-Gram does the opposite; it takes one center word and predicts the surrounding context words.</p><p>The reason I went with Skip-Gram is simple, it generates far more training pairs. A single center word with a window of 2 produces 4 pairs and across 1 million words, that compounds into even more millions of training examples giving the model more chances to learn from every word.</p><blockquote>This architecture was first introduced by Mikolov et al. in <a href="https://arxiv.org/abs/1301.3781">Efficient Estimation of Word Representations in Vector Space (2013)</a>, the paper that gave us Word2Vec.</blockquote><h3>Code Implementation</h3><p>Let’s get into the actual code and I’ll only explain the parts that actually matter.</p><p><strong>Processing:</strong></p><pre>text = text.lower()<br>text = re.sub(r&#39;[^a-z\s]&#39;, &#39;&#39;, text)<br>words = text.split()</pre><p>Lowercasing everything and stripping punctuation so the model doesn’t treat “Crime” and “crime” as two different words.</p><p><strong>Vocabulary and Tokenization:</strong></p><pre>word_to_idx = {w: i for i, w in enumerate(unique_words)}<br>tokens = [word_to_idx[w] for w in words]</pre><p>Every unique word gets a number, and the entire corpus becomes a sequence of those numbers. This is what gets fed into training.</p><p><strong>Two Embedding Tables:</strong></p><pre>input_embeddings = np.random.uniform(-0.5, 0.5, (vocab_size, embedding_dim)) / embedding_dim<br>output_embeddings = np.zeros((vocab_size, embedding_dim))</pre><blockquote>Why two embedding tables?<br>Looking closely, we can see that a single word plays 2 roles once as center embedding which is looked up in input_embeddings and when it acts as context word it is looked up in output_embeddings.</blockquote><p>Initially I trained the model with a single embedding table which didn’t give good results. Love’s embedding closely aligned with crime’s embeddings which made no sense. Then i studied what was failing and found out that when same word plays both the role at different point in training, gradients from both roles collide and interfere with each other during backpropagation (with single table).</p><p>In simple words this made every word look similar. This was changed by two separate tables that kept the updates clean and isolated.</p><p>After training is done, input_embeddings is what you save and use. That is your final word representation.</p><p><strong>Negative Sampling Distribution</strong></p><pre>word_freq = word_freq ** 0.75</pre><p>This one line is actually important. Raw word frequency would make common words like “the” or “and” dominate the negative samples.</p><p>Raising frequency to the power of 0.75 smooths that out, giving rare words a fairer chance of being sampled as negatives. This comes directly from the original Word2Vec paper.</p><p><strong>Training Loop</strong></p><pre>grad = prob - 1<br>input_embeddings[center_idx] -= learning_rate * grad * context_vec<br>output_embeddings[context_idx] -= learning_rate * grad * center_vec</pre><p>This is where actual learning happens. The gradient for real pairs is <br>“Prob-1”and for negative pairs it is just “prob”, this is basically the derivative of binary cross entropy loss with sigmoid activation which I mentioned earlier. Every update physically moves the vectors in space, pulling related words closer and pushing unrelated ones apart.</p><p><strong>Results after training</strong></p><p>Results after training were impressive. I already showed the PCA representation before training. Below are the actual cosine distances between the vectors after 20 epochs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/403/1*KcvCnJMRaDPSFMrAVVNSEA.png" /><figcaption>terminal output of the implementation</figcaption></figure><blockquote>Above image shows the actual cosine distances between the vectors with similar context like school was found similar with supper, university, monastery which are really good.<br>Distance between crime and punishment decreased drastically after training and same for crime and murder</blockquote><h3>Limitations</h3><p>32 dimensions are actually too small to understand the context with higher accuracy.<br>Real Word2Vec used 300 dimensions and 20 epochs are also relatively very few, higher epochs would align the vectors more accurately.</p><p>As I trained the model over single author, it made these embeddings carry Dostoevsky’s worldview.<br>A general language model would look different from this.</p><p>I implemented this model purely over NumPy loops which are slow.<br>Real implementation used Batched Matrix Operation which took days for google even in such a large amount of dataset as they ran over thousands of machines.<br>Whereas mine took 30+ hours even with such a small dataset compared to actual implementation by google.</p><h3>What I learned building this</h3><p>Embeddings do not predict anything on their own. They just give words meaning before passing them into the model.</p><p><strong>The pipeline so far:</strong></p><pre>Text → Tokenizer → Token IDs → Embeddings → [???] → Prediction</pre><p>That gap in the middle is attention. That’s where the model actually figures out which words should influence each other in a sequence.<br>Where actual prediction begins.</p><p>That’s what I’m building next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iXC-qVZ77QGpHdyo0TXRYg.png" /></figure><blockquote>Vinayak this side.<br>Building an LLM from scratch, one step at a time.<br>Follow the full code on <a href="https://github.com/07Codex07/Mini_LLM">GitHub</a> or connect with me on <a href="https://www.linkedin.com/in/vinayak-sahu-8999a9259">LinkedIn</a>.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b3169c1ae674" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/building-an-llm-from-scratch-i-trained-word-embeddings-on-dostoevsky-heres-what-i-found-b3169c1ae674">Building an LLM From Scratch: I Trained Word Embeddings on Dostoevsky. Here’s What I Found.</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an LLM From Scratch. Here’s Where It All Starts.]]></title>
            <link>https://pub.towardsai.net/building-an-llm-from-scratch-heres-where-it-all-starts-071d7f1ab870?source=rss-768eaa045335------2</link>
            <guid isPermaLink="false">https://medium.com/p/071d7f1ab870</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Vinayak]]></dc:creator>
            <pubDate>Sat, 02 May 2026 19:01:01 GMT</pubDate>
            <atom:updated>2026-05-02T19:01:01.316Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YAa4cgnKjynLvA2upPoHEA.png" /><figcaption>Similar but clean Terminal output</figcaption></figure><p>While exploring and building LLM from scratch without any modules, I started from the lowest but very important level i.e. Character Based Tokenization</p><p>In pre-2010, most of the systems were rule based and statistical(N-Grams) were used.<br>This method was used in early in NLP systems for spell checks, language detection and very basic text generation before the Byte Pair Encodings became dominant.<br>But after 2018, since the “Attention is All You Need” came and became the AI revolution most of the model shifted to Sub-Word tokenizers (BPE, WordPiece) and these character level tokenizers were abandoned from large languages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W-kO2EQem7zjcXIt9salRg.png" /><figcaption>Basic Timeline</figcaption></figure><h3>Why Character Level Tokenizers Were Actually Smart:</h3><ol><li><strong>No out of vocabulary problems (OOV problem solved).</strong><br>Since, character level tokenizers use each character and if they were trained in diverse text, there would be no problem of Out of vocabulary problem as all the characters would already be in the training set and probability matrix.</li><li><strong>Language Independent.</strong><br>In Character level encoding, there were no code change eve if we change the language of the training vocabulary because it would just do the same process with that language’s text.<br>Even though there is no code change, but still the language detection becomes complex like in Hindi there are complex character combinations which would require diff type of handling.</li><li><strong>Can handle Typos naturally.</strong><br>“helo” → still understandable, whereas in word level encoding, “helo” would give OOV which is handled by Laplace smoothening.</li></ol><h3>Let’s Build It. From Scratch.</h3><p>Let’s take a training corpus short and simple</p><pre>text = &quot;in the midst of chaos, there is also opportunity. the journey of learning is never easy, but it is always rewarding. every mistake you make is a step closer to understanding. intelligence is not just about knowing facts, but about connecting ideas and seeing patterns where others see noise. persistence and curiosity are the two most powerful tools you can carry. over time, even the most complex problems begin to feel manageable, and what once seemed impossible becomes second nature.&quot;</pre><p>In this text you can see most of the alphabets space and some punctuations too. But machine doesnt understand any of it.<br>So we convert these into numbers by giving them unique indexes</p><pre>char = sorted(set(text))<br><br>char_to_idx = {}<br>for i, ch in enumerate(char):<br>    char_to_idx[ch] = i<br><br>idx_to_char = {}<br>for ch, i in char_to_idx.items():<br>    idx_to_char[i] = ch</pre><p>In this code block we have done 2 things mainly:<br>1. Gave all the characters unique numbers<br>2. Made dictionaries for the index to char and char to index lookups.</p><pre>tokenizer = {<br>    &quot;char_to_idx&quot;: char_to_idx,<br>    &quot;idx_to_char&quot;: idx_to_char<br>}<br>bigram_counts = np.zeros((len(char), len(char)))<br>tokens = [char_to_idx[ch] for ch in text]<br><br>for i in range(len(tokens)-1):<br>    bigram_counts[tokens[i], tokens[i+1]] += 1</pre><p>In this section of the code, we have mainly built the matrix where we find how frequent 2 adjacent pair arrive in the training set and store them in the matrix of order or size same as the number of unique characters.</p><pre>def softmax(x):<br>    exp_x= np.exp(x - np.max(x))<br>    return exp_x / exp_x.sum()<br><br>bigram_probs = np.zeros_like(bigram_counts)<br>for i in range(len(char)):<br>    rows = bigram_counts[i]<br>    if rows.sum() &gt; 0:<br>        bigram_probs[i] = softmax(rows)</pre><p>This is the main mathematical part as we normalize the matrix by applying the SoftMax function and make all the values within the range of [0,1].</p><p>One thing I want to be honest about in my implementation. I used SoftMax to normalize the bigram counts, which is not exactly the right move.</p><p>Softmax was designed for logits, not frequency counts. When you apply it to counts, it subtly squishes the difference between frequent and rare pairs. A pair appearing 50 times vs 2 times feels closer than it actually is.</p><p>The correct approach is just plain normalization:</p><p>python</p><pre>bigram_probs[i] = bigram_counts[i] / bigram_counts[i].sum()</pre><p>Small difference in code, meaningful difference in understanding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mREYGkcZojXizYYM55u7GQ.png" /><figcaption>Thats the matrix after applying SoftMax function.</figcaption></figure><pre>def generation(start_char, max_length):<br>    start_idx = char_to_idx[start_char]<br>    current_idx = start_idx<br>    result = [start_char]<br><br>    for _ in range(max_length - 1):<br>        next_probs = bigram_probs[current_idx]<br>        next_idx = np.random.choice(len(char), p=next_probs)<br>        result.append(idx_to_char[next_idx])<br>        current_idx = next_idx<br>    <br>    return &#39;&#39;.join(result)</pre><p>this function uses all the previous blocks and helps to generate the next character by passing the starting character and the total number of characters we want.</p><p>It basically, works on the Conditional probability equation where the current event is given and with the help of that, we have to find the occurrence of next event</p><p><strong>Bigram Counts:<br>P(next_char | current_char)</strong></p><h3>But Here’s Where It Breaks</h3><ol><li><strong>Too long sequences<br></strong>For example, “I am understanding the concept”:<br>Character level tokenizer would take 30–40 tokens, whereas sub-word level would only take 6–8 tokens.<br>It is a problem because higher tokens explode in transformers (Attention Cost = O(n^2) therefore character level tokens would become 1600 operations, but in sub-word level there would be 86 (approx.)).</li><li><strong>Model must build meaning from scratch</strong><br>To understand “Computers”, word level would be like c -&gt; co -&gt; com -&gt; comp …….<br>Extremely inefficient.</li><li><strong>Long-range dependencies</strong><br>If text: “cat on the mat was hungry”<br>to compute “was hungry” — — Who? — → “the Cat”<br>model must remember the cat to compute was hungry which requires memory, and it didn’t have that.</li></ol><h3>So, What Actually Fixed This?</h3><p>These weren’t small inconveniences. The attention cost alone made character level tokenization a dead end.</p><p>Then came the Byte Pair Encoding. But BPE was not invented for NLP at all.</p><p>Philip Gage invented it in 1994 for file compression. The idea was simple: find the most frequent pair of bytes, replace it with one unused byte and repeat. Smaller file, same information.</p><p>In 2015, Sennrich et al. looked at that and thought, what if we do this with characters in language? Instead of compressing files, build a vocabulary. It worked.</p><p><strong>How does it actually work?</strong></p><p>Let’s take a small corpus, same way we did before:</p><pre>&quot;low low low wide new&quot;</pre><p>Start with individual characters:</p><pre>l o w<br>l o w<br>l o w<br>w i d e<br>n e w</pre><p>Now count every adjacent pair:</p><pre>(l, o) → 3<br>(o, w) → 4   ← most frequent<br>(w, i) → 1</pre><p>Merge the most frequent one. o + w becomes ow:</p><pre>l ow<br>l ow<br>l ow<br>w i d e<br>n e ow</pre><p>Count again. Now (l, ow) appears 3 times. Merge it:</p><pre>low<br>low<br>low<br>w i d e<br>n e ow</pre><p>we keep doing this until we hit our vocabulary size limit. That limit is something we set ourselves. GPT-2 set it at 50,257.</p><p>The result is sub words. Not full words, not single characters. Chunks like “un”, “ing”, “tion” that appear frequently enough to deserve their own token.</p><p>BPE never decides what makes sense. It just merges what appears most often. And somehow that produces meaningful subwords anyway.</p><p><strong>Where BPE is today</strong></p><p>GPT-2, GPT-3, GPT-4 use tiktoken, BPE based. LLaMA and Mistral use SentencePiece, a BPE variant. BERT uses WordPiece, same idea slightly different merge rule.</p><p>Every time you type into ChatGPT, your words are being silently chopped into subwords by logic borrowed from a 1994 file compression paper.</p><p><strong>But it is not perfect</strong></p><p>Vocabulary size is a guess. Too small and rare words get over split. Too large and memory bloats. No perfect number, just tuning.</p><p>It is not semantic aware. BPE does not know “unhappy” and “not happy” mean the same thing. It only sees character patterns.</p><p>And it struggles with non-English languages. Hindi, Arabic, Chinese. A single Hindi word can explode into many tokens. Still an active research problem today.</p><h3><strong>What I Actually Learned Building This</strong></h3><p><strong>Starting from zero, no libraries, no shortcuts, I felt every limitation of character-level tokenization personally. Sequences too long. Context evaporating. Meaning rebuilt from scratch every single time.</strong></p><p><strong>That frustration is what made BPE click for me.</strong></p><p><strong>BPE didn’t make models understand language. It solved the infrastructure problem that was blocking everything. Right sized chunks, not too small, not too large, gave transformers a real fighting chance.</strong></p><p><strong>Sometimes the best ideas just need the right problem to find them.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=071d7f1ab870" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/building-an-llm-from-scratch-heres-where-it-all-starts-071d7f1ab870">Building an LLM From Scratch. Here’s Where It All Starts.</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My RAG App Was Confidently Wrong, That’s When I Found CRAG]]></title>
            <link>https://medium.com/@vinayak1672006/why-rag-isnt-enough-understanding-crag-architecture-532c9bdfdfff?source=rss-768eaa045335------2</link>
            <guid isPermaLink="false">https://medium.com/p/532c9bdfdfff</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Vinayak]]></dc:creator>
            <pubDate>Tue, 17 Mar 2026 00:02:40 GMT</pubDate>
            <atom:updated>2026-05-12T21:00:24.020Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*JcHn0vANFo56_PEj0C_CxA.png" /></figure><p><em>How a small architectural fix changed the way I think about retrieval</em></p><p>I was testing my RAG chatbot one evening and asked it something completely outside its knowledge base — something it had no documents for.</p><p>It answered. Confidently. With full sentences. Structured paragraphs. Zero hesitation.</p><p><strong>It was completely wrong.</strong></p><p>That moment bothered me for days. I had built this system specifically to stop hallucinations. I used a vector database. I retrieved context before generating answers. The whole point of RAG was to ground the LLM in real documents.</p><p><em>So why was it still hallucinating?</em></p><p>That question led me to a research paper that completely changed how I understood retrieval. And once I saw the problem, I couldn’t unsee it.</p><h3>The Problem With RAG That Nobody Mentions</h3><p>When I first learned RAG, I understood it like this: instead of relying on the LLM’s memory, you retrieve relevant documents and give them as context. The LLM answers from those documents. Problem solved.</p><p>That’s correct. But there’s a step nobody talks about carefully — the retrieval step itself.</p><p>Vector databases like FAISS work on cosine similarity or Euclidean distance. They compare your query vector against stored document vectors and return the closest ones.</p><p><strong>Here’s the part that broke my mental model:</strong></p><p><em>They will always return something.</em></p><p>These algorithms are designed to find nearest neighbors — not relevant neighbors. There’s no built-in check for whether the retrieved document actually answers the query. It just returns whatever is mathematically closest, even if that document has nothing to do with what was asked.</p><p>So when I asked my chatbot something out of scope, the vector database didn’t say “I don’t have anything relevant.” It retrieved the closest documents it had — which happened to be loosely related topics — and passed them as context to the LLM.</p><p>The LLM received bad context and did what LLMs do with context — it used it. Confidently.</p><p><strong>RAG didn’t solve hallucination. It just changed the source of it.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/923/1*gQhu6_gf3K9AcLf83oAJsw.png" /><figcaption>Problems with Traditional RAG.</figcaption></figure><h3>People Tried to Fix This — It Wasn’t Enough</h3><p>I wasn’t the first person to notice this. The community had already tried a few approaches.</p><p>One popular fix was combining the standard Top-K vector retrieval with BM25 (Best Matching 25) — a keyword-based scoring method. The idea was that BM25 could rate whether a document was actually relevant by looking at term frequency and document length, not just vector distance.</p><p>It helped. But it didn’t fully solve the hallucination problem because it could rate relevance, but it couldn’t decide what to do when nothing relevant existed. The LLM still received whatever was there.</p><p><em>What we really needed was something that could evaluate the context, make a decision, and act differently based on that decision.</em></p><h3>Then I Found This Paper</h3><p>In 2024, a research paper was published called Corrective Retrieval Augmented Generation (CRAG). The core idea was surprisingly simple — add an evaluation step between the retrieval and the LLM.</p><p><strong>Not a better retriever. Not a smarter LLM. Just a step that asks: is this context actually good?</strong></p><p>This changed everything about how I thought about the architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/916/1*zhQ-BP5Bkrhsh_bQP_SOfA.png" /><figcaption>Solutions that didn’t work</figcaption></figure><h3>How CRAG Actually Works</h3><p>CRAG introduces a Retrieval Evaluator — an agent that sits between the vector database and the LLM. Its job is to classify the retrieved context before anything is passed to the LLM.</p><p>It classifies context into three categories:</p><p>• Good, at least one document is highly relevant</p><p>• Bad, all documents are irrelevant</p><p>• Ambiguous, somewhere in between</p><p>Here’s how the scoring works. The evaluator sends the retrieved documents to a model the original paper used a fine-tuned T5 model to save cost, but you can use any LLM and it rates each document from 0.0 to 1.0.</p><p>You set two thresholds let’s say Upper Threshold = 0.8 and Lower Threshold = 0.4.</p><p>• If any document scores above 0.8 -&gt; context is Good</p><p>• If all documents score below 0.4 -&gt; context is Bad</p><p>• Anything in between -&gt; Ambiguous</p><p>Documents scoring below the lower threshold are automatically removed from context. The system doesn’t pass garbage to the LLM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0vN3Cbm7X8Xa_jp4FNdGzg.png" /></figure><h3>What Happens After Classification</h3><p>If context is Good the system doesn’t stop there. Even good documents contain unnecessary text that costs tokens and adds noise. So an additional step strips each document into sentence-level chunks, filters only the relevant ones, and rebuilds a clean, focused context before passing it to the LLM.</p><p>If context is Bad the web search agent activates. It searches the query, retrieves web documents, strips them the same way, filters the relevant parts, and that becomes the context. The LLM answers from the web, not from bad local documents.</p><p>If context is Ambiguous both happen in parallel. Local documents are stripped and filtered. Web search runs simultaneously. The final context is the combination of both refined sources.</p><p><strong>When I understood this, I realized what was really different about CRAG. Standard RAG is passive it retrieves and trusts. CRAG is active it retrieves, evaluates, and corrects.</strong></p><h3>But It’s Not Perfect — Here’s What I Found</h3><p>I want to be honest here because most articles about CRAG just explain the architecture and stop. The reality of using it is a bit more complicated.</p><p><strong>Higher latency is real.</strong></p><p>You’re now adding evaluation, strip decomposition, filtering, and sometimes web search on top of retrieval. Each step adds time. You can use asyncio to run some of these in parallel but individual task speed doesn’t improve just the total wall time.</p><p><strong>Token costs go up.</strong></p><p>The evaluator model, the strip filtering, the web-retrieved documents — all of this is additional token usage. If you’re using a paid API for the evaluator, costs can increase noticeably. The original paper solved this by using a fine-tuned T5 model locally a clever move worth considering.</p><p><strong>The evaluator itself can be wrong.</strong></p><p>CRAG’s whole logic depends on the retrieval evaluator making the right call. If it misclassifies good context as bad, you lose relevant local documents and fall back to web search unnecessarily. If it misclassifies bad context as good, you’re back to the original problem. The evaluator is doing a lot of heavy lifting — its quality directly defines the system’s quality.</p><h3>What This Actually Changed For Me</h3><p>Before CRAG, I thought about RAG as a pipeline: embed → retrieve → generate. A straight line.</p><p>After understanding CRAG, I started thinking about retrieval as a decision point. Not just “what documents are close?” but “are these documents actually good enough to answer this query, and what do we do if they’re not?”</p><p><em>That shift from passive retrieval to active evaluation is what CRAG is really about. The architecture is the mechanism. The mindset shift is the actual lesson.</em></p><p>If you’re building any kind of LLM application that relies on external knowledge, ask yourself: what happens when your retrieval step returns something irrelevant? If the answer is “the LLM uses it anyway” that’s the problem CRAG was built to solve.</p><p><em>If this helped, follow for more on RAG architectures, LLM systems, and agentic workflows.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=532c9bdfdfff" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>