<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nahom Birhan on Medium]]></title>
        <description><![CDATA[Stories by Nahom Birhan on Medium]]></description>
        <link>https://medium.com/@nahombirhan?source=rss-3f1be332d07b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*FPn2IH7jlLE4Gske</url>
            <title>Stories by Nahom Birhan on Medium</title>
            <link>https://medium.com/@nahombirhan?source=rss-3f1be332d07b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 15:07:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nahombirhan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[My Language Model Saw ‘jumped’ Seven Times and Knew What Came Next]]></title>
            <link>https://medium.com/@nahombirhan/my-language-model-saw-jumped-seven-times-and-knew-what-came-next-9f0c2873fc4b?source=rss-3f1be332d07b------2</link>
            <guid isPermaLink="false">https://medium.com/p/9f0c2873fc4b</guid>
            <dc:creator><![CDATA[Nahom Birhan]]></dc:creator>
            <pubDate>Sat, 17 Oct 2026 06:31:00 GMT</pubDate>
            <atom:updated>2026-10-17T06:52:01.373Z</atom:updated>
            <content:encoded><![CDATA[<h4>A next-token model with every gradient written by hand, and why 580,000 numbers out-predict a 20-million-cell table</h4><p>I spend most of my research time thinking about <strong>agentic AI security</strong>: what happens when language models are given tools, make decisions, and begin interacting with systems outside the model itself. But the further up the stack I work, the more useful it becomes to understand what is happening underneath it. So I am rebuilding that stack from the bottom.</p><p><strong>Tokens to Agents</strong> is a seven-part series that follows the path from classical language models to modern AI agents: n-grams, embeddings, neural language modeling, transformers, LLMs and prompting, agentic AI, and finally agentic AI security. This is <strong>Part 3: Language Modeling</strong>. Code: <a href="https://github.com/NahomMA/tokens-to-agents">github.com/NahomMA/tokens-to-agents</a></p><p>The word <em>jumped</em> appears seven times in my training data. Seven times, in 86,587 word pairs of financial news.</p><p>Ask the count-based bigram from <a href="https://medium.com/@nahombirhan/the-textbook-fix-that-made-my-language-model-5-worse-e8586d5639b5">Part 1</a> what follows <em>jumped</em> and it hedges. It gives a probability of 0.06, and spreads the rest thinly over every other word it knows, because that is what smoothing does when the counts run out. Ask the model in this post, and it says <em>to</em> with probability 0.37. It has never been told that <em>jumped</em> and <em>rose</em> are related. It worked that out, and then it borrowed <em>rose</em>’s seventy-eight sightings to fill in <em>jumped</em>’s seven.</p><p>Both models see the same thing: one previous word, nothing else. Yet on the test set the neural one scores a perplexity of <strong>93.5</strong> against the count table’s <strong>132.6</strong>. That is 29% better with identical context, and within 4% of Part 1’s trigram, which gets a whole extra word to look at. The model is small enough that I could write every gradient by hand. So I did, and checked them against the arithmetic before a single line of training code ran. This post is that derivation, that check, and what the model learned.</p><h3>Two ways to fill in a table</h3><p>A bigram model is a table. Rows are previous words, columns are next words, and each cell holds P(next | previous). With a 4,495-word vocabulary, that is 4,495 × 4,495 = <strong>20.2 million cells</strong>, and I have 86,587 training pairs to fill them with. Almost every cell is empty. Part 1 spent its entire length on what to do about the empty ones. The neural model refuses to store the table. Instead, every word gets 64 numbers (an embedding), and a single matrix turns those 64 numbers into a full row of 4,495 probabilities whenever one is needed:</p><pre>h = E[x]              64 numbers for the previous word<br>z = W h + b           one score per vocabulary word<br>p = softmax(z)        the row of the table, computed on demand</pre><p>That is the whole network: about 580,000 parameters standing in for 20 million cells. The trade is what makes it work. Two previous words with similar embeddings get similar rows, so every pair the corpus shows for one of them quietly teaches the other. <em>rose</em> shows up seventy-eight times, usually followed by <em>to</em> or <em>by</em>; if <em>jumped</em> lands near <em>rose</em> in the embedding space, it inherits that knowledge without ever earning it.</p><p>Which is exactly what happened:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-X_ZciCNpXfdhgJcelct8A.png" /></figure><p>After <em>rose</em>, with real data behind it, the count table’s answers are real estimates. After <em>jumped,</em> they are not. Add-0.01 smoothing across 4,495 words puts 45 phantom observations into every row’s denominator, and 45 phantoms drown seven real ones: the row collapses to 0.06 and 0.02. The neural model’s row for <em>jumped</em> looks like a row for a verb of motion, 0.37 for <em>to</em> and 0.10 for <em>by</em>, because that is what it learned <em>jumped</em> is. There is no smoothing parameter to tune. Sharing is the smoothing, and it fell out of the bottleneck for free.</p><p><a href="https://medium.com/@nahombirhan/the-1990s-method-that-beat-my-neural-network-25e3e032f405">Part 2</a> built this bridge between words and then just looked at it. This is the first time it carries traffic.</p><h3>One derivative</h3><p>Training means: for each (previous, next) pair, compute the loss −log p[next], find how it changes with every parameter, and step the other way. A framework would do this silently. Doing it once on paper shows that the whole backward pass hangs on a single line. Call the true next word <em>y</em>. The derivative of the loss with respect to the scores <em>z</em> is</p><pre>∂L/∂z = p − onehot(y)</pre><p>For the right word, that is probability minus one, a push upward. For every wrong word, it is the probability itself, a push down in proportion to how wrong the model was. Everything else is this vector routed backwards through the two matrix products:</p><pre>∂L/∂W    = (p − onehot(y)) hᵀ       outer product, V × D<br>∂L/∂b    =  p − onehot(y)<br>∂L/∂E[x] =  Wᵀ (p − onehot(y))      only the row that was looked up moves</pre><p>Here is one step, with real digits. Three words, two dimensions, previous word 1, target word 2, a tiny random <em>W</em>. The forward pass gives p ≈ [0.332, 0.334, 0.334], near uniform, and a loss of about 1.10. The derivative is [0.332, 0.334, −0.666]: pull the target up by two thirds, push each wrong word down by a third. With a learning rate of 0.1, the embedding of word 1 moves from [0.000, 0.100] to [0.0017, 0.1017]. I keep that example in the repo because the code reproduces it digit for digit. If it ever stops doing so, something is broken.</p><h3>Where a framework would have covered for me</h3><p>The backward pass in code is the three formulas above, batched:</p><pre>def backward(params, x, y, h, p):<br>    dz = p.copy()<br>    dz[np.arange(len(y)), y] -= 1.0      # p − onehot(y)<br>    dz /= len(y)                          # mean over the batch</pre><pre>    dE = np.zeros_like(params.E)<br>    np.add.at(dE, x, dz @ params.W)       # rows repeat inside a batch: accumulate<br>    return Grads(E=dE, W=dz.T @ h, b=dz.sum(axis=0))</pre><p>The line worth staring at is np.add.at. If <em>the</em> appears forty times in a batch, its embedding row receives forty separate gradients, and the obvious dE[x] = ... would keep only the last one. Nothing crashes. The model just trains a little worse, forever, and you never find out. Autograd handles this correctly on your behalf, which is precisely why most people have never heard of it. The update rule is stochastic gradient descent with nothing added:</p><pre>def sgd_step(params, grads, lr):<br>    params.E -= lr * grads.E<br>    params.W -= lr * grads.W<br>    params.b -= lr * grads.b</pre><p>I resisted momentum and Adam on purpose. A hand-written gradient with a sign error can still converge under a clever optimiser. Plain SGD will not cover for me. So I checked. Nudge each parameter by ±0.00001, measure how the loss moves, and compare that slope with what the formulas say. Over every parameter of the toy model, the worst disagreement is 4.1 × 10⁻⁹, which is floating-point noise. A real mistake shows up as a disagreement of order one.</p><h3>Choosing a step size</h3><blockquote><em>With the gradients trusted, two decisions remain: how far to step, and how many numbers per word. I swept both, every run from the same seed, scored after every epoch on sentences held out from training.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3a-g4AWFnVX78PNTvezF-w.png" /></figure><blockquote><em>At a learning rate of 0.2 the model is still at perplexity 331 after ten epochs and dropping slowly. It would get there eventually; I do not have the patience. At 1.0 it reaches 168. At 5.0 it touches </em><strong><em>122</em></strong><em> in epoch 8, the best in the sweep.</em></blockquote><p>Then there is 20. Its training loss is the lowest of all four runs. On the sentences it has never seen, it is the second-worst model, and getting worse by the epoch. With steps that large, 580,000 parameters can memorise 86,587 pairs faster than they can find the structure that generalises.</p><p>I keep relearning this one. The number on the screen during training is not the number you care about. The run that minimised it was close to the worst thing I built.</p><h3>The scoreboard</h3><p>Hyperparameters chosen on the validation slice, then one pass over Part 1’s untouched 970 test sentences:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GWUP7EvsWqdRSGwynJ9XjA.png" /></figure><ul><li><strong>Unigram, no context: 329.1</strong></li><li><strong>Count bigram, best add-α: 132.6</strong></li><li><strong>Neural bigram, this post: 93.5</strong></li><li><strong>Interpolated trigram, best weights: 90.0</strong></li></ul><p>One word of context, learned instead of counted, closes most of the distance to a model that sees two. How many numbers per word does it take? Fewer than I expected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0_NSqj-6P1wPcMlIPHatQA.png" /></figure><p>Four already works. Sixteen is 24% better than four, sixty-four is another 7%, and 256 buys 3% more for four times the parameters and twice the training time. Most of the value is in the first handful of dimensions. I kept 64. The embeddings were trained to predict the next word and nothing else, so I was curious what they had picked up on the side. Nearest neighbours of <em>rose</em>: <em>fell</em>, <em>grew</em>, <em>owned</em>, <em>dropped</em>, <em>declined</em>. Four movement verbs, antonym included, which is the same geometry Part 2’s co-occurrence model found by counting; the fifth is the kind of noise 86,587 pairs leave behind. <em>eur</em>’s nearest neighbour is <em>$</em>. Nobody asked for a word map. Prediction built one anyway.</p><h3>Why a security researcher bothers with a bigram</h3><p><em>Look at </em><em>∂L/∂z = p − onehot(y) once more. It is an instruction: here is how to move the scores to make y more likely. Route it one step further back and it becomes how to move the input to make y more likely. That is the entire engine behind gradient-based adversarial prompts against modern language models. The same chain rule, pointed at a target the model&#39;s owner did not pick. I do not think you can reason clearly about that attack surface until you have written the backward pass yourself at least once.</em></p><p><em>The other thing to notice is the borrowing. jumped got rose’s knowledge because they share a neighbourhood, and that sharing is the reason the model is good. It is also a path. Change what the model believes about one word and every word nearby shifts a little too. A handful of poisoned training sentences does not stay where it was put. Generalisation and contamination are the same mechanism, seen from opposite ends. Part 7 picks both of these up with real agents in the loop. Here they are just honest observations about a model small enough to fit on one page.</em></p><h3>Try it yourself</h3><pre>git clone https://github.com/NahomMA/tokens-to-agents.git<br>cd tokens-to-agents<br><br>uv sync<br>uv run python parts/03-language-modeling/run.py</pre><p>That reproduces the gradient check, both sweeps, the final model, every figure here, and a results.json holding every number in this post. The model imports numpy and nothing else.</p><h3>Next: transformers</h3><blockquote><em>This model reads one word and forgets everything before it. Part 4 replaces that single lookup with attention, which lets every position read every other, and builds the transformer encoder and decoder out of the same parts used here: an embedding table, matrix products, a softmax. The only new idea is letting the model choose what to look at.</em></blockquote><h3>References</h3><blockquote><em>The model is the feed-forward neural language model of </em><strong><em>Bengio et al., </em></strong><a href="https://www.jmlr.org/papers/v3/bengio03a.html"><strong><em>A Neural Probabilistic Language Model</em></strong></a><strong><em>, JMLR 2003</em></strong><em>, in its one-word-of-context form. The derivation follows </em><strong><em>Jurafsky &amp; Martin, </em></strong><a href="https://web.stanford.edu/~jurafsky/slp3/"><strong><em>Speech and Language Processing</em></strong></a><strong><em>, 3rd ed. draft, </em></strong><a href="https://web.stanford.edu/~jurafsky/slp3/7.pdf"><strong><em>Chapter 7: Neural Networks</em></strong></a><em>. The corpus is the Financial PhraseBank of </em><a href="https://arxiv.org/abs/1307.5336"><em>Malo et al., 2014</em></a><em>, split and closed exactly as in Part 1.</em></blockquote><blockquote><strong><em>Tokens to Agents:</em></strong><em> </em><a href="https://medium.com/@nahombirhan/the-textbook-fix-that-made-my-language-model-5-worse-e8586d5639b5"><em>1. N-grams</em></a><em> · </em><a href="https://medium.com/@nahombirhan/the-1990s-method-that-beat-my-neural-network-25e3e032f405"><em>2. Embeddings</em></a><em> · </em><strong><em>3. Language Modeling</em></strong><em> · 4. Transformers · 5. LLMs &amp; Prompting · 6. Agentic AI · 7. Agentic AI Security</em></blockquote><blockquote><strong><em>Next: Part 4 — Transformers, coming soon.</em></strong></blockquote><blockquote><em>Nahom M. Birhan — PhD student, agentic AI security. </em><a href="https://nahomcyberai.tech/"><em>nahomcyberai.tech</em></a><em> · </em><a href="https://github.com/NahomMA"><em>GitHub</em></a><em> · </em><a href="https://huggingface.co/Nahom-M"><em>Hugging Face</em></a><em> · </em><a href="https://www.linkedin.com/in/nahombirhan"><em>LinkedIn</em></a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f0c2873fc4b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The 1990s Method That Beat My Neural Network]]></title>
            <link>https://medium.com/@nahombirhan/the-1990s-method-that-beat-my-neural-network-25e3e032f405?source=rss-3f1be332d07b------2</link>
            <guid isPermaLink="false">https://medium.com/p/25e3e032f405</guid>
            <dc:creator><![CDATA[Nahom Birhan]]></dc:creator>
            <pubDate>Mon, 14 Sep 2026 02:48:29 GMT</pubDate>
            <atom:updated>2026-09-14T03:27:17.141Z</atom:updated>
            <content:encoded><![CDATA[<h3>Word embeddings from scratch, two ways — and the one with no training loop wins</h3><p>I am actively working on <strong>agentic AI security research</strong>: what happens when language models are given tools and resources, observe, reason, and make decisions, interacting with systems outside the model itself as humans do by interacting with people and other knowledge sources to reach a decision.</p><p>But the further up the stack I work, the more useful it becomes to understand what is happening underneath it. So I am rebuilding that stack from the bottom. <strong>Tokens to Agents</strong> is a seven-part series that follows the path from classical language models to modern AI agents: n-grams, embeddings, neural language modeling, transformers, LLMs and prompting, agentic AI, and finally agentic AI security.</p><p>This is <strong>Part 2: Embeddings</strong>.</p><p>Code: <a href="https://github.com/NahomMA/tokens-to-agents">github.com/NahomMA/tokens-to-agents</a> · Demo: <a href="https://huggingface.co/spaces/Nahom-M/tokens-to-agents-02-embeddings">Embedding Explorer</a></p><p><a href="https://medium.com/@nahombirhan/the-textbook-fix-that-made-my-language-model-5-worse-e8586d5639b5">Part 1</a> ended on a wall. An n-gram model trained on 4,846 financial headlines could compute the probability of <em>profit rose</em>. It could not conclude anything about <em>earnings increased</em> from that — not one bit of evidence transferred. To a counting model, those phrases share nothing. Every context is an island. This post breaks that wall on the <strong>same corpus</strong>, twice, with two methods built from scratch:</p><ul><li><strong>Counting.</strong> Tally which words appear near which, keep the surprises, compress with linear algebra. No training loop exists.</li><li><strong>Predicting.</strong> Give every word a vector, use it to predict its neighbors, and correct the vectors on every mistake — skip-gram with negative sampling, gradients written by hand.</li></ul><p>Then both methods take the same exam. The neural one loses.</p><h3>Path 1: count, then compress</h3><p>Slide a ±2-word window over the corpus and count co-occurrences. That gives every word a row of counts — already a vector, just a terrible one: dominated by frequent words and mostly zeros.</p><p>The whole method uses two classical methods.</p><ol><li><strong>PPMI</strong> keeps only co-occurrences that happen <em>more than chance predicts</em>:</li></ol><pre>total = cooc.sum()<br>p_word = cooc.sum(axis=1) / total<br>ctx = cooc.sum(axis=0) ** 0.75          # context-distribution smoothing<br>p_ctx = ctx / ctx.sum()<br>rows, cols = cooc.nonzero()<br>pmi = np.log2(cooc[rows, cols] / total / (p_word[rows] * p_ctx[cols]))<br>out[rows, cols] = np.maximum(pmi, 0.0)</pre><p>Note the nonzero(): PMI is only computed where a co-occurrence was observed. The zero cells that would send the logarithm to negative infinity are exactly the cells PPMI clips to zero anyway — the zero-probability problem from Part 1, dissolved instead of patched.</p><p><strong>2. SVD</strong> then compresses the 2,360 × 2,360 PPMI matrix into 100 dense dimensions:</p><pre>u, s, _ = np.linalg.svd(ppmi_matrix, full_matrices=False)<br>vectors = u[:, :100] * np.sqrt(s[:100])</pre><p>That is it. On this corpus it builds in about half a minute, and the result already knows things nobody told it:</p><pre>profit → operating, loss, pre-tax, pretax, net<br>rose   → fell, decreased, jumped, narrowed, totalled</pre><p><em>profit</em>’s neighbors are its profit-and-loss family. <em>rose</em>’s neighbors are the other movement verbs — including its own antonym, which is correct: <em>fell</em> is used in exactly the same contexts. Project a handful of words to two dimensions and the structure is visible:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*abxrKPmeF595HSSkzsavUQ.png" /></figure><p>Three regions, zero labels, zero training: movement verbs, profit-and-loss terms, people and organisations.</p><p>And one detail worth staring at: within the movement region, <em>rose</em> and <em>fell</em> sit side by side. Up-words and down-words interleave, because they are used in identical contexts — <em>profit rose 12%</em>, <em>profit fell 12%</em>. Co-occurrence captures <strong>substitutability, not polarity</strong>. Embeddings know which words play the same role; they do not know which direction is good news. Remember that limitation — it returns when these vectors start making decisions.</p><h3>Path 2: predict, and learn from mistakes</h3><p>Skip-gram flips the logic. Instead of counting what co-occurred, it <em>predicts</em> what will: each word gets a vector, and for every (center, context) pair in the corpus, the model asks the center vector to score its true neighbor above a few randomly sampled impostors. The loss for one pair with K negatives:</p><pre>-log σ(u_context · v_center)  -  Σₖ log σ(-u_negativeₖ · v_center)</pre><p>Differentiate it by hand, and the entire learning rule is three lines:</p><pre>g_pos = sigmoid(np.einsum(&quot;bd,bd-&gt;b&quot;, v_c, u_o)) - 1.0<br>g_neg = sigmoid(np.einsum(&quot;bd,bkd-&gt;bk&quot;, v_c, u_n))<br>grad_c = g_pos[:, None] * u_o + np.einsum(&quot;bk,bkd-&gt;bd&quot;, g_neg, u_n)</pre><p>No autograd, no framework — NumPy and the chain rule. This is deliberately the same machinery Part 3 will assemble into a full language model.</p><h3>The exam</h3><p>Both models produce vectors for the same 2,360 words. To score them, I use twenty human-judged related pairs — <em>profit · earnings</em>, <em>staff · employees</em>, <em>talks · negotiations</em> — against 500 random pairs, and ask one question: <strong>how often does a related pair score higher than a random one?</strong></p><p>That probability is an AUC. A coin flip scores 0.5. A perfect model scores 1.0.</p><ul><li><strong>PPMI + SVD (counting): 0.983</strong></li><li><strong>Skip-gram (predicting): 0.918</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_oIsfsprJMPXv-on7KJ1og.png" /></figure><p>The skip-gram number is the best of a tuning sweep over epochs, window sizes, and dimensions — the losing side got every advantage I could give it, and the sweep ships in the repo.</p><p>Where does the gap come from? Look at what each model says about <em>random</em> word pairs. Under PPMI + SVD, two random words score a cosine of about <strong>+0.05</strong> — unrelated words look unrelated. Under skip-gram, random pairs average <strong>+0.35</strong>, and under shorter training runs I measured floors as high as +0.7.</p><p>The gradient-trained model drifts toward making <em>everything</em> resemble everything—a known small-corpus failure of learned embeddings (the representation collapse behind it shows up even in large models as anisotropy). The AUC punishes exactly that loss of contrast.</p><p>None of this means skip-gram is a bad algorithm. It means 102,061 tokens is not enough data for it. Levy and Goldberg showed the two paths are close cousins mathematically — skip-gram implicitly factorises a PMI matrix — but the explicit factorisation needs far less data to get there. <strong>Neural is not a synonym for better; it is a bet on scale.</strong></p><h3>The promise, kept</h3><p>Back to the wall Part 1 hit. Here is what the counting model says about the exact pairs an n-gram model could never connect:</p><p><em>rose · increased</em> at 0.63, against a random floor of +0.05. <em>profit · earnings</em> at 0.34 — seven times the floor, from a word that appears only fifty times in the whole corpus.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FRPKtsZ-Y977k_2SLvCyqw.png" /></figure><p>The signal was in the counts all along. Part 1’s model failed not because counting is weak, but because it never pooled evidence across contexts. An embedding is that pooling — every context a word appears in votes on where it lives in the space, and words that live in the same neighborhoods end up neighbors.</p><p>Every exact context was an island. The embedding built the bridges.</p><h3>What this has to do with security</h3><p>Two things carry forward to the security work this series is walking toward.</p><p>First: <strong>similarity is now a measurable surface.</strong> Once <em>profit rose</em> and <em>earnings increased</em> are near-neighbors, any system that filters, retrieves, or routes text by meaning inherits that geometry — including its mistakes. A safety filter that blocks a phrase but not its nearest neighbors has a measurable gap, and this post just built the instrument that measures it.</p><p>Second: <strong>representation quality is a security property.</strong> The skip-gram model’s collapsed contrast — everything similar to everything — is precisely the failure mode that makes a similarity-based detector unreliable. Knowing <em>why</em> an embedding space has lost its contrast, and how to check, matters long before any attacker shows up.</p><p>Part 7 returns to both. For now they are just honest observations about geometry.</p><h3>Run it</h3><pre>git clone https://github.com/NahomMA/tokens-to-agents.git<br>cd tokens-to-agents<br>uv sync<br>uv run python parts/02-embeddings/run.py            # both models, ~4 minutes<br>uv run python parts/02-embeddings/run.py --sweep    # adds the skip-gram tuning table</pre><p>Everything above — every number, every figure, and the vectors behind the <a href="https://huggingface.co/spaces/Nahom-M/tokens-to-agents-02-embeddings">interactive demo</a> — is regenerated by that one script. The environment is pinned in a committed lockfile.</p><h3>Next: language modeling</h3><p>The counting method won today because the task was small. But the three gradient lines in Path 2 are the seed of everything that follows: Part 3 takes the same hand-derived backpropagation and assembles it into a neural next-token model — the first machine in this series that <em>learns</em> to predict language.</p><p>The counting era ends there. It went out winning.</p><h3>References</h3><p>The implementations follow the textbook treatment in <strong>Jurafsky &amp; Martin, </strong><a href="https://web.stanford.edu/~jurafsky/slp3/"><strong><em>Speech and Language Processing</em></strong></a><strong>, 3rd ed. draft, </strong><a href="https://web.stanford.edu/~jurafsky/slp3/5.pdf"><strong>Chapter 5: Embeddings</strong></a> — PPMI, SVD, and skip-gram with negative sampling are all built from that chapter’s formulation. The counting-vs-predicting equivalence is <a href="https://papers.nips.cc/paper_files/paper/2014/hash/b78666971ceae55a8e87efb7cbfd9ad4-Abstract.html">Levy &amp; Goldberg, NeurIPS 2014</a>; skip-gram itself is <a href="https://papers.nips.cc/paper/2013/hash/9aa42b31882ec039965f3c4923ce901b-Abstract.html">Mikolov et al., NeurIPS 2013</a>; the corpus is the Financial PhraseBank of <a href="https://arxiv.org/abs/1307.5336">Malo et al., 2014</a>.</p><p><strong>Tokens to Agents:</strong> <a href="https://medium.com/@nahombirhan/the-textbook-fix-that-made-my-language-model-5-worse-e8586d5639b5">1. N-grams</a> · <strong>2. Embeddings</strong> · 3. Language Modeling · 4. Transformers · 5. LLMs &amp; Prompting · 6. Agentic AI · 7. Agentic AI Security</p><p><strong>Next: Part 3 — Language Modeling, coming soon.</strong></p><p><em>Nahom M. Birhan — PhD student, agentic AI security.</em> <a href="https://nahomcyberai.tech/">nahomcyberai.tech</a> · <a href="https://github.com/NahomMA">GitHub</a> · <a href="https://huggingface.co/Nahom-M">Hugging Face</a> · <a href="https://www.linkedin.com/in/nahombirhan">LinkedIn</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=25e3e032f405" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Textbook Fix That Made My Language Model 5× Worse]]></title>
            <link>https://medium.com/@nahombirhan/the-textbook-fix-that-made-my-language-model-5-worse-e8586d5639b5?source=rss-3f1be332d07b------2</link>
            <guid isPermaLink="false">https://medium.com/p/e8586d5639b5</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[ngrams]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Nahom Birhan]]></dc:creator>
            <pubDate>Sun, 06 Sep 2026 21:49:18 GMT</pubDate>
            <atom:updated>2026-09-08T13:30:50.195Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building an n-gram language model from scratch and seeing what smoothing actually does</h3><p>Building an n-gram language model from scratch and seeing what smoothing actually does.</p><p>I spend most of my research time thinking about agentic <strong>AI security. What</strong> happens when language models are given tools, make decisions, and begin interacting with systems outside the model itself? But the higher up the stack I go, the more useful it becomes to understand what is happening underneath. So I started rebuilding that stack from the bottom.</p><blockquote><strong>Tokens to Agents:</strong> is a seven-part series that follows the path from classical language models to modern AI agents: n-grams, embeddings, neural language modeling, transformers, LLMs and prompting, agentic AI, and finally agentic AI security.</blockquote><p>The goal is not to avoid modern libraries. It is to understand the mechanisms they eventually abstract away.</p><h3>This is Part 1: N-grams.</h3><p>Code: GitHub<a href="http://git@github.com:NahomMA/tokens-to-agents.git"> <strong>link here</strong></a></p><p>You encounter a standard problem almost immediately when building an n-gram language model. Suppose the model has never seen a particular word after a particular context. The estimated probability of that event becomes zero. And once a language model assigns zero probability to something that actually appears in the test set, its perplexity becomes infinite.</p><blockquote>The textbook answer is usually straightforward: <strong><em>add-one, or Laplace, smoothing.</em></strong></blockquote><p>Add one to every count. Nothing has probability zero anymore. Problem solved. Except that when I implemented it on 4,846 financial-news sentences, something unexpected happened. My smoothed trigram model reached a perplexity of <strong>1,642.4</strong>. The unigram model, which ignores context completely, scored <strong>329.1</strong>.</p><blockquote>In other words, the standard fix made the trigram model roughly <strong>4.9 </strong>times worse than simply counting individual words.</blockquote><p>That result became the interesting part of the experiment. N-grams are simple enough that when something goes wrong, there is nowhere for the explanation to hide. You can follow the failure all the way down to the arithmetic. And that is a useful place to begin.</p><p><strong><em>Before language models learned representations, they counted. </em></strong>At its core, a language model answers a simple question: g<em>iven what I have seen so far, what is likely to come next? </em>An n-gram model answers that question by counting. Take: <strong><em>operating profit rose to EUR 12 million. </em></strong>A trigram model might ask, g<em>iven </em><strong><em>operating profit</em></strong><em>, how often did </em><strong><em>rose</em></strong><em> come next? </em>Its estimate is simply:</p><blockquote>P(rose | operating profit) = count(operating profit rose) / count(operating profit)</blockquote><p>Repeat that across a corpus, and you have a language model. There is no gradient descent here. No hidden state. No attention mechanism. No GPU training run. The counts themselves define the probability distribution.</p><p>Modern autoregressive language models solve a much richer version of the same next-token prediction problem. What changes dramatically is how they represent context and estimate the probability of the next token.</p><p>That makes an n-gram model a useful place to meet several ideas that survive far beyond n-grams. One of them is<strong><em> perplexity</em></strong>. It can be computed as the exponentiated average negative log-probability of the test tokens:</p><pre>def perplexity(model, sentences):<br>  total, count = 0.0, 0<br>  for p in _scored(model, sentences):<br>    if p &lt;= 0.0:<br>        return inf<br>  total += log(p)<br>  count += 1<br>  return exp(-total / count)</pre><p>One intuitive way to read perplexity is as an effective branching factor. A model with lower perplexity is, on average, assigning more probability to the sequence that actually occurred. There is one especially important line in that implementation:</p><pre>if p &lt;= 0.0:<br>    return inf</pre><p>One zero-probability event is enough to make the perplexity of the entire test set infinite. And that is exactly what happened. <strong><em>The first problem: the model keeps seeing contexts it has never seen before. </em></strong>I split the dataset into 80% training and 20% testing, then built unigram, bigram, and trigram models directly from the training counts. The result was:</p><blockquote><strong>Unigram: 329.1, Bigram: ∞ , Trigram: ∞</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0EwpPb-DE80g0vqtEfrRdA.png" /></figure><p>The reason becomes clear when you look at coverage.</p><p>About <strong>22.8%</strong> of the evaluated bigram events received zero probability. For trigrams, that number rose to <strong>53.0%</strong>. More than half of the test events were being evaluated under a trigram context for which the model had no matching evidence.</p><p>That is the central tradeoff of a simple n-gram model. A trigram can use more context than a bigram. That should make it more informative when the relevant context has been observed. But the additional specificity also makes exact matches much rarer.</p><p>The more specific the question becomes, the less likely the training corpus has already answered it. So the model does not merely say:</p><blockquote><em>I think this next word is unlikely. </em>It says: <em>This next word is impossible.</em></blockquote><p>The test data immediately proves otherwise. <strong>Closing the vocabulary first. </strong>There is another source of zero probabilities that needs to be handled separately: words that never appeared in training. Before training the models, I therefore create a closed vocabulary. Words occurring below the selected frequency threshold are mapped to `&lt;unk&gt;`, and test-time words outside the training vocabulary are mapped there as well.</p><pre>counts = Counter(t for s in train for t in s)<br>vocab = {t for t, c in counts.items()<br>          if c &gt;= min_count<br>          }</pre><p>This distinction matters. Otherwise, the experiment would mix two different questions:</p><blockquote><em>1. Has the model seen this </em><strong><em>word</em></strong><em> before?</em></blockquote><blockquote><em>2. Has the model seen this word in this particular </em><strong><em>context</em></strong><em> before?</em></blockquote><p>I wanted to study the second. With the vocabulary closed, the unigram has a finite perplexity. Its problem is not coverage. Its problem is that it knows nothing about context. The higher-order models have the opposite problem. And that brings us to smoothing.</p><h3>The textbook fix</h3><p>Add-α smoothing modifies the estimate by pretending that every possible event has been observed α additional times:</p><pre>def prob(self, context, token):<br>    numerator = (<br>        self._ngrams[context + (token,)]<br>        + self.alpha<br>    )<br>    <br>    denominator = (<br>        self._contexts[context]<br>        + self.alpha * self._vocab_size<br>    )<br>    <br>    return (<br>        numerator / denominator<br>        if denominator<br>        else 0.0<br>    )</pre><p>When α = 1, this becomes Laplace smoothing. Now every possible next word receives some probability. And, as expected, the infinities disappear. Perplexity before and after smoothing:</p><ul><li><strong>1-gram</strong> — 329.1 unsmoothed, <strong>332.6</strong> with Laplace</li><li><strong>2-gram</strong> — ∞ unsmoothed, <strong>469.6</strong> with Laplace</li><li><strong>3-gram</strong> — ∞ unsmoothed, <strong>1,642.4</strong> with Laplace</li></ul><p>Technically, the zero-probability problem is fixed. But the model itself has become much worse. The trigram reaches <strong>1,642.4 perplexity</strong>, almost five times the unigram. Why?</p><p>The denominator tells most of the story. My vocabulary contains <strong>4,495 words</strong>. Under Laplace smoothing:</p><pre>α × V = 1 × 4,495</pre><p>So every context effectively receives 4,495 units of additional count mass spread across the vocabulary. Now imagine a trigram context that appeared only three times in the training data. The observed evidence has a total count of 3. Laplace smoothing asks that evidence be treated as if it had 4,495 units introduced by the smoothing rule. The estimate becomes dominated by the correction rather than by the data. And trigram contexts suffer the most because they are the sparsest. The model is not failing because using more context is inherently bad. It is failing because the amount of smoothing is enormous relative to the amount of evidence available for many contexts.</p><h3>What happens when α is actually tuned?</h3><p>The obvious next experiment was to stop assuming that α should equal 1. I swept smaller values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nanm50TQlaWznfEIylFfuQ.png" /></figure><p>Performance improved sharply as α decreased. The best bigram on the tested grid was used.</p><blockquote><em>α = 0.01 </em>and reached a perplexity of 132.6</blockquote><blockquote>The same bigram with α = 1 scored 469.6, <strong>That is a 3.5× improvement</strong> from changing one hyperparameter.</blockquote><p>For this dataset, Laplace smoothing was not simply suboptimal. It was the worst smoothing strength I evaluated for every model order. That does not make Laplace smoothing useless. It makes it a useful baseline rather than a universal setting. The important lesson for me was more general:</p><h4>A method can solve the mathematical failure it was designed to solve while still damaging the model statistically.</h4><p>Laplace removed zero probabilities exactly as promised. It simply redistributed far too much probability mass for this setting.</p><h3>The better answer was not choosing one model</h3><blockquote>After tuning, I had three usable models: unigram: 329.1, bigram: 132.6, trigram: 358.4</blockquote><p>The bigram was clearly the strongest single model. But choosing only the bigram throws away something useful about the others. A trigram can be highly informative when its context has been observed enough times. A unigram is much less specific, but it is reliable because it does not depend on context. That suggests a different solution: let all three contribute.</p><p>With linear interpolation:</p><pre>def prob(self, context, token):<br>    return sum( w * m.prob(<br>          context[<br>          len(context) - (m.n - 1):<br>          ] if m.n &gt; 1 else (),<br>          token,<br>          )<br>          for w, m in zip(<br>          self.weights,<br>          self.models,<br>    )<br>    )</pre><p>Each model contributes part of the final probability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bRughMdjrxNvuwGGXPWTxA.png" /></figure><p>The best combination I found assigned:<strong> unigram = 0.2, bigram = 0.4, trigram = 0.4. </strong>Its perplexity was <strong>90.0. </strong>That is about 32% lower than the best individually tuned model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jkhh10QoCX3nWbZTSB_JVg.png" /></figure><p>There is something I like about this result. The trigram had initially looked terrible. With Laplace smoothing, it scored<strong> 1,642</strong>. But the problem was not that the trigram contained no useful information. Once it was combined with more reliable lower-order estimates, that contextual information became useful. Instead of asking one model to be good everywhere, interpolation lets different levels of specificity support one another.</p><h3>Where n-grams reach their limit</h3><p>At this point it is natural to ask why we moved beyond n-grams at all. The answer becomes visible in one simple example. Suppose the training corpus contains <strong><em>profit rose</em></strong><em> </em>and later the model encounters <strong><em>earnings increased</em></strong><em>. </em>To a count-based word-level model, those phrases have no relationship.</p><blockquote><strong>profit</strong> and<strong> earnings</strong> are different symbols.</blockquote><blockquote><strong>rose </strong>and <strong>increased</strong> are different symbols.</blockquote><p>Nothing in the representation tells the model that the meanings are related. Every exact context is essentially its own island. That is a much deeper limitation than smoothing. Smoothing can decide how probability should be distributed when evidence is missing. It cannot teach the model that two different words or phrases are similar. And that is where the next part of the story begins. Instead of representing a word only by its identity, we need a representation in which words appearing in similar contexts can share statistical information. That is the motivation for <strong>embeddings</strong>.</p><p>Understanding the failure of counting makes distributed representations feel much less arbitrary. They solve a problem that the n-gram model makes impossible to ignore.</p><h3>What survives when the models get much larger?</h3><p>The implementation changes dramatically after this point, but several ideas introduced here continue through modern language modeling.</p><ul><li><strong>Next-token probability</strong> — becomes the autoregressive objective that modern language models are trained on.</li><li><strong>Perplexity</strong> — remains a common intrinsic measure of predictive performance.</li><li><strong>Limited n-gram history</strong> — becomes a much larger learned context representation.</li><li><strong>Exact symbolic contexts</strong> — become distributed representations that can share information across similar inputs.</li><li><strong>&lt;unk&gt; Vocabulary handling</strong> — becomes subword tokenization, which reduces dependence on whole-word vocabularies.</li><li><strong>Smoothing</strong> — a first example of introducing assumptions where the observed data are insufficient.</li><li><strong>Interpolation</strong> — combining estimates that differ in specificity and reliability.</li></ul><p>Not every analogy is exact.</p><p>A transformer is not simply a very large n-gram model, and modern neural language models do not suffer from literal unseen-context zeros in the same way. What carries forward is the underlying statistical question: <strong>How should a model behave when the evidence available to it is limited? </strong>That question appears again and again as the models become more capable.</p><h3>And eventually, why this matters to security</h3><p>This series eventually ends where my own research currently lives: agentic AI security. I do not think an n-gram model is a miniature security model for a modern agent. The systems are far too different for that analogy to be useful.</p><blockquote>But building this model exposes one idea that does matter later: <strong>models have regions where their estimates are supported well by their data, and regions where they are much less reliable.</strong></blockquote><p>With an unsmoothed n-gram, that boundary is unusually visible. The model literally returns probability zero. Smoothing removes the hard boundary, but it does not create new evidence. It introduces an assumption about what should happen where evidence is missing.</p><p>Modern language models handle uncertainty very differently, but the underlying research question remains important: how does a system behave when it encounters inputs, contexts, or interactions that differ from the situations on which its behavior was learned?</p><p>For security, those boundary cases matter because adversarial inputs are often deliberately constructed to push systems into unusual regions of their behavior. Perplexity can sometimes be one signal of distributional unfamiliarity, although it is certainly not a security detector by itself.</p><p>Later in this series, when I move from language models to tool-using agents, this question becomes much more consequential. A bad probability estimate produces a bad prediction. A bad decision from an agent may trigger a tool. That is why I wanted this series to start here. Before worrying about how an autonomous system behaves under attack, I want to understand the machinery that eventually becomes its model.</p><h3>Reproduce the experiment</h3><p>Everything in this article can be reproduced from the repository:</p><pre>git clone git@github.com:NahomMA/tokens-to-agents.git<br>cd tokens-to-agents<br>uv sync<br>uv run python parts/01-ngrams/run.py</pre><p>The implementation is roughly 200 lines of model code.</p><blockquote><strong>run.py</strong> recomputes the reported results and regenerates the figures used in this article.</blockquote><p>The environment is pinned through `uv.lock` and `.python-version` so that the experiment is reproducible without manually reconstructing dependencies.</p><h3>Next: Embeddings</h3><p>N-grams showed us exactly where counting stops being enough. The model knows that <strong><em>profit </em></strong>appeared in one place and<strong><em> earnings</em></strong> appeared somewhere else, but it has no representation of the fact that the two words may play similar roles.</p><h3>Part 2 starts there.</h3><p>We will build representations that allow words to share information based on the contexts in which they occur, moving from <strong><em>counts to vectors</em></strong>. And from there, one step closer to the models behind today’s agents.</p><p><strong>Tokens to Agents</strong></p><p><strong>1. N-grams</strong> → 2. Embeddings → 3. Language Modeling → 4. Transformers → 5. LLMs &amp; Prompting → 6. Agentic AI → 7. Agentic AI Security</p><p><em>Nahom M. Birhan — PhD researcher working on agentic AI security and robust AI.</em></p><p><a href="https://github.com/NahomMA">GitHub</a> · <a href="https://huggingface.co/Nahom-M">Hugging Face</a>. <a href="https://nahomcyberai.tech/">Website</a>. <a href="https://www.linkedin.com/in/nahombirhan/">LinkedIn</a>. <a href="https://scholar.google.com/citations?user=HIpthEQAAAAJ&amp;hl=en">Google Scholar</a> · <a href="https://openreview.net/profile?id=%7ENahom_BIRHAN1">OpenReview</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e8586d5639b5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Netcat: A Simple Way to Transfer Files in the same LAN]]></title>
            <link>https://medium.com/@nahombirhan/netcat-a-simple-way-to-transfer-files-in-the-same-lan-fea7bb126114?source=rss-3f1be332d07b------2</link>
            <guid isPermaLink="false">https://medium.com/p/fea7bb126114</guid>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[internet]]></category>
            <category><![CDATA[ai-in-cybersecurity]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[research]]></category>
            <dc:creator><![CDATA[Nahom Birhan]]></dc:creator>
            <pubDate>Wed, 31 Jul 2024 09:52:19 GMT</pubDate>
            <atom:updated>2024-07-31T10:01:42.534Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction:</h3><p>Netcat is a versatile networking utility that is often referred to as the “Swiss Army knife” of networking tools. It allows you to read and write data across network connections using TCP or UDP protocols. In this blog post, we will demonstrate how to use Netcat to transfer files between two Linux-based devices quickly and efficiently.</p><h3>Prerequisites:</h3><ul><li>Two Linux-based devices (e.g., PCs, Raspberry Pis, or servers) connected to the same local network</li><li>Netcat installed on both devices (use the command sudo apt install netcat for Debian-based systems or sudo yum install netcat for RHEL-based systems)</li></ul><h3>Step 1: Setting up the receiving device</h3><p>On the device that will receive the file, use the following command to start listening for incoming connections on a specific port (replace PORT_NUMBER with the desired port number):</p><blockquote><em>nc -l -p PORT_NUMBER &gt; output_filename</em></blockquote><p>For example, to listen on port 12345 and save the incoming file as “received_file.txt”:</p><blockquote>nc -l -p 12345 &gt; received_file.txt</blockquote><h3>Step 2: Sending the file from the source device</h3><p>On the device from which you want to send the file, use the following command (replace RECEIVER_IP_ADDRESS with the IP address of the receiving device, PORT_NUMBER with the port number used in Step 1, and input_filename with the file you want to transfer):</p><p><em>nc RECEIVER_IP_ADDRESS PORT_NUMBER &lt; input_filename</em></p><p>For example, to send a file named “source_file.txt” to a device with IP address 192.168.1.100 listening on port 12345:</p><p><em>nc 192.168.1.100 12345 &lt; source_file.txt</em></p><h3>Step 3: Confirming successful file transfer</h3><p>After executing the commands in Step 2, the file transfer will begin. Once the transfer is complete, you should see the output file on the receiving device with the same content as the input file from the source device. To confirm that the transfer was successful, you can compare the checksums of the original file and the received file using the md5sum command:</p><h4>On the source device:</h4><p><em>md5sum source_file.txt</em></p><h4>On the receiving device:</h4><p><em>md5sum received_file.txt</em></p><p>If the checksums match, the file transfer was successful.</p><h3>Conclusion:</h3><p>Netcat is a powerful and straightforward utility for transferring files between Linux-based devices. In just a few simple steps, you can quickly and efficiently transfer files across your local network without the need for additional software or complex configurations. The flexibility of Netcat also allows for more advanced use cases, such as creating reverse shells, port scanning, and network debugging.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fea7bb126114" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>