<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Aquib Khan on Medium]]></title>
        <description><![CDATA[Stories by Aquib Khan on Medium]]></description>
        <link>https://medium.com/@aquibjkhan?source=rss-55ee1333f970------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*9L6iTwjFypbIbAwEpOmoUA.png</url>
            <title>Stories by Aquib Khan on Medium</title>
            <link>https://medium.com/@aquibjkhan?source=rss-55ee1333f970------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 00:19:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aquibjkhan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Analog In-Memory Computing: A Dot Product Without a Multiplier.]]></title>
            <link>https://aquibjkhan.medium.com/analog-in-memory-computing-a-dot-product-without-a-multiplier-0e0e6725f654?source=rss-55ee1333f970------2</link>
            <guid isPermaLink="false">https://medium.com/p/0e0e6725f654</guid>
            <category><![CDATA[hardware]]></category>
            <category><![CDATA[analog]]></category>
            <category><![CDATA[ml-on-device]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Aquib Khan]]></dc:creator>
            <pubDate>Sun, 28 Jun 2026 13:35:48 GMT</pubDate>
            <atom:updated>2026-06-28T13:35:48.757Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UOpHB7waoDCJaKjF.jpeg" /></figure><p>This post, and the follow-up to it, focus on <strong><em>analog in-memory multiply with op-amp accumulation</em></strong>, running the multiply inside the <strong><em>resistive element</em></strong> that stores the <strong>weights of a trained model</strong>, and summing the resulting signals with an operational amplifier implemented on real hardware alongside some simulation. The goal is to run a tiny machine-learning model (most likely a logistic regression) on commodity ICs such as the <strong>MCP4251</strong>, a dual digital potentiometer (a programmable variable resistor).</p><p>In conventional inference, a weight matrix is fetched from disk into memory, input signals are applied, a dot product is computed, and an activation function is applied. We reproduce that pipeline physically, but for a deliberately simple linear model because deploying onto programmable resistors and elementary analog components imposes real constraints:</p><ul><li><strong>Weights are non-negative.</strong> A divider can only attenuate, so every weight is a resistance ratio ≥ 0 - there are no negative numbers.</li><li><strong>Weights are at most 1.</strong> The ratio D/256 never exceeds unity, so values above 1 are off the table. <br><em>D = wiper position code. 256 = number of steps (8-bit).</em></li><li><strong>Weights are quantised.</strong> A wiper has only 257 discrete positions, not a true continuum. <br><em>257 = number of positions (codes 0 to 256) for MCP 4251 IC</em></li><li><strong>Features must be scaled too.</strong> Inputs are voltages in [0, V_ref], so every feature has to be normalised into [0, 1] as well.<br><em>V_ref = divider’s reference voltage.</em></li><li><strong>Summation needs hardware.</strong> Wiper voltages can’t be tied together to add; an op-amp does the accumulate.</li></ul><blockquote><em>The key idea is </em><strong><em>constrained optimisation</em></strong><em>: Train it </em>under the conditions in which it will be deployed<em>. Weights and features scaled into </em><em>[0, 1] from the start. So, the learned model is, by construction, one the circuit can physically realise. <br>This isn&#39;t to say negative weights are impossible. This can be </em><strong><em>implemented</em></strong><em>, by adding components to </em>invert the signal&#39;s polarity while preserving its magnitude<em><br></em>The goal of this experiment is to perform inference with an extremely minimal component setup, and to confirm success by comparing the chip&#39;s results against an off-chip CPU on the test set. <br>* difference in accuracy between off-chip cpu and on-chip (MCP 4251) on test set</blockquote><p>W =[w0, w1] <strong>and </strong>Assume that W is the weight vector and X is the input feature vector. <strong>Dot product</strong> W*X = w0*x0 + w1*x1 <strong>this yields a scalar value.</strong><br>How this element-wise multiplication in minimum hardware setup will explore this now using a simple circuit called voltage divider.</p><blockquote><em>Two resistors in series across </em><em>V_in, output </em><em>V_out tapped between them, giving </em><em>V_out = V_in × R2/(R1+R2). The resistance ratio is a constant </em><em>m ∈ [0, 1], so the divider&#39;s only job is to scale its input by </em><em>m. A multiplication done by physics, not arithmetic. Read as a vector operation, one divider is one element-wise multiply.</em></blockquote><p>wᵢ · xᵢ: the feature xᵢ enters as a voltage, the weight wᵢ is the ratio, and the product appears at the output. Stack N dividers and sum their outputs and you have an N-dimensional dot product Σ wᵢ·xᵢ which is why I can take this divider as the atomic unit of <em>analog in-memory (almost)</em> inference.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/0*Pcf5vRTnIz9XkEBl.png" /></figure><p>Here, the circuit shows that when 5.0 V is applied across the divider ( R1 in series with R2) and we measure the voltage at the node between R1 and R2 with respect to ground.</p><p>The output is scaled by the factor R2/(R1+R2). Here 5k/10k = 0.5<br>then, 5 V × 0.5 gives 2.5 V. <br>That scaling by a fixed ratio is the multiplication: V_out = V_in × 0.5. This ratio R2/(R1+R2) is the weight m. The number we set by programming the wiper of a digital potentiometer, which represents a weight of the ML model.</p><p>Let&#39;s see this in simulation. I used <a href="https://everycircuit.com/">everycircuit</a> it&#39;s great without any hassle (login/signup) you can do the same.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FbewikcTbrkI%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DbewikcTbrkI&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FbewikcTbrkI%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/0a98643f51f232962f8b002e00704488/href">https://medium.com/media/0a98643f51f232962f8b002e00704488/href</a></iframe><p>So far we’ve multiplied one signal by one weight, with the weight set by the resistance ratio R2/(R1+R2), One unit of computation: a multiply. Next we&#39;ll move to another simulation tool and add more inputs and weights, <br>W = [w0, w1] and X = [x0, x1], to build up the full dot product.</p><p>Life would have been much easier if we could just twist the wiper wires together and have the voltages add up. But physics doesn’t work that way. The moment we join two wires they become one node, one point in the circuit and a point can only sit at one voltage. It can’t be V1 and V2 at once, so there&#39;s nowhere for a sum to live. So, <strong><em>the two divider outputs don&#39;t addthey fight</em></strong>, <strong><em>pushing current into each other until they settle on a single shared voltage.</em></strong> Not V1 + V2, but closer to their average, <br>(V1 + V2)/2.</p><blockquote><em>So if we can’t add voltages, we add </em>currents<em> instead.</em></blockquote><blockquote><em>Feed each divider output through a resistor into the </em>inverting<em> input of an op-amp. Feedback pins that input at </em><em>0 V a &quot;virtual ground.&quot; (The op-amp&#39;s gain is enormous, so to keep its output finite it drives the gap between its two inputs to almost nothing; with the </em><em>+ input grounded, the input is dragged to ~0 V too.) It&#39;s not wired to ground the op-amp just holds it there.</em></blockquote><blockquote><em>Because that node sits at zero, the inputs stop seeing each other. Each pushes its own current </em><em>V/R through its resistor, and the currents meet and sum at the node (Kirchhoff&#39;s current law). That combined current has nowhere to go but the feedback resistor, where it comes back out as a voltage. With equal resistors, </em><em>V_out = −(V1 + V2) the dot product, finally added up.</em></blockquote><blockquote><em>The minus sign it’s just the inverting setup flipping the sum. This we can fix it later; for now, the addition is real.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*hQPk-VQUK6yXCuWV.png" /></figure><p>The op-amp keeps its inverting input at the virtual ground ( 0 V) and draws no current itself - so every current pushed in through the input resistors has to leave through the feedback resistor. That&#39;s just Kirchhoff&#39;s current law at the node:</p><pre>  ΣVᵢ/R   = −V_out/R_f<br>→ V_out   = −(R_f/R) · Σ Vᵢ <br>→ V_out   = −Σ Vᵢ , when R_f = R</pre><p>(Each input current is Vᵢ/R because the node sits at 0 V, so the far end of each resistor <em>is</em> the voltage drop across it.)</p><p>So far we’ve kept the two operations apart. The divider gave us a multiply, <br>V_out = V_in × m. The op-amp gave us an addition, V_out = −Σ Vᵢ. <br>But looking closely at what&#39;s feeding that adder- those Vᵢ aren&#39;t just any voltages. Each one is the output of a divider, which means each one is already a product <br>wᵢ·xᵢ. The feature xᵢ went in as a voltage, the weight wᵢ scaled it, and wᵢ·xᵢ came out. The summer doesn&#39;t know or care where its inputs came from; it just adds them.</p><p>On substitute. Replacing each Vᵢ with the product it actually is:</p><pre>V_out = −Σ Vᵢ = −Σ (wᵢ · xᵢ) = −(w₁x₁ + w₂x₂ + ... + w_n x_n)</pre><p>That sum, Σ wᵢxᵢ, is the dot product W·X, the operation a neural network runs more than any other, and the one GPUs are specifically built to do fast. And here it falls out of a handful of resistors and one op-amp, with no multiplier and no adder in the digital sense. The multiplies happened inside the dividers, in parallel, for free. The add happened in the wire, as current. The minus sign is still just the inverting topology, and we&#39;ll deal with it when we reach the activation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hK5Ye9iwu27O7N8e5KXvUQ.png" /></figure><h3>One neuron : fixed weights [ W₁=0.75, W₂=0.25 ]with varying inputs.</h3><pre>x = [x₁]<br>    [x₂]<br><br>w = [w₁]<br>    [w₂]<br><br>wᵀx = [w₁  w₂] *  [x₁]<br>                  [x₂]<br><br>    = w₁x₁ + w₂x₂<br>    = Σᵢ₌₁² wᵢxᵢ<br>    = z (a scalar)</pre><pre>| x₁ | x₂ | 0.75x₁ + 0.25x₂ | Class |<br>|----|----|----------------:|:-----:|<br>| 0  | 0  | 0.00            |   0   |<br>| 1  | 1  | 1.00            |   0   |<br>| 1  | 5  | 2.00            |   0   |<br>| 5  | 1  | 4.00            |   1   |<br>| 4  | 1  | 3.25            |   0   |<br>| 4  | 2  | 3.50            |   1   |<br>| 5  | 5  | 5.00            |   1   |</pre><p>Weights fixed at [0.75, 0.25]; only the inputs change. The circuit computes <br>0.75·x₁ + 0.25·x₂ and fires when it clears 3.3v. The same two input numbers swapped, the decision flips because x₁ carries the heavier weight.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FIHCbMS8Fpfo%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DIHCbMS8Fpfo&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FIHCbMS8Fpfo%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/fbcca91ae2c2c3afbf366688ac7951a8/href">https://medium.com/media/fbcca91ae2c2c3afbf366688ac7951a8/href</a></iframe><p>So the hard part, <em>multiply and add,</em> in hardware, with no processor, is already done. In the next post I’ll close the loop: train an actual classifier, then deploy its learned weights onto the digipots in simulation and watch the circuit make the same decisions the CPU did.</p><p><em>Originally published at </em><a href="https://aquibjk.substack.com/p/analog-computing-a-dot-product-without"><em>https://aquibjk.substack.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0e0e6725f654" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Evolution of Attention [Part 2. Improved Alignment / Attention]]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://aquibjkhan.medium.com/evolution-of-attention-part-2-improved-alignment-attention-94160b3075ac?source=rss-55ee1333f970------2"><img src="https://cdn-images-1.medium.com/max/1080/0*Br6f2tUodE9R_u3w.jpeg" width="1080"></a></p><p class="medium-feed-snippet">In Part 1, we saw how Bahdanau gave the decoder a way to look back &#x2014; at every step, the alignment model scans all encoder states&#x2026;</p><p class="medium-feed-link"><a href="https://aquibjkhan.medium.com/evolution-of-attention-part-2-improved-alignment-attention-94160b3075ac?source=rss-55ee1333f970------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://aquibjkhan.medium.com/evolution-of-attention-part-2-improved-alignment-attention-94160b3075ac?source=rss-55ee1333f970------2</link>
            <guid isPermaLink="false">https://medium.com/p/94160b3075ac</guid>
            <category><![CDATA[self-attention]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Aquib Khan]]></dc:creator>
            <pubDate>Sat, 13 Jun 2026 17:37:33 GMT</pubDate>
            <atom:updated>2026-06-13T17:37:33.991Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Evolution of Attention [Part 1. Alignment method]]]></title>
            <link>https://aquibjkhan.medium.com/evolution-of-attention-part-1-alignment-method-309b70dce25e?source=rss-55ee1333f970------2</link>
            <guid isPermaLink="false">https://medium.com/p/309b70dce25e</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[recurrent-neural-network]]></category>
            <category><![CDATA[attention]]></category>
            <dc:creator><![CDATA[Aquib Khan]]></dc:creator>
            <pubDate>Sun, 05 Apr 2026 12:03:34 GMT</pubDate>
            <atom:updated>2026-04-05T13:28:36.647Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7qKcLezJj4VHOEGv.jpeg" /></figure><p>Learning a new language has humbled me in ways I did not expect. Despite being surrounded by the language every single day, there are still moments where I find myself standing in front of someone, knowing roughly what I want to say, and watching the words dissolve before they reach my mouth.</p><p>In this post we will go through the <a href="https://arxiv.org/pdf/1409.0473">Bahdanau et al. (2015)</a> paper with working code at every stage. Starting, with encoder-decoder RNN model training it on sample data covering the the bottleneck in vanilla encoder-decoder model. Then, implementing the soft alignment method explained in <a href="https://arxiv.org/pdf/1409.0473">Bahdanau et al. (2015</a>).</p><p>I intentionally kept the experiments small, so anyone can follow every step and re-run them yourself on whatever machine you have in front of you.</p><p><strong>Dataset:</strong> <a href="https://huggingface.co/datasets/bentrevett/multi30k"><em>bentrevett/multi30K</em></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/1*8EjbPEpcnniaxp0BNY9fnw.png" /></figure><p><strong>Metrics: </strong>BLEU ( <a href="https://en.wikipedia.org/wiki/BLEU">Bilingual Evaluation Understudy</a>) score is an automated metric used to measure the quality of machine-translated text by comparing it against one or more human reference translations.</p><p>This is the bottleneck in motion. A single RNN cell reads each word, updates its hidden state, and moves on; but moving on means letting go. By t=6, the cell has seen the whole sentence, yet the colours tell the real story: the early words have faded, overwritten by the ones that came after. And that single grey vector on the right? That is all the decoder will ever see.</p><p>High level workflow for Neural Machine Translation <strong>Encoder-Decoder</strong> only:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FiaB6VoPwCX0%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DiaB6VoPwCX0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FiaB6VoPwCX0%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/8580ef25f85818130538812b7fdcd177/href">https://medium.com/media/8580ef25f85818130538812b7fdcd177/href</a></iframe><p>In the video above, the <strong>“Context” vector C</strong> is the only vector that carries all the information from the input sentence and this is the <strong>real bottleneck</strong>. The encoder has no mechanism to selectively <strong>focus</strong> on <em>individual words</em>. Instead, it squeezes the entire sentence into a single fixed-length vector, and the <strong>decoder</strong> must <em>rely solely on that vector</em> for every step of the translation. No matter how long or complex the source sentence is, the decoder never gets to look back.</p><p>In the video below, each word enters the RNN cell in its own colour, “The” in red, “cat” in green. By t=2, the cell has already become a blend of both, <em>visually representing how the hidden state carries a mixture of everything the model has seen so far</em>. As more words arrive, earlier colours fade and newer ones dominate which is precisely where the problem lies. By the time the encoder finishes, the first words are barely a trace in the final hidden state.</p><blockquote><em>After training this model for 10 epochs: <br>Epoch: 01 | Train Loss: 5.021 | Val Loss: 4.792 | BLEU: 0.0292 <br>Epoch: 02 | Train Loss: 4.063 | Val Loss: 4.362 | BLEU: 0.0531 <br>Epoch: 03 | Train Loss: 3.614 | Val Loss: 4.209 | BLEU: 0.0683 <br>Epoch: 04 | Train Loss: 3.269 | Val Loss: 4.084 | BLEU: 0.1023 <br>Epoch: 05 | Train Loss: 3.007 | Val Loss: 4.076 | BLEU: 0.1320 <br>Epoch: 06 | Train Loss: 2.769 | Val Loss: 4.007 | BLEU: 0.1234 <br>Epoch: 07 | Train Loss: 2.567 | Val Loss: 4.015 | BLEU: 0.1199 <br>Epoch: 08 | Train Loss: 2.383 | Val Loss: 3.990 | BLEU: 0.1230 <br>Epoch: 09 | Train Loss: 2.230 | Val Loss: 3.993 | BLEU: 0.1187 <br>Epoch: 10 | Train Loss: 2.086 | Val Loss: 4.018 | BLEU: 0.1118 <br>Early stopping at epoch 10</em></blockquote><blockquote><em>Best BLEU score: 0.1320</em></blockquote><pre>test_examples = [<br>    &quot;A group of people stand in the back of a truck filled with cotton.&quot;,<br>    &quot;A man in a blue shirt is standing on a ladder cleaning a window.&quot;,<br>    &quot;Two young guys with shaggy hair look at their hands while hanging out in the yard.&quot;<br>]<br><br>for example in test_examples:<br>    translation = translate_sentence(model, example, src_vocab, trg_vocab, DEVICE)<br>    print(f&quot;English: {example}&quot;)<br>    print(f&quot;German: {translation}&quot;)<br>    print()</pre><pre>English: A group of people stand in the back of a truck filled with cotton.<br>German: eine gruppe von menschen steht auf einem &lt;unk&gt; mit &lt;unk&gt; &lt;unk&gt; „stop &lt;unk&gt;<br><br>English: A man in a blue shirt is standing on a ladder cleaning a window.<br>German: ein mann in einem blauen hemd steht an einer leiter und putzt eine wand.<br><br>English: Two young guys with shaggy hair look at their hands while hanging out in the yard.<br>German: zwei junge mit &lt;unk&gt; blicken in dieselbe richtung.</pre><p>English: A man in a blue shirt is standing on a ladder cleaning a window.<br>German: ein mann in einem blauen hemd steht an einer leiter und putzt eine wand.</p><p>English: Two young guys with shaggy hair look at their hands while hanging out in the yard.<br>German: zwei junge mit &lt;unk&gt; blicken in dieselbe richtung.</p><h3>Alignment/ Attention Model</h3><p>Results are not that bad right ;)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/281/1*xQ8JwQhC7fobeYXWWN6DHg.png" /></figure><p>The vanilla encoder-decoder with a fixed <strong>context</strong> is more like <em>summarising a 10 page book into a single sticky note &amp; then trying to translate from only that sticky note</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/194/1*LptSHoQVGZfFHqrxJp9XTQ.png" /></figure><p>Let’s forget about <strong>encoder-decoder</strong> model for now, When we read a long sentence in a foreign language, we don’t translate the whole thing at once. We glance back at specific words as we need them. The alignment model teaches a neural network to do exactly that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/143/1*qs3r605GcyNWoUMazEZXwQ.png" /></figure><p>The output of <strong>alignment model </strong>goes through <strong>softmax function</strong>, thus forcing the raw score from model into probability distribution.</p><p>Once, we get these probabilities this helps us to update the context vector at every decoder step, so depending on what token is generated from the decoder RNN the <strong>alignment model </strong>runs with that input and all the initial encoder states.</p><p>In the alignment model, one could ask, where does the first decoder state s₀ come from? Because think about it, the decoder hasn’t started yet. Sure, the &lt;sos&gt; token will be the first input, but to process even that, we need a context vector, and for the context vector we need alignment scores, then probabilities, then the updated context. So it’s kind of circular — everything depends on everything else.</p><p>To break out of this, we initialise s₀ from ←h₁, which is the final hidden state of the backward encoder RNN. Why ←h₁ specifically? Because the backward encoder reads the source sentence from end to start, so by the time it reaches position 1, it has already seen the entire input. That makes it the richest single summary of the source we have a solid starting point for the decoder.</p><h3>Let’s take a visual look;</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FwveZInDda18%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DwveZInDda18&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FwveZInDda18%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/cf6923edff5ee6a0b779fefce253ce62/href">https://medium.com/media/cf6923edff5ee6a0b779fefce253ce62/href</a></iframe><p>Below is the attention matrix from our English→German experiment. Each cell represents how much attention the decoder pays to a source word when generating a target word, darker means more attention. I chose English→German specifically because German sentence structure differs significantly from English, especially verb placement, making the alignment patterns more interesting to observe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*xUq058OvDNotEFSb2bR3mA.png" /></figure><p><strong>Input:</strong> “a man is riding a horse in the park.” <br><strong>Output:</strong> “ein mann reitet auf einem pferd im park.”</p><p>The attention shows a clean <em>near-diagonal</em> pattern, direct mappings like “ <em>mann</em>”→” <em>man</em>” “ <em>reitet</em>”→ “ <em>riding”</em> and “ <em>pferd</em>” → “ <em>horse</em>” confirm the model learned strong word-level alignments. “ <em>auf einem</em>” spreads attention across “ <em>riding</em>” and “ <em>a horse</em>”, showing the model gathers context from multiple source words to choose the correct German preposition and article case. “ <em>im</em>” and “ <em>park</em>” both anchor to “ <em>park</em>” and &lt;eos&gt;, correctly compressing &quot; <em>in the park</em> &quot; into two German tokens.</p><pre>## SEQ2SEQ with alignment score<br>class AttnEncoder(nn.Module):<br>    def __init__(self, input_dim, emb_dim, hidden_dim, dropout):<br>        super().__init__()<br>        self.embedding = nn.Embedding(input_dim, emb_dim)<br>        self.rnn = nn.GRU(emb_dim, hidden_dim, bidirectional=True)<br>        self.output_proj = nn.Linear(hidden_dim * 2, hidden_dim)<br>        self.hidden_proj = nn.Linear(hidden_dim * 2, hidden_dim)<br>        self.dropout = nn.Dropout(dropout)<br><br>    def forward(self, src):<br>        embedded = self.dropout(self.embedding(src))<br>        outputs, hidden = self.rnn(embedded)<br>        outputs = self.output_proj(outputs)<br>        hidden = self.hidden_proj(<br>            torch.cat([hidden[0], hidden[1]], dim=1)<br>        ).unsqueeze(0)<br>        return outputs, hidden<br><br>class AttnDecoder(nn.Module):<br>    def __init__(self, output_dim, emb_dim, hidden_dim, dropout):<br>        super().__init__()<br>        self.embedding = nn.Embedding(output_dim, emb_dim)<br>        self.attn_fc = nn.Linear(hidden_dim * 2, hidden_dim)<br>        self.v = nn.Linear(hidden_dim, 1, bias=False)<br>        self.rnn = nn.GRU(emb_dim + hidden_dim, hidden_dim)<br>        self.fc_out = nn.Linear(hidden_dim * 2, output_dim)<br>        self.dropout = nn.Dropout(dropout)<br><br>    def forward(self, input, hidden, encoder_outputs):<br>        embedded = self.dropout(self.embedding(input.unsqueeze(0)))<br><br>        # alignment - eᵢⱼ = vᵀ tanh(W [sᵢ₋₁ ; hⱼ])<br>        src_len = encoder_outputs.shape[0]<br>        h = hidden.expand(src_len, -1, -1)<br>        e = self.v(torch.tanh(<br>            self.attn_fc(torch.cat([h, encoder_outputs], dim=2))<br>        )).squeeze(2)<br><br>        alpha = F.softmax(e, dim=0)<br><br>        # context vector - cᵢ = Σ αᵢⱼ hⱼ<br>        c = (alpha.unsqueeze(2) * encoder_outputs).sum(0, keepdim=True)<br><br>        # decoder step - sᵢ = GRU([yᵢ₋₁ ; cᵢ], sᵢ₋₁)<br>        rnn_in = torch.cat([embedded, c], dim=2)<br>        output, hidden = self.rnn(rnn_in, hidden)<br><br>        pred = self.fc_out(<br>            torch.cat([output.squeeze(0), c.squeeze(0)], dim=1)<br>        )<br>        return pred, hidden<br><br>class AttnSeq2Seq(nn.Module):<br>    def __init__(self, encoder, decoder, device):<br>        super().__init__()<br>        self.encoder = encoder<br>        self.decoder = decoder<br>        self.device = device<br><br>    def forward(self, src, trg, teacher_forcing_ratio=0.5):<br>        batch_size = src.shape[1]<br>        trg_len = trg.shape[0]<br>        trg_vocab_size = self.decoder.fc_out.out_features<br><br>        outputs = torch.zeros(trg_len, batch_size, trg_vocab_size).to(self.device)<br><br>        encoder_outputs, hidden = self.encoder(src)<br>        input = trg[0, :]<br><br>        for t in range(1, trg_len):<br>            output, hidden = self.decoder(input, hidden, encoder_outputs)<br>            outputs[t] = output<br>            teacher_force = torch.rand(1).item() &lt; teacher_forcing_ratio<br>            input = trg[t] if teacher_force else output.argmax(1)<br><br>        return outputs</pre><p><em>Originally published at </em><a href="https://aquibjk.substack.com/p/evolution-of-attention-part-1-alignment"><em>https://aquibjk.substack.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=309b70dce25e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deduplication, near duplicate: a short guide]]></title>
            <link>https://aquibjkhan.medium.com/deduplication-near-duplicate-a-short-guide-b7ecbf348f97?source=rss-55ee1333f970------2</link>
            <guid isPermaLink="false">https://medium.com/p/b7ecbf348f97</guid>
            <category><![CDATA[machine-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Aquib Khan]]></dc:creator>
            <pubDate>Sat, 21 May 2022 12:28:07 GMT</pubDate>
            <atom:updated>2022-05-21T12:59:11.485Z</atom:updated>
            <content:encoded><![CDATA[<p>Hola Everyone!,</p><p>In this short and brief blog post, I will talk about a few methods and “out of the box” solutions to deal with deduplication and near-duplicated data.<br>So, why is this important? if we see from the ML model training point of view, ML models do not generalize well, when you have duplicate data in the training dataset.</p><p>Okie, so let’s start now,</p><p>There are many ways to calculate the similarity between two sentences/documents, let’s a have a look at the following:</p><ul><li>Jaccard similarity (Direct tokens matching), doesn’t perform well, because it doesn’t capture context, even the different synonyms are the problem here.</li><li>Vector-based similarity (cosine similarity, any distance-based):<br>* TF-IDF vectors<br>* Different vectors based (transformers/ fasttext/ word2vec…. etc)</li><li>We can leverage the zero-shot models</li><li>Train transformer using different loss functions (e.g, triplet loss functions)</li></ul><h3>Hashing</h3><p>In order to find exact data (duplicate data), we can use hashing (md5) and then group by.</p><pre>&gt;&gt;&gt; import hashlib<br>&gt;&gt;&gt; hashlib.md5(&quot;text to encode&quot;.encode()).hexdigest()<br>&#39;07161b7ecf4ca0760c721d6b8b9badfb&#39;</pre><p>For images: <a href="https://github.com/idealo/imagededup">https://github.com/idealo/imagededup</a></p><p>Let’s have a look at quick near-duplicate/similar documents with the “out of the box solutions”</p><pre>!pip install transformers<br>!pip install sentence-transformers</pre><p>after installation,</p><pre>from sentence_transformers import SentenceTransformer <br>from sklearn.metrics.pairwise import cosine_similarity</pre><pre>model = SentenceTransformer(&#39;bert-base-nli-mean-tokens&#39;)</pre><pre>def check_similarity(sent1, sent2):<br>    sent_vec_1 = model.encode(sent1)<br>    sent_vec_2 = model.encode(sent2)<br>    sent_score = cosine_similarity([sent_vec_1],[sent_vec_2]).tolist()[0][0]<br>   if sent_score &gt; 0.5:<br>        print(f&quot;The following: {sent1 } and {sent2 } are :: SIMILAR&quot;) <br>    print(f&quot;The following: {sent1} and {sent2} are :: DISSIMILAR&quot;)</pre><pre>sentence_1 = &quot;I like to eat when I travel&quot; <br>sentence_2 = &quot;I like walking when I travel&quot; </pre><pre>check_similarity(sent1=sentence_1, sent2=sentence_2)</pre><pre>check_similarity(sent1=sentence_1, sent2=sentence_2)</pre><p>The following: <br><em>I like to eat when I travel </em>and <em>I like walking when I travel</em> are :: <strong>SIMILAR</strong></p><p>The following: <br><em>I like to eat when I travel</em> and <em>I like to play cricket</em> are :: <strong>DISSIMILAR</strong></p><p>The above was simple <em>transformer</em> embeddings and cosine similarity, To improve the quality we can train model using triplet loss function to get better quality of embeddings.</p><p>We can leverage zero-shot as well,</p><pre>from transformers import pipeline<br>classifier = pipeline(&quot;zero-shot-classification&quot;,<br>                      model=&quot;facebook/bart-large-mnli&quot;)<br>                      <br>sentence_1 = &quot;I like to eat when I travel&quot;<br>sentences_to_match = [&quot;I like walking when I travel&quot;, &quot;I like to play cricket&quot;]</pre><pre>print(classifier(sentence_1, sentences_to_match))<br>-------------------------------------------------</pre><pre>{&#39;labels&#39;: [&#39;I like walking when I travel&#39;, &#39;I like to play cricket&#39;], &#39;scores&#39;: [0.6023975610733032, 0.397602379322052],<br>&#39;sequence&#39;: &#39;I like to eat when I travel&#39;}</pre><p><strong>The above were the two methods for similarity “out of the box solution”, with somefine tuning it can perform well</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b7ecbf348f97" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PERT]]></title>
            <link>https://aquibjkhan.medium.com/pert-c8823164afd7?source=rss-55ee1333f970------2</link>
            <guid isPermaLink="false">https://medium.com/p/c8823164afd7</guid>
            <dc:creator><![CDATA[Aquib Khan]]></dc:creator>
            <pubDate>Mon, 04 Apr 2022 20:51:17 GMT</pubDate>
            <atom:updated>2022-04-04T20:51:17.638Z</atom:updated>
            <content:encoded><![CDATA[<h4>PRE-TRAINING BERT WITH PERMUTED LANGUAGE MODEL<br><a href="https://arxiv.org/abs/2203.06906">https://arxiv.org/abs/2203.06906</a></h4><p>Pre-trained language models are widely in various machine learning tasks (NLP and Computer vision) this article talks about a new method for pretraining language models.</p><p>PLMs are classified into two categories based on their training protocols:<br><strong>auto-encoding</strong> and <strong>auto-regressive</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HubbmqfDfqzhlzj6QQ4fWw.png" /><figcaption>High-level architecture of both, <strong>auto-encoding</strong> and <strong>auto-regressive</strong></figcaption></figure><blockquote><strong>Permuted Language Model (PerLM): </strong><em>In this architecture, a proportion of the input text is permuted, and the training objective is to predict the position of the original text.</em></blockquote><blockquote>The proposed PerLM tries to recover the word orders from a disordered sentence, and the objective is to predict the position of the original word.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nU9c_ek8irfys1lYAnQmEg.png" /><figcaption><a href="https://arxiv.org/abs/2203.06906">https://arxiv.org/abs/2203.06906</a></figcaption></figure><p>The pre-training task used in the PERT, Permuted Language Model (PerLM)</p><ul><li>The masking strategy, here, whole word and n-gram masking strategies for selecting candidate tokens for masking with a percentage of 40%, 30%, 20%, 10% for word-level unigram to 4-gram.</li><li>Randomly select a set of 90% tokens and shuffle their orders. Note that the shuffle process only takes place for these 90% tokens, not the whole input sequence.</li><li>For the rest of the 10% tokens, we keep them unchanged, treating them as negative samples.</li></ul><blockquote>* PerLM does not employ the artificial token [MASK] for masking purposes, which alleviates the pretraining-finetuning discrepancy issue (but could still suffer from unnatural word orders).</blockquote><blockquote>* The prediction space for PerLM is the input sequence, rather than on the whole. vocabulary, making it computationally efficient than MLM.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c8823164afd7" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>