<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Amin Fadaeinejad on Medium]]></title>
        <description><![CDATA[Stories by Amin Fadaeinejad on Medium]]></description>
        <link>https://medium.com/@aminfadaeinejad.edu?source=rss-aff4faf055a7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*2icRuhEZWrcLW-Dz7th9Aw.jpeg</url>
            <title>Stories by Amin Fadaeinejad on Medium</title>
            <link>https://medium.com/@aminfadaeinejad.edu?source=rss-aff4faf055a7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 18:30:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aminfadaeinejad.edu/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Fridge That Watches the Door]]></title>
            <link>https://medium.com/@aminfadaeinejad.edu/the-fridge-that-watches-the-door-59179e3dd43e?source=rss-aff4faf055a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/59179e3dd43e</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Amin Fadaeinejad]]></dc:creator>
            <pubDate>Wed, 04 Nov 2026 00:25:47 GMT</pubDate>
            <atom:updated>2026-11-04T00:25:47.871Z</atom:updated>
            <content:encoded><![CDATA[<h4>How we built an AI fridge assistant with one camera, one Gemini call per door cycle, and seven small agents that each do one job.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-kWHyWEx6dY1A6EnuBzPWQ.png" /></figure><h3>The milk always goes bad first</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*9lw-_WyuiJJCqcnAyMCIXQ.gif" /><figcaption>The first seven seconds of our demo video. This is the whole problem in one scene.</figcaption></figure><p>Everybody knows this moment. You open the fridge, take the milk, and it is already bad. Or you find a block of cheese in the back that you forgot two weeks ago. In one home it is a small annoyance. Across a country it is a big number: the <a href="https://www.usda.gov/about-food/food-safety/food-loss-and-waste/food-waste-faqs">USDA</a> estimates that 30 to 40 percent of the U.S. food supply is wasted, and a 2025 <a href="https://www.epa.gov/land-research/estimating-cost-food-waste-american-consumers">EPA report</a> puts the cost at $728 per person per year, or $2,913 for a family of four.</p><p>When we talked about this in our group, we noticed that the problem is not that people do not want to cook. The problem is visibility. Nobody remembers what is in the fridge, when it went in, or what will spoil first.</p><p>So we built a fridge assistant that remembers for you. It is one camera inside the fridge and a team of small agents. When you open the door, it records what happens, works out what went in or came out, and gives every item an expiry date. From there it knows what is about to spoil, builds a daily menu around those items, and answers questions like “what can I cook tonight?”</p><p>In this post I go through how it works, the three design decisions that made it work, and the parts that still do not work.</p><h3>What we built</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*513KcWklVCQzQpL1to3tCQ.gif" /><figcaption>The Today view of the dashboard (we call it savor). What to eat first, the urgent list, today’s menu, and the waste ledger.</figcaption></figure><p>The system has three parts:</p><ul><li><strong>A camera inside the fridge.</strong> A normal webcam is enough. There is no door sensor and no wiring.</li><li><strong>A Python backend</strong> (FastAPI) with seven agents, one SQLite database, and an event bus that pushes every change to the browser over server-sent events (SSE).</li><li><strong>A web dashboard</strong> with three views: Today, Inventory and Meal plan. It shows what is in the fridge, what spoils first, a daily menu, a waste ledger, and a chat box where you can talk to the fridge by text or by voice.</li></ul><p>Here is the full two-minute demo:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FaknCMnhWXZA%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DaknCMnhWXZA&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FaknCMnhWXZA%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/3593b50d29cb33559ebf0a2deb0f94f2/href">https://medium.com/media/3593b50d29cb33559ebf0a2deb0f94f2/href</a></iframe><p>I built this together with Omid Omidi, Aida Mohammadshahi, Amir Shahbandegan and Amin Shamshiri. The code is open source on <a href="https://github.com/aminfadaei116/fridgeAI">GitHub</a>, and there is a live version at <a href="https://fridge.shahbandegan.me/">fridge.shahbandegan.me</a>.</p><h3>The whole system on one page</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zCC_bShj62tnNlaovsUa0Q.png" /><figcaption>Figure 1. One door cycle, from the camera to the dashboard. Cards marked “model” call an LLM. Cards marked “code” are deterministic.</figcaption></figure><p>Every door cycle, from the camera or from an uploaded clip, goes through one function, process_cycle(clip). The camera thread calls it when the door closes, and the upload button on the dashboard calls the same function with a recorded clip. So the demo path and the real path cannot drift apart. (Manual adds and deletes, for when the camera is wrong, go straight to the curator.)</p><p>The flow is short. A DoorWatcher thread decides when the door is open and records everything in between to one .mp4 file. The vision agent sends that clip to Gemini in one call and gets back a list of crossings. The curator decides which detections are reliable enough to write, the shelf-life agent gives each new item an expiry date and a price, and SQLite stores the result. Everything else (the daily briefing, the menu, the chat) reads from that database.</p><p>We also kept a strict split between model calls and plain code. Every place where a wrong answer is expensive is deterministic Python: the door trigger, the netting of crossings, the confidence gate, and the used-versus-wasted decision. The model only does what models are good at: watching video, writing recipes, and understanding sentences.</p><h3>Decision 1: the door sensor is free</h3><p>A camera inside a closed fridge sees black. So we need no reed switch, no GPIO pin and no wiring: the mean brightness of the frame <em>is</em> the door sensor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y9o9GiGXvcJfF3XQCgmN5g.png" /><figcaption>Figure 2. The door state machine on an illustrative brightness trace. The clip starts when brightness reaches 60 and ends when it falls below 40.</figcaption></figure><p>The important detail is that there are two thresholds, not one. The door opens when brightness reaches 60 and closes only when it falls below 40. With a single threshold, a hand shadow or a flickering bulb near the border would close and reopen the door several times, and one door cycle would become three short clips. The band between 40 and 60 absorbs this noise.</p><p>The whole sensor is a few lines in capture.py (simplified here):</p><pre>def frame_brightness(frame):<br>    &quot;&quot;&quot;Mean luminance, 0-255. The whole door sensor is this one number.&quot;&quot;&quot;<br>    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)<br>    return float(np.mean(gray))<br># inside the DoorWatcher loop<br>if not door_open and brightness &gt;= 60:      # door_open_brightness<br>    door_open = True<br>    recorder = ClipRecorder(new_clip_path(), fps, max_seconds=60)<br>if door_open:<br>    if brightness &gt;= 40:                    # door_close_brightness<br>        recorder.write(frame)<br>    else:<br>        door_open = False<br>        finish_cycle(recorder)              # the clip goes to process_cycle()</pre><p>Two small details matter in practice. When the door is closed, the loop sleeps 0.2 seconds between frames, so an idle camera costs almost nothing. When the door is open, the loop runs at the camera’s own frame rate, so the clip plays at close to real time and the timestamps the model gives back stay honest. A clip is also capped at 60 seconds, because a door left open must not record forever.</p><h3>Decision 2: watch the crossing, do not diff the shelf</h3><p>Our first version did what many inventory demos do. It took one photo when the door opened and one before it closed, and asked the model what is different. This has three problems:</p><ul><li><strong>Direction is guessed, not seen.</strong> If the yogurt is missing in the second photo, did someone take it out, or is it only hidden behind the juice?</li><li><strong>Occlusion breaks it.</strong> A new item placed behind an old one never appears in the second photo.</li><li><strong>In-and-out is invisible.</strong> If you take the milk out, pour a glass and put it back, the two photos look the same. The answer “no change” is correct, but only by luck.</li></ul><p>So we changed the question. The whole door cycle goes to the model as one video, and we ask: <em>what crossed the door plane, in or out, and when?</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*WdWjM20Xx68B3RuUKHIeUw.gif" /><figcaption>A door cycle in the demo (1.5x speed). We upload a recorded clip, the dashboard shows “Door closed, watching the clip”, and a few seconds later seven items land (spaghetti, olive oil, garlic, red pepper flakes, parsley, kosher salt, parmesan), each with its days left.</figcaption></figure><p>The model returns a list of crossings, bound to a strict schema with structured outputs:</p><pre>class Crossing(BaseModel):<br>    item_id: int         # same physical item -&gt; same id, even if it crosses twice<br>    name: str            # &quot;bell pepper&quot;: lowercase, singular, no brand<br>    category: Category<br>    quantity: float<br>    unit: str<br>    action: Literal[&quot;in&quot;, &quot;out&quot;]<br>    time_s: float        # seconds from the start of the clip<br>    confidence: float    # honest probability that this crossing happened<br>    note: str</pre><p>The prompt is strict about honesty. A clear, well-lit item should get 0.9 or more, a partly hidden one 0.4 to 0.7, and a guess from a silhouette less than 0.4. It also says that an empty list is a correct and common answer, and that the model must never invent a crossing to look useful.</p><p>The model does not decide what changed. That is the job of reduce_crossings, about twenty lines of plain Python that replay the crossings per item_id:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I3aEHs3SamZIhYr40gNb1Q.png" /><figcaption>Figure 3. Five items in one door cycle. The model reports crossings, the reducer nets them into a VisionDiff.</figcaption></figure><p>An item is added only if it was not inside before and ended up in. It is removed only if it was inside before and ended up out. Everything else nets to zero. Because this is code and not a second model call, it is fast, free, and covered by unit tests. It is still one API call per door cycle, and we did not have to write any hand tracking.</p><p>A few details that mattered:</p><ul><li><strong>Sampling rate.</strong> Gemini samples video at 1 frame per second by default. That misses a fast hand through the door, so we set fps=2 in the video metadata.</li><li><strong>Upload path.</strong> Clips up to 20 MB go inline in the request. Bigger clips go through the Gemini Files API and are deleted after the call.</li><li><strong>Evidence photo.</strong> We freeze the clip at the first in crossing and save that frame next to the item. The item is in a hand and in focus, which is a much better thumbnail than the shelf after the door closes.</li><li><strong>Fallback.</strong> The OpenAI-compatible endpoint does not accept video. So on OpenAI, or when the video call fails, the vision agent pulls two stills from the same clip (at 10% and 90% of its length) and asks the old two-frame question. Everything downstream still receives a VisionDiff.</li></ul><h3>Decision 3: let the fridge say “I am not sure”</h3><p>Many inventory demos pretend to be certain. Ours shows its work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jhI4vk63pVkxzQtTgxPF7w.png" /><figcaption>Figure 4. The curator’s confidence gate, with the default thresholds from config.py.</figcaption></figure><p>The curator sits between what the camera thinks it saw and what the fridge believes is true. At 0.80 confidence or more, a detection is written automatically. Between 0.25 and 0.80, the fridge asks you on the dashboard. Below 0.25 it says nothing, because a guess that weak is not worth an interruption. When you answer Yes, the item is committed and its confidence is stored as 1.0, because a human answer is ground truth.</p><p>We made this gate deterministic on purpose. It is a threshold comparison, not a judgement call. So it is easy to test, and the most interesting moment of the demo, the fridge admitting that it is unsure, happens the same way every time.</p><p>Removals need one more check: the item must exist in the inventory. The vision agent may say “red bell pepper” today and “bell pepper” tomorrow, so the curator tries the exact name first (oldest item first) and then the closest name by token overlap, if the overlap is at least 0.5. Without this, the fridge would slowly fill with duplicates.</p><h3>From a detection to an expiry date and a dollar figure</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VqhUswtA0810kOOpIj_uGA.png" /><figcaption>Figure 5. Left: how the shelf-life agent finds an expiry date. Right: how a departure becomes “used” or “wasted” in the ledger.</figcaption></figure><p>When the curator commits a new item, the shelf-life agent answers three questions: how long does it keep, what did it cost, and how should it be stored. Putting a bell pepper on the board should never wait for a network call, so the answer comes from the cheapest source first. The repo ships a seed table with 54 common foods. A miss falls through to a cache of earlier answers, and only then to one model call, whose answer is also cached. The prompt asks the model to be conservative, because this number decides when somebody is told to throw food away. Raw chicken breast resolves to the 2-day food-safety limit, not to an optimistic number.</p><p>The other half happens when something leaves. If it leaves before its expiry date, its cost goes into the ledger as <em>used in time</em>. If it leaves after, it goes in as <em>thrown out</em>. This is a rule and not an opinion, and that is why the dollar figure on the dashboard means something. With the demo data, the week shows $30.15 used in time and $7.48 thrown out.</p><h3>Seven agents, seven contracts</h3><p>Each agent has one job and one output contract. All contracts are pydantic models in schemas.py, and every model call is bound to one of them with structured outputs. Nothing in the system parses prose, so if a model changes its writing style, no downstream feature breaks.</p><ul><li><strong>Vision</strong> watches the door-cycle clip and reports crossings. Returns a VisionDiff.</li><li><strong>Curator</strong> commits, asks or ignores, and decides used versus wasted. Plain code. Returns a CurationResult.</li><li><strong>Shelf life</strong> says how long a food keeps, what it cost and how to store it. Returns a ShelfLifeVerdict.</li><li><strong>Chef</strong> proposes meals built on what is closest to spoiling, respecting diet and servings. Returns a RecipeBoard.</li><li><strong>Nutrition</strong> scores the chef’s recipes against a calorie target. Returns a NutritionReport.</li><li><strong>Sentinel</strong> writes the daily briefing on what is about to go. Returns a DailyDigest.</li><li><strong>Concierge</strong> is the only agent you talk to, and it routes your sentence to the others. Returns a ChatReply.</li></ul><p>Not every agent needs the same model. Vision, the chef and the concierge use the strong model. The bulk work (shelf life, nutrition, the briefing) uses a cheaper one. On Gemini these are gemini-flash-latest and gemini-flash-lite-latest, on OpenAI gpt-4o and gpt-4o-mini.</p><p>Two choices here are worth explaining.</p><p><strong>The nutritionist is separate from the chef.</strong> One model that invents a recipe and also scores it will mark its own homework generously. A second pass that sees only the finished ingredient list is more honest when the number does not land.</p><p><strong>The concierge owns no domain logic.</strong> It exposes the other agents as tools and runs a bounded tool-calling loop of at most four rounds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xji__wdovlo0dVUpY477qQ.png" /><figcaption>Figure 6. “I’m vegetarian and there are four of us tonight” becomes two tool calls in one turn. Arguments shown for illustration.</figcaption></figure><p>The sentence becomes a save_profile call, which is durable and applies to every future plan, and a plan_meals call for tonight. Adding a new capability means adding a tool, not rewriting the conversation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*pXkrNbeRTRQ1VsRRAmzgFA.gif" /><figcaption>“Ask the fridge” with one-tap prompts, then today’s menu: a spinach, mushroom and cheddar frittata built on the three items closest to their date.</figcaption></figure><p>The daily menu does not wait for you to ask. Every day, the chef and the nutritionist build breakfast, lunch and dinner with two options each, around whatever is closest to spoiling. A full board is two model calls over the whole fridge, so it is cached per day. The cache key includes an inventory signature (every item name with its days left), so a menu built around spinach is thrown away as soon as the spinach is eaten, or when it moves one day closer to its date.</p><p>All prompts follow one more rule: talk about food, dates and money, never about the person. No remarks about weight, discipline or what someone should eat. A calorie target is an input, not an opening for an opinion.</p><h3>Shipping it</h3><p><strong>One client, two providers.</strong> All model access lives in llm.py. Gemini is reached through its OpenAI-compatible endpoint, so switching providers is a base URL and a set of model names, with no second client and no branching in any agent. The one exception is video: the compatible surface takes only text and images, so the clip goes through Google&#39;s own google-genai SDK. Voice follows the same idea. On OpenAI we use Whisper and TTS. Gemini&#39;s compatible surface has no audio endpoints, so the dashboard falls back to the browser&#39;s Web Speech API for dictation and playback.</p><p><strong>No key, no crash.</strong> Without an API key everything still boots, and the agents degrade honestly instead of inventing results. The board, the ledger and the expiry logic are local. The chef falls back to a simple use-it-up recipe built on the most urgent items, and the chat says plainly that it cannot reach the model.</p><p><strong>Tests without a network.</strong> The suite has 98 tests and needs no API key and no network. It covers the deterministic core (the confidence gate, the used-versus-wasted rule, expiry arithmetic, FIFO matching, the brightness threshold, the crossing reducer) and the model plumbing against a stub client. We do not assert on LLM judgement. We assert that every agent degrades honestly when the model is unreachable.</p><p><strong>One fridge, one container.</strong> The whole app runs on Cloudflare Containers behind one Worker. The same FastAPI process serves the API and the dashboard, so there is one origin and no CORS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ethYRlStcikKD5h3nnRYwg.png" /><figcaption>Figure 7. Deployment. The Worker routes every request to the same container instance.</figcaption></figure><p>FridgePipeline is a singleton per process, sitting on SQLite and an in-memory event bus. A second instance would be a second fridge with a different inventory. So the Worker sends every request to getByName(&quot;singleton&quot;), and max_instances: 1 enforces the same rule on the platform side. The container sleeps after ten idle minutes and its disk stops with it, which resets the demo fridge. For a demo that is fine. For a real kitchen, persistent storage is the first thing to change.</p><h3>What does not work yet</h3><p>The confidence gate exists because this system will sometimes be wrong. Here is where it is wrong today:</p><ul><li><strong>Lighting and packaging.</strong> Vision accuracy depends on both. A $5 LED strip inside the fridge is the single biggest improvement we know of.</li><li><strong>Quantities are coarse.</strong> “3 bell peppers” is tracked. Half a pepper going back in is not.</li><li><strong>Costs are typical, not yours.</strong> Prices come from typical Canadian retail figures in the seed table, not from your receipts.</li><li><strong>Shelf life starts at the door.</strong> The clock starts when the item enters the fridge. It does not know that the chicken sat in a hot car for an hour first.</li><li><strong>Identity across door cycles.</strong> After our LinkedIn post, someone asked: if I take the milk out, pour a glass and put it back, is it the same carton or a new one? Inside one door cycle the reducer handles it, because both crossings share an item_id and cancel out. Across two door cycles it does not. The carton leaves and is booked as used, then comes back as a new item with a fresh expiry date. Fixing this needs some re-identification of items across clips, and it is still our main open problem.</li><li><strong>Crossings the camera cannot see.</strong> Watching the door helps a lot with occlusion, because an item is in a hand when it passes the door plane. But if a body blocks the camera at that moment, the crossing is lost.</li></ul><h3>Where we want to take it</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*f72FlSrld4i2ny5ttInxTg.gif" /><figcaption>From the concept teaser in our demo video. The phone app is a design direction, not part of the current code.</figcaption></figure><p>The teaser in our video shows the direction we want to go:</p><ul><li><strong>A phone app with push notifications</strong>, so the fridge can tell you “the chicken expires tomorrow” while you are still at work, not only when you open the dashboard.</li><li><strong>Small vision-language models near the fridge.</strong> Today every door cycle is one API call. A small VLM running on a local device would cut cost and latency, and the video would never leave the house.</li><li><strong>Re-identification across door cycles</strong>, to solve the milk-carton problem above.</li><li><strong>Receipts instead of averages</strong>, so the dollar figures in the ledger are your own.</li></ul><p>The code is on <a href="https://github.com/aminfadaei116/fridgeAI">GitHub</a> and the live demo is at <a href="https://fridge.shahbandegan.me/">fridge.shahbandegan.me</a>. If you have a good idea for the identity problem, I would really like to hear it in the comments.</p><p>Thanks to my teammates Omid Omidi, Aida Mohammadshahi, Amir Shahbandegan and Amin Shamshiri. Building this together was a lot of fun.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=59179e3dd43e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[22. RoPE Part 2: From 3D Video Grids to FLUX.2’s 4D Position IDs]]></title>
            <link>https://medium.com/@aminfadaeinejad.edu/22-rope-part-2-from-3d-video-grids-to-flux-2s-4d-position-ids-33b05b711554?source=rss-aff4faf055a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/33b05b711554</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Amin Fadaeinejad]]></dc:creator>
            <pubDate>Mon, 12 Oct 2026 06:40:41 GMT</pubDate>
            <atom:updated>2026-10-12T06:40:41.005Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Part 2 of a series on Rotary Position Embedding. Part 1 covered 1D RoPE and 2D axial RoPE for Vision Transformers.</em></p><h3>1. Recap</h3><p>Everything in this part builds on one property, which Part 1 derived from <em>RoFormer: Enhanced Transformer with Rotary Position Embedding</em> [1]:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cdio8pw7nWR1b5G0vud8mQ.png" /></figure><p>The attention logit sees only the offset n − m, not the absolute positions.</p><p>Axial 2D RoPE rotates one chunk of the head dimension by x and another chunk by y, so the logit depends on (Δx, Δy). Here I extend the same split to more axes: three for video DiTs and VLMs, four for FLUX.2. The code keeps the rotate_half convention of Part 1, and I added a switch for the interleaved layout, because FLUX checkpoints use it.</p><h3>2. General N-D Axial RoPE</h3><p>A token now has a position p = (p_1, …, p_A), one coordinate per axis. The head dimension is split as d = d_1 + ⋯ + d_A, and each chunk gets its own rotation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uvdJD12eX4YSXC-7qImKxg.png" /></figure><p>Each axis is a small 1D RoPE with its own geometric schedule.</p><p>Because R_p is block-diagonal, the logit splits by axis:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xzoZPAtVwV0pLvF0iQCrIQ.png" /></figure><p>The score depends only on Δp = p′ − p, as a sum of independent per-axis terms.</p><p>Two things follow from this sum. First, no term mixes axes, so the logit has no frequency along a diagonal such as Δh + Δw. This is the axial limitation that Part 1 discussed with RoPE-Mixed from <em>Rotary Position Embedding for Vision Transformer</em> [2]. Second, if all tokens get the same shift on one axis, nothing changes. An offset matters only through the differences it creates between groups of tokens, and the FLUX offsets in Sections 7 to 9 rely on exactly this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TAnZaLTZjAVEMLqQ9R64ww.png" /></figure><p><em>Figure 1. N-D axial RoPE: one rotated chunk per axis.</em></p><p>When I read the code of these models, I found two families. FLUX.1, FLUX.2, HunyuanVideo (<em>HunyuanVideo: A Systematic Framework For Large Video Generative Models</em> [3]), Wan 2.1 (<em>Wan: Open and Advanced Large-Scale Video Generative Models</em> [4]) and CogVideoX (<em>CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer</em> [5]) compute one schedule per axis, so every axis covers the full range from fast to slow. M-RoPE, from <em>Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution</em> [6], cuts one schedule for the whole head into contiguous bands. The logit formula is the same in both, but the spectra are very different (Section 6).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9WgOx-srxDezNHHhOBcsaQ.png" /></figure><p><em>Figure 2. Where the frequencies go: one schedule per axis, bands of one schedule, and interleaved bands.</em></p><h3>3. Head-Dimension Budget</h3><p>An axis with d_a channels has d_a / 2 frequencies. The base sets the range, and d_a sets how densely the range is sampled:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-oGsZsl8er4aMuKjmU1wHg.png" /></figure><p>This ratio is the step between neighboring “rulers” of the axis: a larger step means fewer distinct scales for measuring distance.</p><p>The step is 1.61 for a FLUX.2 axis (32 channels, base 2000) and 1.39 for the FLUX.1 h and w axes (56 channels, base 10000). I took the budgets below from the official code and configs:</p><pre>Model                      Head dim   Channels (t, h, w[, l])   Base<br>FLUX.1 [dev]               128        16, 56, 56                10000<br>FLUX.2 [dev]               128        32, 32, 32, 32            2000<br>HunyuanVideo               128        16, 56, 56                256<br>Wan 2.1 (1.3B, 14B)        128        44, 42, 42                10000<br>CogVideoX-5B               64         16, 24, 24                10000<br>Qwen2-VL / Qwen2.5-VL 7B   128        32, 48, 48 (bands)        1,000,000</pre><p>For Qwen, mrope_section = [16, 24, 24] counts frequencies, which gives 32, 48 and 48 channels. CogVideoX-2B is not in the table because it has no RoPE (use_rotary_positional_embeddings is false). Look at the last column: the base varies even more than the split.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1OPHU6-Ju71kyUPvDqf-Zg.png" /></figure><p><em>Figure 3. Head-dimension budget per axis.</em></p><h3>4. 3D RoPE for Video DiTs</h3><p>A 3D VAE compresses the clip into a latent of shape (T′, H′, W′). The DiT patchifies it, with patch size (1, 2, 2) in Wan 2.1 and HunyuanVideo, and each token gets the (t, h, w) of its patch: latent frame, row and column.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yiYeMgEa7-6vA5d4ugkV0w.png" /></figure><p><em>Figure 4. Video token grid (t, h, w).</em></p><p>Text tokens are not rotated in any of the three models. In the code, HunyuanVideo rotates only the video part of q and k, CogVideoX rotates only the tokens after text_length, and Wan 2.1 sends text through cross-attention.</p><p>I expected a common rule for the share of t, but the code shows none. HunyuanVideo gives t 16 of 128 channels, and CogVideoX-5B 16 of 64. Wan 2.1 makes each spatial chunk 2⌊d/6⌋ = 42 channels and gives t the remainder, 44. My guess for the small t chunks is that latent clips are short: the default CogVideoX-5B sample has 13 latent frames but a 30 × 45 patch grid.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1rgRE4P3a4eyeEgpaV41Eg.gif" /></figure><p><em>Animation 1. The axial split in Wan 2.1: one 128-channel head becomes three chunks of 44, 42 and 42 channels, and each chunk is rotated by one coordinate.</em></p><p>The base is the other lever. With base 256, 7 of the 28 spatial frequencies of HunyuanVideo rotate by less than 1 rad across a 64-token side, against 15 of 28 for FLUX.1 at base 10000 (same computation as in Section 8). A pair that barely rotates across the grid is close to the identity and carries almost no position signal.</p><h3>5. Latent Time vs. Real Time</h3><p>The t coordinate is a latent frame index, not a timestamp. CogVideoX uses temporal_compression_ratio = 4 and HunyuanVideo expects 4n + 1 input frames, so one step in t covers four video frames:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AuPL4iB8i89m5gsgpjjj3w.png" /></figure><p>The same rotation angle means 0.5 s at 8 fps and 0.17 s at 24 fps.</p><p>A model trained at one frame rate therefore learns motion per latent step, and its position signal carries no seconds. A longer video is extrapolation on t only, so the context-extension methods from Part 1 can be applied per axis: Position Interpolation from <em>Extending Context Window of Large Language Models via Positional Interpolation</em> [7] on t alone, or NTK-aware scaling of the t base only, as described in <em>YaRN: Efficient Context Window Extension of Large Language Models</em> [8]. I did not find an evaluation of these per-axis variants, so for now this is only a direction to try.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Uh6e_hmW8V2ay0eOxnXkvA.gif" /></figure><p><em>Animation 2. The time axis in Wan 2.1: 81 pixel frames become 21 latent frames, and a t pair with a period close to the training length repeats its phase in a longer video. The last part shows the idea of RIFLEx: A Free Lunch for Length Extrapolation in Video Diffusion Transformers [15].</em></p><p>Implementations also set hard limits. Wan 2.1 precomputes the frequencies for 1024 positions per axis (rope_params(1024, ...)), so its ids stop at 1023.</p><h3>6. 3D RoPE in VLMs</h3><p><em>Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution</em> [6] defines M-RoPE with simple rules. Text tokens use identical ids on the three axes, so M-RoPE reduces to 1D RoPE on text. An image has a constant temporal id, and a video increments t per frame. Each modality starts at the maximum id of the previous one plus one.</p><p>The band layout has a bigger effect than these rules. For Qwen2-VL 7B (head dimension 128, base one million), mrope_section = [16, 24, 24] gives the 16 fastest frequencies to t, the next 24 to h and the 24 slowest to w. The wavelengths are 6.3 to 160 tokens for t, 199 to 28,000 for h, and 35,000 to 5 million for w. Across a 100-token-wide image, the fastest w pair rotates by 0.018 rad. I read this as follows: inside the language model, the logits can hardly see horizontal position, so it has to come from the vision features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WjzOA_cHIWeyCF4bp9hslA.gif" /></figure><p><em>Animation 3. One frequency schedule per axis in Wan 2.1 and HunyuanVideo, against the bands of one shared schedule in the M-RoPE of Qwen2-VL.</em></p><p>The <em>Qwen2.5-VL Technical Report</em> [9] aligns temporal ids with absolute time. In the code, the step between temporal ids is tokens_per_second × temporal_patch_size / fps, and both values are 2 in the 7B config, so one temporal id always means 0.5 s. So Qwen2.5-VL solves the problem of Section 5 on the VLM side.</p><p><em>VideoRoPE: What Makes for Good Video Rotary Position Embedding?</em> [10] states that M-RoPE gives the higher frequencies to t, which causes periodic oscillation and “hash collisions” between distant frames, measured with V-NIAH-D (a needle-in-a-haystack task with periodic distractors). Its fix is low-frequency temporal allocation, a diagonal layout and adjustable temporal spacing. The <em>Qwen3-VL Technical Report</em> [11], mentioned in Part 1, interleaves the bands instead.</p><h3>7. Position IDs as Routing: FLUX.1 and Kontext</h3><p>FLUX.1 has three axes with (16, 56, 56) channels. In the official code, txt_ids are zeros and image ids are (0, y, x). A zero id gives cos = 1 and sin = 0, so text q and k are never rotated, which in effect is what HunyuanVideo and CogVideoX do. Per pair type:</p><ul><li>text to text: Δ = 0, so the DiT has no order signal between text tokens. Word order comes only from the T5 features.</li><li>text to image: Δ = (0, y, x), the absolute patch coordinate.</li><li>image to image: 2D axial RoPE, with the first axis unused.</li></ul><p>The text-to-image logit is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z4lV9BQvv4LEKaEvOlGf-Q.png" /></figure><p>so a text query sees each patch through a rotation fixed by the patch’s absolute position.</p><p><em>FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space</em> [12] uses the free first axis. The paper sets u_x = (0, h, w) for target tokens and u_(y_i) = (i, h, w), i = 1, …, N, for context tokens, a “virtual time step”. The official code has one context image with img_cond_ids[..., 0] = 1. Both blocks share the (h, w) grid, so aligned patches differ only on t:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kYQgIzk0bhiaQ5VCKo8wRg.png" /></figure><p>The spatial part of the score is not rotated, so aligned patches look like “the same place”, and only the t chunk says which block the key comes from. Position ids work here as routing labels more than as geometry.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9AYs1VNemuUccGUuJvJYaA.png" /></figure><p><em>Figure 5. FLUX.1 vs FLUX.2 id layout.</em></p><h3>8. FLUX.2 4D RoPE</h3><p>In the code, Flux2Transformer2DModel in diffusers [13] uses attention_head_dim = 128, axes_dims_rope = (32, 32, 32, 32) and rope_theta = 2000. The official Flux2Params [14] and both FLUX.2 [klein] configs (4B and 9B) use the same values. Each axis has 16 frequencies:</p><ul><li>t: which image (target at 0, references at 10, 20, …)</li><li>h, w: the latent grid. With the diffusers defaults (vae_scale_factor = 8 and 2 × 2 packing), a token covers 16 × 16 pixels, so a 1024 px side gives 64 tokens.</li><li>l: the text index.</li></ul><p>The l axis gives text a real order inside the DiT: text-to-text logits now see Δl = j′ − j. It also keeps text order off the spatial channels. In M-RoPE, text and images share all channels; in FLUX.2, every image token has l = 0 and never rotates the l channels.</p><p>The base is the less obvious choice. For one 32-channel axis, the snippet below computes the lowest frequency, its period, and how many of the 16 frequencies rotate by less than 1 rad across a 128-token side (a 2048 px image). It also computes</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uyLSnGhJeSaldqKTjR2GwQ.png" /></figure><p>the normalized logit of an axis chunk with itself at offset Δ, when the energy is spread evenly over its F pairs. K close to 1 means that the offset is almost invisible on that axis.</p><pre>import torch<br> <br> <br>def axis_freqs(d_a: int, base: float) -&gt; torch.Tensor:<br>    return base ** (-torch.arange(0, d_a, 2, dtype=torch.float64) / d_a)  # (d_a/2,)<br> <br> <br>def kernel(d_a: int, base: float, delta: float) -&gt; float:<br>    &quot;&quot;&quot;K(delta) = mean_i cos(delta * theta_i): normalized score of a chunk with itself.&quot;&quot;&quot;<br>    return torch.cos(delta * axis_freqs(d_a, base)).mean().item()<br> <br> <br>for base in (2000.0, 10000.0):<br>    theta = axis_freqs(32, base)  # (16,)<br>    slow = int((theta * 127 &lt; 1.0).sum())  # &lt; 1 rad over 128 tokens<br>    print(f&quot;base {base:&gt;7.0f}: theta_min = {theta[-1]:.1e}, period = &quot;<br>          f&quot;{2 * torch.pi / theta[-1]:,.0f} tokens, {slow}/16 turn &lt; 1 rad over a 128-token side&quot;)<br> <br>print(&quot;FLUX.2 t axis:&quot;, {dt: round(kernel(32, 2000.0, dt), 2) for dt in (1, 10, 20, 30, 40)})<br>print(&quot;FLUX.2 h axis:&quot;, {y: round(kernel(32, 2000.0, y), 2) for y in (0, 4, 16, 32, 63)})<br>print(&quot;FLUX.1 t axis:&quot;, {dt: round(kernel(16, 10000.0, dt), 2) for dt in (1, 2)})</pre><p>Output:</p><pre>base    2000: theta_min = 8.0e-04, period = 7,814 tokens, 5/16 turn &lt; 1 rad over a 128-token side<br>base   10000: theta_min = 1.8e-04, period = 35,333 tokens, 7/16 turn &lt; 1 rad over a 128-token side<br>FLUX.2 t axis: {1: 0.95, 10: 0.57, 20: 0.58, 30: 0.57, 40: 0.33}<br>FLUX.2 h axis: {0: 1.0, 4: 0.68, 16: 0.44, 32: 0.58, 63: 0.37}<br>FLUX.1 t axis: {1: 0.94, 2: 0.8}</pre><p>With base 2000, the slowest pair completes one turn after 7,814 positions, so relative distances stay unambiguous up to about 3,900 positions. No image grid comes close, so wrap-around is not the limit. The limit is at the other end: across a 128-token side, 7 of 16 frequencies barely move with base 10000, and only 5 with base 2000. BFL does not explain this choice, so the following is my reading: with only 16 frequencies per axis, a smaller base keeps about two more of them useful at image-grid scale.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U_n96Zk_dga3Jz_jdPu6uQ.png" /></figure><p><em>Figure 6. Spectrum of one 32-channel axis, rope_theta 2000 vs 10000.</em></p><h3>9. Token Placement in 4D</h3><p>In the official sampling code [14], prc_txt, prc_img and encode_image_refs give this placement:</p><pre>Token                              (t, h, w, l)<br>text token j                       (0, 0, 0, j)<br>target patch (y, x)                (0, y, x, 0)<br>reference i (from 0), patch (y, x) (10 + 10i, y, x, 0)</pre><p>Each reference keeps its own grid (arange over its own latent height and width), so every grid starts at (0, 0). One reference is resized to at most 2024² pixels, several to at most 1024² each. Reference tokens are concatenated to the target tokens, and the output is sliced back to the target.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lpvQKA0FcHV9ezTFGKX39Q.gif" /></figure><p><em>Animation 4. FLUX.2 position ids: text runs along l, and the reference images share the (h, w) grid of the target, 10 apart on t.</em></p><p>The first consequence is a spatial-alignment prior. A target patch and a reference patch at the same (y, x) differ only on t: Δ = (10 + 10i, 0, 0, 0). As in Kontext, their h, w and l chunks are not rotated against each other, which gives editing a prior for “take it from the same place”. There is a catch: the grids are not rescaled, so for a reference with another size or aspect ratio the same (y, x) is not the same relative location.</p><p>The second consequence is less intuitive: the references are distinct, but they are not ordered. At Δt = 10, the 10 slowest t-frequencies rotate by less than 0.6 rad, and pair 1 turns by 6.22 rad, almost one full turn. Only about five fast pairs separate a reference from the target, and K(10) = 0.57, K(20) = 0.58 and K(30) = 0.57 are almost equal. In my reading, the offset makes each reference clearly different from the target and from the other references, at about the same distance, but nothing in these numbers orders them.</p><p>Sequence order plays no role: the plain denoising path uses [text, target, references], the KV-cached path [text, references, target], and RoPE sees only the ids.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fYwkVxi24_NKKT2tjEMLbg.png" /></figure><p><em>Figure 7. FLUX.2 4D placement.</em></p><h3>10. What the Attention Score Sees</h3><p>With Δ = key id − query id:</p><pre>Pair (query → key)            Δ = (Δt, Δh, Δw, Δl)       Axes with signal<br>text j → text j′              (0, 0, 0, j′ − j)          l<br>text j → target (y, x)        (0, y, x, −j)              h, w, l, all absolute<br>target (y, x) → text j        (0, −y, −x, j)             the same, mirrored<br>target → target               (0, Δy, Δx, 0)             h, w<br>target → reference i          (10 + 10i, Δy, Δx, 0)      t, h, w<br>reference i → reference k     (10(k − i), Δy, Δx, 0)     t, h, w</pre><p>The attention pattern depends on the path. The plain denoise path uses full attention. In denoise_cached, the first step (forward_kv_extract) lets text and target tokens attend to all tokens, while reference tokens attend only to themselves, with a fixed timestep of 0 for their modulation. Their K and V are cached for later steps, so the rows “reference → text” and “reference → target” do not exist in this path.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dK0n45B_dLHre5HhD5BNDA.png" /></figure><p><em>Figure 8. Pair types vs axes.</em></p><p>The text rows are the interesting ones. Text sits at h = w = 0, so the text-to-image Δ on h and w is the patch coordinate itself, and on l it is the text index. The logit depends on two absolute positions. This holds for any fixed text anchor with one rotation per token, so FLUX.1, HunyuanVideo and CogVideoX share it. In the extreme case, text token 0 and the top-left target patch have the same id, (0, 0, 0, 0) (see the table in Section 11).</p><p>The next point is my own analysis and not a claim from BFL: this placement may favor the top-left region. If a text query and an image key were aligned inside one axis chunk, the factor along that axis would be K(y): 1.0 at y = 0, 0.68 at y = 4, 0.44 at y = 16, and 0.37 to 0.58 further away. All else equal, text would get slightly larger logits with patches near row 0 and column 0. Learned q and k can cancel this by moving their energy into fast pairs, where K oscillates, or slow pairs, where it stays near 1. A simple test would be to average text-to-image attention maps over many prompts in FLUX.2 [klein] 4B (Apache 2.0) and compare the four corners. I have not run this test.</p><h3>11. Implementation</h3><p>The four blocks form one script, so run them in order. C1 builds the angles p_a · θ_(a,i) for each axis and concatenates them into a table with d/2 entries. With rotate_half, as in Part 1, the table is repeated as [ang, ang], so pair i uses channels i and i + d/2. With interleaved=True each angle is repeated in place, so pair i uses channels 2i and 2i + 1, and each axis owns 32 contiguous channels. This is the layout of the official FLUX code, and you need it to load FLUX weights.</p><pre>import torch<br>from torch import Tensor<br> <br> <br>def rotate(x: Tensor, interleaved: bool) -&gt; Tensor:<br>    if interleaved:  # pairs (2i, 2i+1): FLUX, GPT-J<br>        x = x.unflatten(-1, (-1, 2))  # (..., d/2, 2)<br>        return torch.stack((-x[..., 1], x[..., 0]), dim=-1).flatten(-2)  # (..., d)<br>    x1, x2 = x.chunk(2, dim=-1)  # pairs (i, i + d/2): rotate_half<br>    return torch.cat((-x2, x1), dim=-1)  # (..., d)<br> <br> <br>def apply_rope(x: Tensor, cos: Tensor, sin: Tensor, interleaved: bool = False) -&gt; Tensor:<br>    return x * cos + rotate(x, interleaved) * sin  # (B, H, N, d)<br> <br> <br>class EmbedND(torch.nn.Module):<br>    &quot;&quot;&quot;N-D axial RoPE: one frequency schedule per axis, theta_{a,i} = theta^(-2i/d_a).&quot;&quot;&quot;<br> <br>    def __init__(self, axes_dims: list[int], theta: float, interleaved: bool = False) -&gt; None:<br>        super().__init__()<br>        assert all(d_a % 2 == 0 for d_a in axes_dims)<br>        self.axes_dims, self.theta, self.interleaved = axes_dims, theta, interleaved<br> <br>    def forward(self, ids: Tensor) -&gt; tuple[Tensor, Tensor]:<br>        # ids: (B, N, A), one coordinate per axis<br>        angles = []<br>        for a, d_a in enumerate(self.axes_dims):<br>            i = torch.arange(0, d_a, 2, dtype=torch.float32, device=ids.device)  # (d_a/2,)<br>            inv_freq = self.theta ** (-i / d_a)  # (d_a/2,)<br>            angles.append(ids[..., a, None].float() * inv_freq)  # (B, N, d_a/2)<br>        ang = torch.cat(angles, dim=-1)  # (B, N, d/2)<br> <br>        if self.interleaved:<br>            ang = ang.repeat_interleave(2, dim=-1)  # (B, N, d)<br>        else:<br>            ang = torch.cat((ang, ang), dim=-1)  # (B, N, d)<br>        return ang.cos()[:, None], ang.sin()[:, None]  # (B, 1, N, d) each</pre><p>C2 builds the FLUX.2 ids for L text tokens, an H × W target grid and references of any size. It follows prc_txt, prc_img and encode_image_refs.</p><pre>def grid_ids(t: int, h: int, w: int) -&gt; Tensor:<br>    &quot;&quot;&quot;(t, h, w, l) ids for one image: fixed t, (h, w) grid anchored at (0, 0), l = 0.&quot;&quot;&quot;<br>    return torch.cartesian_prod(torch.tensor([t]), torch.arange(h), torch.arange(w),<br>                                torch.tensor([0]))  # (h*w, 4)<br> <br> <br>def build_flux2_ids(txt_len: int, img_hw: tuple[int, int], ref_hws: list[tuple[int, int]],<br>                    scale: int = 10) -&gt; tuple[Tensor, Tensor, Tensor]:<br>    &quot;&quot;&quot;Position ids as in black-forest-labs/flux2 sampling.py (prc_txt, prc_img, encode_image_refs).&quot;&quot;&quot;<br>    txt_ids = torch.cartesian_prod(torch.tensor([0]), torch.tensor([0]), torch.tensor([0]),<br>                                   torch.arange(txt_len))  # (L, 4)<br>    img_ids = grid_ids(0, *img_hw)  # (H*W, 4)<br>    refs = [grid_ids(scale + scale * i, h, w) for i, (h, w) in enumerate(ref_hws)]<br>    ref_ids = torch.cat(refs) if refs else torch.zeros(0, 4, dtype=torch.long)  # (sum h_i*w_i, 4)<br>    return txt_ids, img_ids, ref_ids</pre><p>C3 runs three checks with random q and k and a head dimension of 128.</p><pre>def scores(q: Tensor, k: Tensor, ids: Tensor, rope: EmbedND) -&gt; Tensor:<br>    cos, sin = rope(ids[None])  # (1, 1, N, d) each<br>    q_r = apply_rope(q, cos, sin, rope.interleaved)  # (1, H, N, d)<br>    k_r = apply_rope(k, cos, sin, rope.interleaved)  # (1, H, N, d)<br>    return q_r @ k_r.transpose(-1, -2)  # (1, H, N, N)<br> <br> <br>torch.manual_seed(0)<br>txt, img, ref = build_flux2_ids(txt_len=6, img_hw=(4, 4), ref_hws=[(4, 4), (3, 5)])<br>ids = torch.cat([txt, img, ref]).float()  # (N, 4)<br>q, k = torch.randn(2, 1, 2, ids.shape[0], 128).unbind(0)  # (1, H, N, 128) each<br>half, inter = EmbedND([32] * 4, 2000.0), EmbedND([32] * 4, 2000.0, interleaved=True)<br> <br># (a) the same shift on one axis for all tokens changes no score<br>for a, name in enumerate(&quot;thwl&quot;):<br>    shift = torch.zeros(4); shift[a] = 37.0  # (4,)<br>    err = (scores(q, k, ids + shift, half) - scores(q, k, ids, half)).abs().max()<br>    print(f&quot;(a) shift on {name}: max |score change| = {err:.1e}&quot;)<br> <br># (b) target patch vs ref patch at the same (h, w): the score depends only on dt<br>pair = torch.cat([q[..., :1, :], k[..., :1, :]], dim=2)  # (1, H, 2, 128): query, key<br>for dt in (10, 20):<br>    s = [scores(pair, pair, torch.tensor([[0., y, x, 0.], [dt, y, x, 0.]]), half)[0, 0, 0, 1]<br>         for y, x in [(0, 0), (3, 1), (40, 17)]]  # query at target, key at ref<br>    print(f&quot;(b) dt = {dt}: score at three (h, w) positions = {[round(v.item(), 4) for v in s]}&quot;)<br> <br># (c) interleaved = rotate_half after a fixed channel permutation<br>perm = torch.stack([torch.arange(64), torch.arange(64) + 64], dim=-1).flatten()  # (128,)<br>err = (scores(q[..., perm], k[..., perm], ids, inter) - scores(q, k, ids, half)).abs().max()<br>print(f&quot;(c) interleaved vs rotate_half (permuted channels): max diff = {err:.1e}&quot;)</pre><p>Output:</p><pre>(a) shift on t: max |score change| = 1.3e-05<br>(a) shift on h: max |score change| = 1.2e-05<br>(a) shift on w: max |score change| = 1.5e-05<br>(a) shift on l: max |score change| = 1.5e-05<br>(b) dt = 10: score at three (h, w) positions = [23.2711, 23.2711, 23.2711]<br>(b) dt = 20: score at three (h, w) positions = [18.6125, 18.6125, 18.6125]<br>(c) interleaved vs rotate_half (permuted channels): max diff = 1.9e-05</pre><p>The first check is the shift invariance of Section 2 on each axis, up to float32 round-off. The second check confirms the first consequence of Section 9: for a fixed Δt the score does not depend on where the aligned pair sits. The third check shows that the two layouts give the same scores after a fixed channel permutation. To run a FLUX checkpoint with rotate_half, apply the inverse permutation to the rows of W_q and W_k in each head and to the QK-norm scales, because FLUX applies QK-norm before RoPE.</p><p>C4 prints the ids for L = 4, a 2 × 2 target, and references of 2 × 2 and 1 × 2 tokens.</p><pre>txt, img, ref = build_flux2_ids(txt_len=4, img_hw=(2, 2), ref_hws=[(2, 2), (1, 2)])<br>labels = ([f&quot;text j={j}&quot; for j in range(len(txt))] + [&quot;target&quot;] * len(img)<br>          + [&quot;ref 0&quot;] * 4 + [&quot;ref 1&quot;] * 2)<br>print(f&quot;{&#39;token&#39;:&lt;10}{&#39;t&#39;:&gt;4}{&#39;h&#39;:&gt;4}{&#39;w&#39;:&gt;4}{&#39;l&#39;:&gt;4}&quot;)<br>for name, (t, h, w, l) in zip(labels, torch.cat([txt, img, ref]).tolist()):  # (14, 4)<br>    print(f&quot;{name:&lt;10}{t:&gt;4}{h:&gt;4}{w:&gt;4}{l:&gt;4}&quot;)</pre><p>Output:</p><pre>token        t   h   w   l<br>text j=0     0   0   0   0<br>text j=1     0   0   0   1<br>text j=2     0   0   0   2<br>text j=3     0   0   0   3<br>target       0   0   0   0<br>target       0   0   1   0<br>target       0   1   0   0<br>target       0   1   1   0<br>ref 0       10   0   0   0<br>ref 0       10   0   1   0<br>ref 0       10   1   0   0<br>ref 0       10   1   1   0<br>ref 1       20   0   0   0<br>ref 1       20   0   1   0</pre><p>Rows 1 and 5 show the collision from Section 10.</p><h3>12. Design Trade-offs</h3><p>Start with the axis budget. FLUX.1 spent 16 channels on an axis that was zero everywhere except in Kontext. FLUX.2 gives the image index and the text index as many channels as each spatial axis, and the spatial axes pay for it: h and w drop from 28 to 16 frequencies, and the step between neighbors grows from 1.39 to 1.61.</p><p>Next, rope_theta. The models here use 256, 2000, 10000 and one million. A smaller base leaves fewer almost-static pairs on short axes and wraps around earlier, but for image grids wrap-around is far away even at 2000.</p><p>The reference offset is the third choice. Kontext uses Δt = 1 on a 16-channel axis at base 10000, where K(1) = 0.94, a weak signal. FLUX.2 uses steps of 10 on a 32-channel axis at base 2000, where K(10) = 0.57. The signal is much stronger, but it saturates: 10, 20 and 30 give almost the same K.</p><p>The last choice is the dummy axes for text. Any fixed (h, w) for text makes text-to-image logits depend on absolute patch positions; the anchor only decides where a possible preferred region would be. The anchor at (0, 0) also makes text token 0 collide with the top-left patch. A center anchor would make such an effect symmetric, but this is only my suggestion and I have not tested it.</p><p>Three questions I could not answer:</p><ol><li>Does anchoring text at the grid center (h = H/2, w = W/2) remove a measurable top-left bias in text-to-image attention?</li><li>With K almost flat between Δt = 10 and 30, do more than four references need a larger or non-uniform t step, or is the phase pattern of the fast pairs enough to keep them apart?</li><li>For references with another aspect ratio, do fractional (h, w) ids rescaled to the target grid, like the linspace grids of CogVideoX 1.0, improve spatial alignment in editing?</li></ol><h3>13. Key Takeaways</h3><ul><li>N-D axial RoPE is a block-diagonal rotation with one chunk per axis; the logit is a sum of per-axis terms that depends only on Δp.</li><li>Check how a model builds its spectrum: one schedule per axis (FLUX, Wan 2.1, HunyuanVideo, CogVideoX) or bands of one schedule (M-RoPE, where the Qwen2-VL 7B w band rotates by less than 0.02 rad across a 100-token-wide image).</li><li>An id of 0 is the identity rotation, so FLUX.1 text tokens are never rotated.</li><li>FLUX.2 places text at (0, 0, 0, j), the target at (0, y, x, 0) and reference i at (10 + 10i, y, x, 0), so aligned patches differ only on t.</li><li>With rope_theta = 2000, the slowest pair of a 32-channel axis has a period of 7,814 tokens, and 5 of 16 pairs are almost static over a 128-token side (7 with base 10000).</li><li>Position ids are routing labels: sequence order is irrelevant, and text-to-image logits depend on absolute positions.</li></ul><h3>References</h3><p>Part 1 of this series: [link to be added]</p><p>[1] RoFormer: Enhanced Transformer with Rotary Position Embedding, <a href="https://arxiv.org/abs/2104.09864">arXiv 2104.09864</a></p><p>[2] Rotary Position Embedding for Vision Transformer, <a href="https://arxiv.org/abs/2403.13298">arXiv 2403.13298</a></p><p>[3] HunyuanVideo: A Systematic Framework For Large Video Generative Models, <a href="https://arxiv.org/abs/2412.03603">arXiv 2412.03603</a></p><p>[4] Wan: Open and Advanced Large-Scale Video Generative Models, <a href="https://arxiv.org/abs/2503.20314">arXiv 2503.20314</a></p><p>[5] CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer, <a href="https://arxiv.org/abs/2408.06072">arXiv 2408.06072</a></p><p>[6] Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution, <a href="https://arxiv.org/abs/2409.12191">arXiv 2409.12191</a></p><p>[7] Extending Context Window of Large Language Models via Positional Interpolation, <a href="https://arxiv.org/abs/2306.15595">arXiv 2306.15595</a></p><p>[8] YaRN: Efficient Context Window Extension of Large Language Models, <a href="https://arxiv.org/abs/2309.00071">arXiv 2309.00071</a></p><p>[9] Qwen2.5-VL Technical Report, <a href="https://arxiv.org/abs/2502.13923">arXiv 2502.13923</a></p><p>[10] VideoRoPE: What Makes for Good Video Rotary Position Embedding?, <a href="https://arxiv.org/abs/2502.05173">arXiv 2502.05173</a></p><p>[11] Qwen3-VL Technical Report, <a href="https://arxiv.org/abs/2511.21631">arXiv 2511.21631</a></p><p>[12] FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space, <a href="https://arxiv.org/abs/2506.15742">arXiv 2506.15742</a></p><p>[13] diffusers Flux2Transformer2DModel docs: <a href="https://huggingface.co/docs/diffusers/main/api/models/flux2_transformer">huggingface.co/docs/diffusers/main/api/models/flux2_transformer</a></p><p>[14] FLUX.2 official inference repo: <a href="https://github.com/black-forest-labs/flux2">github.com/black-forest-labs/flux2</a></p><p>[15] RIFLEx: A Free Lunch for Length Extrapolation in Video Diffusion Transformers, <a href="https://arxiv.org/abs/2502.15894">arXiv 2502.15894</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=33b05b711554" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[21. Rotary Position Embedding from 1D to 2D: Derivation, Intuition, and PyTorch Code]]></title>
            <link>https://medium.com/@aminfadaeinejad.edu/21-rotary-position-embedding-from-1d-to-2d-derivation-intuition-and-pytorch-code-be76460daf06?source=rss-aff4faf055a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/be76460daf06</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Amin Fadaeinejad]]></dc:creator>
            <pubDate>Sat, 10 Oct 2026 05:35:54 GMT</pubDate>
            <atom:updated>2026-10-10T18:02:42.222Z</atom:updated>
            <content:encoded><![CDATA[<p>This is Part 1 of a two-part series on RoPE. It covers 1D RoPE, why it encodes relative position, and its 2D extension for Vision Transformers. Part 2 covers 3D and 4D RoPE in FLUX.2. The code blocks are meant to be run in order, and the shape comments use B, H, N and D for batch, heads, tokens and head dimension.</p><h4>1. Why Attention Needs Position Information</h4><p>Self-attention has no built-in notion of order. Take a token matrix <strong>X ∈ ℝ^{N×C}</strong>, a permutation matrix <strong>P</strong>, and the attention logits <strong>A = XW_q (XW_k)^T / √d</strong>. If we feed <strong>PX</strong> instead of <strong>X</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w4hinZX3JICJuFAkQra6hQ.png" /></figure><p>Permuting the tokens only permutes the rows of the output, so the layer cannot know which token came first. The second step holds because a row-wise softmax commutes with permuting rows and columns together, and <strong>P^T P = I</strong>. The same holds for a full Transformer block, because LayerNorm and the MLP work per token. A causal mask breaks this symmetry, but ViT encoders and diffusion Transformers use full attention and need an explicit position signal.</p><pre>import torch<br>import torch.nn.functional as F<br><br><br>def self_attention(x: torch.Tensor, w_q: torch.Tensor, w_k: torch.Tensor, w_v: torch.Tensor) -&gt; torch.Tensor:<br>    &quot;&quot;&quot;Single-head attention, no position information. x: (B, N, C).&quot;&quot;&quot;<br>    q, k, v = x @ w_q, x @ w_k, x @ w_v                               # (B, N, C) each<br>    scores = q @ k.transpose(-2, -1) / q.shape[-1] ** 0.5             # (B, N, N)<br>    return F.softmax(scores, dim=-1) @ v                              # (B, N, C)<br><br><br>torch.manual_seed(0)<br>B, N, C = 2, 6, 32<br>x = torch.randn(B, N, C)                                              # (B, N, C)<br>w_q, w_k, w_v = (torch.randn(C, C) / C**0.5 for _ in range(3))        # (C, C) each<br>perm = torch.randperm(N)                                              # (N,)<br><br>out = self_attention(x, w_q, w_k, w_v)                                # (B, N, C)<br>out_of_permuted = self_attention(x[:, perm], w_q, w_k, w_v)           # (B, N, C)<br>print(torch.allclose(out[:, perm], out_of_permuted, atol=1e-6))       # True</pre><h4>2. A Short History</h4><p><a href="https://arxiv.org/abs/1706.03762"><em>Attention Is All You Need</em></a><em> </em>[2] adds a fixed sinusoidal vector to each token embedding at the input:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lOSkNHroONJuy13_X1SkSA.png" /></figure><p>Each channel pair is a clock that ticks at its own speed, and RoPE reuses the same speeds.</p><p><a href="https://arxiv.org/abs/1810.04805"><em>BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding</em></a> [3] and <a href="https://arxiv.org/abs/2010.11929"><em>An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale</em></a> [4] use a learned table instead. Both variants are absolute and are added once, at the input.</p><p>Relative methods work inside attention. <a href="https://arxiv.org/abs/1803.02155"><em>Self-Attention with Relative Position Representations</em></a> [5] adds learned embeddings of the clipped distance to the keys and values. T5 (<a href="https://arxiv.org/abs/1910.10683"><em>Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer</em></a> [6]) uses one learned scalar per head for each of 32 log-spaced distance buckets. ALiBi (<a href="https://arxiv.org/abs/2108.12409"><em>Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation</em></a> [7]) uses a fixed distance penalty with a different slope per head. T5 and ALiBi have this form:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2Uf9Mo_joo0mTUc56xF9FQ.png" /></figure><p>Content and position are separate terms, and position is an extra bias on the attention logits.</p><p><a href="https://arxiv.org/abs/2104.09864"><em>RoFormer: Enhanced Transformer with Rotary Position Embedding</em></a> [1] moves relative position into the dot product itself, with a map <strong>f</strong> such that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ENE6zNtxlFEvDO-EK-AGgQ.png" /></figure><p>The score still depends on content, but position enters only as the offset <strong>n − m</strong>, and there is no bias term. Figure 1 shows the two approaches side by side.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C9DOeM3dU4xY4Z5XAs6RwA.png" /><figcaption>Figure 1. Left: sinusoidal encoding is added to the token embedding once, so q, k and v all carry absolute position. Right: RoPE rotates q and k (not v) in every layer, and the logit sees only n − m.</figcaption></figure><h4>3. Position as a Rotation</h4><p>RoPE encodes position <strong>m</strong> by rotating the query and the key by an angle proportional to <strong>m</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pChFQLCZR9qBRpYcQEvc9A.png" /></figure><p>The content goes through the usual projections, and position is applied afterwards as a rotation that depends only on the index.</p><ul><li>Only q and k are rotated. The target is a relative attention score, and a rotated v would write absolute position into the output features.</li><li>The rotation is applied in every layer and every head, after the Q/K projections and the head reshape (and after QK-norm, if present), so position is injected again in each layer.</li><li>RoPE adds no parameters, and a rotation preserves norms. With a KV cache, the keys are stored after the rotation.</li></ul><h4>4. RoPE in One 2D Subspace</h4><p>Start with a head dimension of 2. The rotation matrix and the position-dependent map are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PaEZd5JBY8L08etPGsGT7w.png" /></figure><p>The vector keeps its length and turns by <strong>mθ</strong>, so position changes only its direction.</p><p>Two properties of rotations are all we need later:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LoMvpqOZGmppjTRGW_H9zw.png" /></figure><p>Undoing one rotation and then applying another is the same as a single rotation by the difference in angles.</p><p>Figure 2 shows the consequence: the query turns by <strong>mθ</strong>, the key by <strong>nθ</strong>, and the angle between them changes by exactly <strong>(n − m)θ</strong>. With the lengths unchanged, the dot product depends only on the offset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JFOVW57FUl8sFqqKOqttng.png" /><figcaption>Figure 2. One 2D pair. q is rotated by mθ and k by nθ; the angle between the rotated vectors changes by (n − m)θ.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*HQNEdVj95L4M4akZfMeb_A.gif" /><figcaption>Animation of Figure 2: when m and n are shifted by the same amount, the angle between the rotated vectors and their dot product stay the same.</figcaption></figure><h4>5. Full 1D RoPE</h4><p>A real head dimension <strong>d</strong> is split into <strong>d/2</strong> pairs <strong>(q_{2i}, q_{2i+1})</strong>, and each pair rotates with its own frequency:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kvCKuiA3D-UOcllmnYOwJQ.png" /></figure><p>The matrix is block-diagonal, so each pair rotates independently and pairs never mix (Figure 3).</p><p>The frequencies follow the schedule of the sinusoidal encoding:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cj8vnaeRcs-UzhSv3beBdA.png" /></figure><p>Pair 0 turns by one radian per token, and every next pair is slower by the constant factor <strong>base^{2/d}</strong>.</p><p>The wavelength of pair <strong>i</strong>, the number of tokens for one full turn, is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*usv2antsIBCJXM_36JVu1A.png" /></figure><p>Wavelengths grow exponentially with <strong>i</strong>, so they cover all distance scales with the same density on a log axis. For <strong>d = 128</strong> the wavelengths go from <strong>2π ≈ 6.3</strong> to about 54,400 tokens, with a factor of 1.155 between neighbouring pairs (Figure 4). It works like the digits of a number: fast pairs resolve the last digit of the offset, slow pairs the leading digits. The base sets the longest wavelength, which is why long-context models use a larger one; the 7B checkpoint of <a href="https://arxiv.org/abs/2409.12191"><em>Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution</em></a> [18] uses 1,000,000.</p><p>In code, <strong>R_m</strong> is never built. With <strong>rot(q)</strong> mapping each pair <strong>(a, b)</strong> to <strong>(−b, a)</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zq4mXhPai_JdwUtq8rr3Tg.png" /></figure><p>A 2D rotation mixes a vector with its copy turned by 90 degrees, so two element-wise products replace the matrix, at <strong>O(Nd)</strong> cost.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XINTe7f5BvoDzh_6eKAaIw.png" /><figcaption>Figure 3. Block-diagonal R_m for d = 8: four 2x2 rotation blocks, from the fastest frequency θ_0 to the slowest θ_{d/2−1}, zeros everywhere else.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nj-T4nwUDIhzgh11QZ3h3Q.png" /><figcaption>Figure 4. RoPE frequency schedule for head dimension 128 on a log scale. The dashed line is the NTK-aware base for 4x context (Section 10).</figcaption></figure><h4>6. Relative Position Derivation</h4><p>The main result is the following identity:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PcsnX1ftTvzxpGQ_MeblfA.png" /></figure><p>The absolute positions cancel, and only their difference reaches the attention logits. The proof goes block by block. <strong>R_m^T</strong> and <strong>R_n</strong> are block-diagonal with the same <strong>2 × 2</strong> structure, so their product is block-diagonal too. Its <strong>i</strong>-th block is <strong>R(mθ_i)^T R(nθ_i) = R((n − m)θ_i)</strong> by the property from Section 4, which is the <strong>i</strong>-th block of <strong>R_{n−m}</strong>. For the full attention logit:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dCDAZvLSD3xIKUm7Q8KQgw.png" /></figure><p>This is the form RoFormer asked for: a function of the two contents and of <strong>n − m</strong>, with no bias term.</p><p>Two consequences follow. First, shifting all positions by a constant leaves every logit unchanged, so with full attention and no other absolute signal the whole network is shift-invariant (Section 9 tests this). Second, the property holds at the logit level only: <strong>R_m q</strong> alone still depends on <strong>m</strong>.</p><h4>7. Complex-Number View</h4><p>Each pair can be written as one complex number, and the rotation becomes a multiplication by a unit phasor:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K3a_bz57Q6E7hBH1sEFNCg.png" /></figure><p>Multiplying by <strong>e^{iα}</strong> turns a complex number by <strong>α</strong>, so the <strong>2 × 2</strong> matrix becomes one complex product. The dot product of two 2D vectors <strong>a</strong>, <strong>b</strong> equals <strong>Re(ã conj(b̃))</strong>, so the relative property follows in one line:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MrtWWywExB2eInhVf8QcJQ.png" /></figure><p>The two phase factors merge into one factor that depends only on <strong>m − n</strong> (the sign differs from Section 6 only because the conjugate sits on the key).</p><p>This is the original RoFormer formulation, and also how the Meta reference code of <a href="https://arxiv.org/abs/2302.13971"><em>LLaMA: Open and Efficient Foundation Language Models</em></a> [8] implements RoPE: torch.polar builds the phasors, and torch.view_as_complex groups channels <strong>2i</strong> and <strong>2i + 1</strong> into complex number <strong>i</strong>. So this code uses the interleaved layout, which matters in Section 9.</p><h4>8. Frequency Intuition</h4><p>If we write <strong>q̃_i conj(k̃_i) = |q̃_i||k̃_i| e^{iφ_i}</strong>, the attention logit becomes a sum of cosines of the offset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yHuAAP17fy4IQmYdzi__pg.png" /></figure><p>Each pair adds a cosine of the distance at its own frequency, and the content decides the amplitude and the phase of each term. The two ends of the spectrum do different jobs:</p><ul><li>High-frequency pairs turn by up to one radian (57°) per token and separate direct neighbors well, but they wrap around after <strong>λ_i</strong> tokens: pair 0 alone can hardly tell distance 1 from distance 7.</li><li>Low-frequency pairs are almost constant over short ranges. At <strong>d = 128</strong> the slowest pair turns by 0.66° over 100 tokens, and 18 of the 64 pairs never complete one turn inside a 4,096-token context (Figure 4). They carry coarse long-range position, and they meet unseen angles when the context grows (Section 10).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*q1qd-M-ow2cbo2y1SbmQ6A.gif" /><figcaption>Animation: the four pairs of a head with d = 8 (base 10000) as clocks. Pair 0 turns about 9.5 times while position m goes from 0 to 60, and pair 3 barely moves.</figcaption></figure><p>Sec. 3.4.3 of <em>RoFormer: Enhanced Transformer with Rotary Position Embedding</em> [1] also derives a long-term decay. With <strong>h_i = q̃_i conj(k̃_i)</strong> (with <strong>h_{d/2} = 0</strong>) and the partial sums <strong>S_j = Σ_{k&lt;j} e^{i(m−n)θ_k}</strong>, Abel summation gives:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tI1JWY14Lue21wXn356cgA.png" /></figure><p>The content part is bounded by how much <strong>h</strong> changes between neighboring pairs, and the position part is a sum of unit phasors that cancel more when the distance grows. Figure 5 plots the mean of <strong>|S_j|</strong>. It is 32.5 at distance 0, about 10 at distance 100, and it oscillates between about 4 and 12 up to distance 1,024.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0JgIPJC8V6Lz7U08czQOTA.png" /><figcaption>Figure 5. Relative upper bound from RoFormer, Sec. 3.4.3, for d = 128 and base 10000.</figcaption></figure><p>This is only an upper bound, and trained models do not have to follow it. <a href="https://arxiv.org/abs/2410.06205"><em>Round and Round We Go! What makes Rotary Positional Encodings useful?</em></a> [11] found that Gemma 7B uses the highest frequencies for sharp positional attention patterns and prefers the lowest frequencies, which barely rotate, for semantic information.</p><h4>9. Implementing 1D RoPE</h4><p>The implementation has three parts: a cos/sin cache, a function that applies the rotation, and the call inside attention.</p><pre>import torch<br><br><br>def build_rope_cache(pos: torch.Tensor, head_dim: int, base: float = 10000.0) -&gt; tuple[torch.Tensor, torch.Tensor]:<br>    &quot;&quot;&quot;cos/sin of the rotation angles pos * theta_i. Always float32. pos: (N,).&quot;&quot;&quot;<br>    i = torch.arange(head_dim // 2, dtype=torch.float32)             # (D/2,)<br>    theta = base ** (-2.0 * i / head_dim)                             # (D/2,)<br>    angles = torch.outer(pos.to(torch.float32), theta)               # (N, D/2)<br>    return angles.cos(), angles.sin()                                 # (N, D/2) each<br><br><br>def rotate_half(x: torch.Tensor) -&gt; torch.Tensor:<br>    x1, x2 = x.chunk(2, dim=-1)                                       # (..., D/2) each<br>    return torch.cat([-x2, x1], dim=-1)                               # (..., D)<br><br><br>def apply_rope_half(x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -&gt; torch.Tensor:<br>    &quot;&quot;&quot;GPT-NeoX / Hugging Face LLaMA layout: pair i is (x[i], x[i + D/2]).<br>    x: (B, H, N, D), cos/sin: (N, D/2).&quot;&quot;&quot;<br>    cos = torch.cat([cos, cos], dim=-1)                               # (N, D)<br>    sin = torch.cat([sin, sin], dim=-1)                               # (N, D)<br>    x32 = x.float()                                                   # (B, H, N, D)<br>    out = x32 * cos + rotate_half(x32) * sin                          # (B, H, N, D)<br>    return out.to(x.dtype)                                            # (B, H, N, D)<br><br><br># Usage inside attention, after the Q/K projections and the head reshape:<br>B, H, N, D = 2, 4, 16, 64<br>q, k = torch.randn(B, H, N, D), torch.randn(B, H, N, D)               # (B, H, N, D) each<br>cos, sin = build_rope_cache(torch.arange(N), D)                       # (N, D/2) each<br>q, k = apply_rope_half(q, cos, sin), apply_rope_half(k, cos, sin)     # (B, H, N, D) each<br># v is not rotated; continue with F.scaled_dot_product_attention(q, k, v)</pre><p>The angles are computed in float32, even when the model runs in bf16. bf16 has 8 significant bits, so 257 is not representable and 1001 rounds to 1000. In my quick test, bf16 angles near position 1,000 were off by 2 to 5 rad, depending on how much of the computation runs in bf16. For the fast pairs this is an almost random rotation.</p><p>There are two conventions for which channels form a pair (Figure 6):</p><ul><li><strong>Interleaved:</strong> pair <strong>i</strong> is <strong>(x_{2i}, x_{2i+1})</strong>. This is the RoFormer layout. GPT-J uses it (rotate_every_two in the <a href="https://github.com/kingoflolz/mesh-transformer-jax">Mesh Transformer JAX repository</a> [10]), and the Meta reference code of LLaMA uses it through complex numbers.</li><li><strong>Half-split, rotate_half:</strong> pair <strong>i</strong> is <strong>(x_i, x_{i+d/2})</strong>. This is the layout of <a href="https://arxiv.org/abs/2204.06745"><em>GPT-NeoX-20B: An Open-Source Autoregressive Language Model</em></a> [9] and of the Hugging Face LLaMA implementation.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g3ihDhlN0JxiyLoszKzPnQ.png" /><figcaption>Figure 6. Head vector of size 8. Top: interleaved pairs (0,1), (2,3), (4,5), (6,7). Bottom: rotate_half pairs (0,4), (1,5), (2,6), (3,7). Same colors mean the same frequency.</figcaption></figure><pre>def apply_rope_interleaved(x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -&gt; torch.Tensor:<br>    &quot;&quot;&quot;GPT-J / RoFormer / Meta LLaMA layout: pair i is (x[2i], x[2i + 1]).<br>    x: (B, H, N, D), cos/sin: (N, D/2).&quot;&quot;&quot;<br>    x32 = x.float()                                                   # (B, H, N, D)<br>    x_even, x_odd = x32[..., 0::2], x32[..., 1::2]                    # (B, H, N, D/2) each<br>    out_even = x_even * cos - x_odd * sin                             # (B, H, N, D/2)<br>    out_odd = x_even * sin + x_odd * cos                              # (B, H, N, D/2)<br>    out = torch.stack([out_even, out_odd], dim=-1).flatten(-2)        # (B, H, N, D)<br>    return out.to(x.dtype)                                            # (B, H, N, D)<br><br><br>def interleaved_to_half_perm(head_dim: int) -&gt; torch.Tensor:<br>    &quot;&quot;&quot;Channel order that turns the interleaved layout into the half-split layout.<br>    Apply it per head to the output rows of W_q and W_k when converting a checkpoint.&quot;&quot;&quot;<br>    return torch.cat([torch.arange(0, head_dim, 2), torch.arange(1, head_dim, 2)])  # (D,)</pre><p>The two layouts are related by a fixed permutation of the head dimension. A model trained with one layout has learned to put related features into the paired channels. With the other layout, the rotation mixes channels that were never paired in training. Nothing crashes because the shapes match, but the attention logits are wrong. The fix is to permute the output rows of <strong>W_q</strong> and <strong>W_k</strong> per head, and to leave <strong>W_v</strong> unchanged. The Hugging Face conversion script for the Meta LLaMA checkpoints does this (“permute for sliced rotary”). Since q and k get the same permutation, the dot products do not change.</p><p>The sanity check tests shift invariance and the layout equivalence:</p><pre>def rope_logits(q: torch.Tensor, k: torch.Tensor, pos: torch.Tensor, apply_fn) -&gt; torch.Tensor:<br>    cos, sin = build_rope_cache(pos, q.shape[-1])                     # (N, D/2) each<br>    q_rot, k_rot = apply_fn(q, cos, sin), apply_fn(k, cos, sin)       # (B, H, N, D) each<br>    return q_rot @ k_rot.transpose(-2, -1)                            # (B, H, N, N)<br><br><br>torch.manual_seed(0)<br>B, H, N, D = 1, 2, 16, 64<br>q, k = torch.randn(B, H, N, D), torch.randn(B, H, N, D)               # (B, H, N, D) each<br>pos = torch.arange(N)                                                 # (N,)<br>ref = rope_logits(q, k, pos, apply_rope_half)                         # (B, H, N, N)<br><br># (a) Shift every position by the same constant: the logits must not change.<br>shifted = rope_logits(q, k, pos + 100, apply_rope_half)               # (B, H, N, N)<br>print(f&quot;(a) shift by 100, max |diff|: {(ref - shifted).abs().max():.1e}   logit std: {ref.std():.1f}&quot;)<br><br># (b) Interleaved equals rotate_half after a fixed channel permutation.<br>perm = interleaved_to_half_perm(D)                                    # (D,)<br>inter = rope_logits(q, k, pos, apply_rope_interleaved)                # (B, H, N, N)<br>half_perm = rope_logits(q[..., perm], k[..., perm], pos, apply_rope_half)  # (B, H, N, N)<br>print(f&quot;(b) interleaved vs permuted rotate_half: {(inter - half_perm).abs().max():.1e}&quot;)<br><br># (c) The same q, k with the other layout and no permutation: silently different logits.<br>print(f&quot;(c) interleaved vs rotate_half, no permutation: {(inter - ref).abs().max():.1e}&quot;)</pre><pre>(a) shift by 100, max |diff|: 2.6e-05   logit std: 8.0<br>(b) interleaved vs permuted rotate_half: 6.7e-06<br>(c) interleaved vs rotate_half, no permutation: 2.7e+01</pre><p>The residual in (a) and (b) is float32 rounding. Check (c) is the silent failure: the other layout moves the logits by up to 27 on a scale of 8.</p><h4>10. Context Extension</h4><p>A model trained with context length <strong>L</strong> is used at <strong>L′ = sL</strong>. The problem is the slow pairs: beyond <strong>L</strong> they reach angles they never saw in training.</p><p><a href="https://arxiv.org/abs/2306.15595"><em>Extending Context Window of Large Language Models via Positional Interpolation</em></a> [12] rescales the positions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*abxaNbwIYDSGkn_fmBc1hw.png" /></figure><p>All angles stay inside the trained range, but the fast pairs also slow down by <strong>s</strong>, which blurs local order, so the paper adds a short fine-tune (within 1,000 steps).</p><p>NTK-aware scaling, first shared as a community post and formalized in <a href="https://arxiv.org/abs/2309.00071"><em>YaRN: Efficient Context Window Extension of Large Language Models</em></a> [13], changes the base instead:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1g2j2JSeKZ7n9-nTamGTKw.png" /></figure><p>The fastest pair keeps its frequency, and the exponent is chosen so that the slowest pair is interpolated by exactly <strong>s</strong> (dashed line in Figure 4).</p><p>YaRN decides per pair. With <strong>r_i = L/λ_i</strong>, the number of full turns of pair <strong>i</strong> inside the training context, and a ramp <strong>γ</strong> that is 0 below <strong>α</strong> and 1 above <strong>β</strong> (<strong>α = 1</strong>, <strong>β = 32</strong> for LLaMA):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JTLSbcFM3Tgc5N79kB2Ztg.png" /></figure><p>Pairs that turned many times during training keep their frequency, and pairs that never completed one turn are interpolated. YaRN also divides the logits by a temperature <strong>t</strong>, with <strong>√(1/t) = 0.1 ln s + 1</strong>.</p><h4>11. From Sequences to Images</h4><p>A ViT turns an image into an <strong>H × W</strong> grid of patches and flattens it row by row:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ftue27G3yQ49y15hu_-tFg.png" /></figure><p>The index walks along a row and then jumps to the start of the next row. If we apply 1D RoPE to <strong>n</strong>, the 2D structure is lost (Figure 7). A horizontal neighbor is at distance 1, but a vertical neighbor is at distance <strong>W</strong>, which is 14 for a 224 px image with 16 px patches. At the end of each row, patches <strong>(r, W−1)</strong> and <strong>(r+1, 0)</strong> are neighbors in the index but far apart in the image. Also, a 2D offset <strong>(Δc, Δr)</strong> maps to the 1D distance <strong>Δc + WΔr</strong>, so its positional code changes with the resolution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IwYAuVDIcXonZLPz1ibFuA.png" /><figcaption>Figure 7. A 4x4 patch grid flattened row by row. The right neighbor of patch 5 is at distance 1, its lower neighbor at distance 4. Patches 3 and 4 are adjacent in the index but far apart in the image.</figcaption></figure><h4>12. Axial 2D RoPE</h4><p>Axial 2D RoPE gives each axis its own part of the head dimension. For a patch at column <strong>x</strong> and row <strong>y</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w8sVTPCnv_l3CVZbLP2uPw.png" /></figure><p>The first half of the head dimension is a 1D RoPE over columns, the second half is a 1D RoPE over rows, and the two halves do not interact (Figure 8). <strong>R^{(d/2)}_x</strong> is the 1D RoPE matrix of Section 5 for dimension <strong>d/2</strong>, and each half has its own schedule:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mTg15ILM82PY-KTtnsAszg.png" /></figure><p>Each half has <strong>d/4</strong> pairs and gets a complete geometric schedule from fast to slow. Applying Section 6 to each half gives the attention score:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tIe526M-TQ9KaMrJzDigTQ.png" /></figure><p>The logit is a 1D RoPE score along x plus a 1D RoPE score along y, so it depends only on the 2D offset <strong>(Δx, Δy) = (x_n − x_m, y_n − y_m)</strong>. Vertical and horizontal neighbours are now treated the same way, and the code of an offset no longer depends on <strong>W</strong>.</p><p>The base needs a second look, because a patch grid has only 14 to 32 positions per axis. <a href="https://arxiv.org/abs/2403.13298"><em>Rotary Position Embedding for Vision Transformer</em></a> [14] uses base 100 for the axial variant. The paper does not explain this, but a quick computation does: with <strong>d = 64</strong> and base 10000, the slowest x-pair turns by only 0.13° across a 14-patch row; with base 100 it turns by about 10°. The <a href="https://github.com/naver-ai/rope-vit">reference code</a> uses the same split as here (first half x, second half y) but interleaved pairs inside each half, while the code below uses rotate_half. Match the layout when you load their checkpoints.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6kPFFPpuNNBVgxgSqMfHeg.png" /><figcaption>Figure 8. The head vector split into an x-half (rotated by the column index) and a y-half (rotated by the row index), each with its own block-diagonal rotation.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*lRPIJZwQxzpkdDzmYU8pGQ.gif" /><figcaption>Animation of Figure 8: moving the patch along x turns only the clocks of the x-half, and moving it along y turns only the clocks of the y-half.</figcaption></figure><h4>13. Limitation and RoPE-Mixed</h4><p>In axial RoPE, the phase of every pair depends on one axis only:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KPZRzxRvByokpqlIKsNsUw.png" /></figure><p>Every pair is a wave that runs either horizontally or vertically. The positional part of the logit has the form <strong>g(Δx) + h(Δy)</strong>. A diagonal wave, <strong>cos(ωΔx + ωΔy) = cos ωΔx cos ωΔy − sin ωΔx sin ωΔy</strong>, contains products of x and y terms, so it is not such a sum. Axial RoPE can still attend to a diagonal neighbour, but it cannot represent oriented periodic patterns directly.</p><p><em>Rotary Position Embedding for Vision Transformer</em> [14] (Heo et al., ECCV 2024) proposes RoPE-Mixed, where every pair uses both axes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mIvssPnXtFo3oSQbHltYeg.png" /></figure><p>Each pair is a plane wave in the offset plane, with its own direction and frequency (Figure 9). The frequencies <strong>(θ^x_t, θ^y_t)</strong> are learnable, separate for every head and every layer. Since the phase is linear in position, the relative property still holds, and axial RoPE is the special case where one component is zero for every pair. The reference code initializes each head with axial-style magnitudes (base 10) in two orthogonal directions, rotated by a random angle. The cost is <strong>d</strong> parameters per head and layer, plus one cos/sin cache per layer.</p><p>The paper trains ViT-B on ImageNet-1k at 224 px and tests it at other resolutions (top-1, Table A.4):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VixcxmzG1SGv-QKDiq7hZQ.png" /></figure><p>From 224 px to 512 px, the absolute position embedding (APE) loses 2.9 points, and RoPE-Mixed loses 0.9. RoPE-Mixed also improves COCO detection (ViT-L: 51.1 to 52.9 AP) and ADE20k segmentation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nDj9U4uaSquvhMuRYxOTgg.png" /><figcaption>Figure 9. Frequency vectors in the (Δx, Δy) plane. Axial: every arrow lies on the x-axis or the y-axis. RoPE-Mixed: arrows in any direction, learned per head, with diagonal iso-phase lines.</figcaption></figure><h4>14. Implementing 2D RoPE for a ViT</h4><p>2D RoPE reuses the 1D functions. The position grid comes from torch.meshgrid with indexing=&quot;ij&quot;, flattened row by row. This matches the token order of a Conv2d patch embedding followed by flatten(2). If the two orders differ, x and y are silently swapped.</p><pre>def build_2d_positions(h: int, w: int) -&gt; tuple[torch.Tensor, torch.Tensor]:<br>    &quot;&quot;&quot;Row-major (x, y) coordinates of an h x w patch grid.&quot;&quot;&quot;<br>    ys, xs = torch.meshgrid(torch.arange(h), torch.arange(w), indexing=&quot;ij&quot;)  # (h, w) each<br>    return xs.flatten().float(), ys.flatten().float()                 # (N,), (N,) with N = h * w<br><br><br>def build_axial_cache(xs: torch.Tensor, ys: torch.Tensor, head_dim: int,<br>                      base: float = 100.0) -&gt; tuple[torch.Tensor, ...]:<br>    &quot;&quot;&quot;Each axis owns head_dim / 2 channels (head_dim / 4 pairs) and its own schedule.&quot;&quot;&quot;<br>    cos_x, sin_x = build_rope_cache(xs, head_dim // 2, base)          # (N, D/4) each<br>    cos_y, sin_y = build_rope_cache(ys, head_dim // 2, base)          # (N, D/4) each<br>    return cos_x, sin_x, cos_y, sin_y<br><br><br>def grid_positions(h: int, w: int, train_hw: tuple[int, int] | None = None) -&gt; tuple[torch.Tensor, torch.Tensor]:<br>    &quot;&quot;&quot;train_hw=None: integer positions (extrapolation, as in RoPE-ViT).<br>    train_hw=(h0, w0): positions scaled into the training range (interpolation).&quot;&quot;&quot;<br>    xs, ys = build_2d_positions(h, w)                                 # (N,), (N,)<br>    if train_hw is not None:<br>        xs, ys = xs * train_hw[1] / w, ys * train_hw[0] / h           # (N,), (N,)<br>    return xs, ys</pre><pre>def apply_rope_2d(x: torch.Tensor, cache: tuple[torch.Tensor, ...], num_prefix_tokens: int = 1) -&gt; torch.Tensor:<br>    &quot;&quot;&quot;Axial 2D RoPE. x: (B, H, P + N, D) with P prefix tokens (CLS, registers) first.<br>    First half of the head dim is rotated by x (column), second half by y (row).&quot;&quot;&quot;<br>    cos_x, sin_x, cos_y, sin_y = cache                                # (N, D/4) each<br>    prefix = x[..., :num_prefix_tokens, :]                            # (B, H, P, D)<br>    patches = x[..., num_prefix_tokens:, :]                           # (B, H, N, D)<br>    p_x, p_y = patches.chunk(2, dim=-1)                               # (B, H, N, D/2) each<br>    p_x = apply_rope_half(p_x, cos_x, sin_x)                          # (B, H, N, D/2)<br>    p_y = apply_rope_half(p_y, cos_y, sin_y)                          # (B, H, N, D/2)<br>    patches = torch.cat([p_x, p_y], dim=-1)                           # (B, H, N, D)<br>    return torch.cat([prefix, patches], dim=-2)                       # (B, H, P + N, D)<br><br><br># Usage in a ViT block with a 14 x 14 patch grid and one CLS token:<br>B, H, D, h, w = 2, 6, 64, 14, 14<br>q = torch.randn(B, H, 1 + h * w, D)                                   # (B, H, 1 + N, D)<br>k = torch.randn(B, H, 1 + h * w, D)                                   # (B, H, 1 + N, D)<br>cache = build_axial_cache(*grid_positions(h, w), D)                   # 4 x (N, D/4)<br>q, k = apply_rope_2d(q, cache), apply_rope_2d(k, cache)               # (B, H, 1 + N, D) each</pre><p><strong>CLS token.</strong> The RoPE-ViT reference code rotates only the patch tokens. “No rotation” and “position (0, 0)” are the same thing: an unrotated token behaves exactly like a token rotated by angle zero. So the CLS token shares its position with the top-left patch, and its logits with the patches depend on their offset from that corner. The same holds for the register tokens of <a href="https://arxiv.org/abs/2309.16588"><em>Vision Transformers Need Registers</em></a> [15]. This works in practice (the results in Section 13 use it). For a symmetric positional term, one option is to place the CLS token at the grid center, <strong>((W−1)/2, (H−1)/2)</strong>. This placement is my own suggestion; I have not seen it in a paper.</p><p><strong>Resolution change.</strong> grid_positions supports both options. Extrapolation keeps integer positions, so the grid grows and larger offsets appear than in training; RoPE-ViT does this. Interpolation scales the positions into the training range, so offsets stay in range, but one patch step becomes a fraction of a step. The trade-off is the same as in Section 10, and I would evaluate both at the target resolution.</p><pre>def rope_2d_logits(q: torch.Tensor, k: torch.Tensor, xs: torch.Tensor, ys: torch.Tensor) -&gt; torch.Tensor:<br>    cache = build_axial_cache(xs, ys, q.shape[-1])                    # 4 x (N, D/4)<br>    q_rot, k_rot = apply_rope_2d(q, cache), apply_rope_2d(k, cache)   # (B, H, 1 + N, D) each<br>    return q_rot @ k_rot.transpose(-2, -1)                            # (B, H, 1 + N, 1 + N)<br><br><br>torch.manual_seed(0)<br>B, H, D, h, w = 1, 2, 64, 7, 9<br>q = torch.randn(B, H, 1 + h * w, D)                                   # (B, H, 1 + N, D)<br>k = torch.randn(B, H, 1 + h * w, D)                                   # (B, H, 1 + N, D)<br>xs, ys = build_2d_positions(h, w)                                     # (N,), (N,)<br>ref = rope_2d_logits(q, k, xs, ys)                                    # (B, H, 1 + N, 1 + N)<br>shifted = rope_2d_logits(q, k, xs + 5, ys + 3)                        # (B, H, 1 + N, 1 + N)<br><br>diff = (ref - shifted).abs()                                          # (B, H, 1 + N, 1 + N)<br>print(f&quot;patch-patch logits, max |diff|: {diff[..., 1:, 1:].max():.1e}&quot;)    # ~1e-6<br>print(f&quot;CLS-patch logits,   max |diff|: {diff[..., 0, 1:].max():.1e}&quot;)      # large: CLS acts as (0, 0)<br><br># Same (dx, dy) = (2, 1) offset at two places in the grid gives the same positional term:<br>unit_q, unit_k = torch.ones(1, 1, 1 + h * w, D), torch.ones(1, 1, 1 + h * w, D)  # (1, 1, 1 + N, D)<br>L = rope_2d_logits(unit_q, unit_k, xs, ys)[0, 0, 1:, 1:]             # (N, N)<br>idx = lambda x, y: y * w + x                                          # row-major index<br>print(f&quot;{L[idx(0, 0), idx(2, 1)]:.4f} == {L[idx(4, 3), idx(6, 4)]:.4f}&quot;)</pre><pre>patch-patch logits, max |diff|: 1.5e-05<br>CLS-patch logits,   max |diff|: 2.1e+01<br>54.4425 == 54.4425</pre><p>The patch-patch logits do not change when the grid is shifted by (5, 3). The CLS-patch logits change because the CLS token stays at (0, 0) while the patches move. The last line shows the same positional term for the offset (2, 1) at two places in the grid.</p><h4>15. Beyond 2D</h4><p>The axial split generalizes directly to more axes. In video diffusion Transformers, every token has a position <strong>(t, h, w)</strong>, and the head dimension is split into three chunks, each a 1D RoPE with its own schedule. <a href="https://arxiv.org/abs/2412.03603"><em>HunyuanVideo: A Systematic Framework For Large Video Generative Models</em></a> [16] splits a 128-dim head into 16, 56 and 56 channels for <strong>(t, h, w)</strong>, and <a href="https://arxiv.org/abs/2503.20314"><em>Wan: Open and Advanced Large-Scale Video Generative Models</em></a> [17] uses 44, 42 and 42.</p><p>M-RoPE in <em>Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution</em> [18] applies the same idea to a multimodal sequence. Text tokens get equal <strong>(t, h, w)</strong> IDs, so for text it reduces to 1D RoPE. Image tokens share one <strong>t</strong> and take <strong>(h, w)</strong> from the patch grid, and video frames increment <strong>t</strong>. Unlike the axial split, Qwen2-VL cuts one 64-pair schedule into contiguous bands (mrope_section = [16, 24, 24]): <strong>t</strong> gets the 16 fastest pairs, <strong>h</strong> the next 24, <strong>w</strong> the 24 slowest. The <a href="https://arxiv.org/abs/2511.21631"><em>Qwen3-VL Technical Report</em></a> [19] replaces this with interleaved MRoPE, which spreads all three axes over the full spectrum. Figure 10 compares the three allocations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y7q28lIsxrDrU6yRWO_mDg.png" /><figcaption>Figure 10. Frequency allocation over 64 pairs: contiguous bands [16 t | 24 h | 24 w] in Qwen2-VL, interleaved t, h, w in Qwen3-VL, and three independent schedules in video diffusion Transformers.</figcaption></figure><p>Part 2 continues from here with 3D and 4D RoPE in FLUX.2.</p><h4>16. Summary</h4><ul><li>RoPE rotates q and k, never v, after the Q/K projections in every layer. The attention logits depend on content and on <strong>n − m</strong> only.</li><li>The wavelengths grow geometrically, from about 6 to 54,000 tokens at <strong>d = 128</strong> and base 10000. For short patch grids, a base of 100 keeps the slow pairs useful.</li><li>Interleaved and rotate_half differ only in channel pairing. With pretrained weights, the wrong layout fails silently; permute the rows of <strong>W_q</strong> and <strong>W_k</strong> per head to convert.</li><li>Compute rotation angles in float32. bf16 rounds positions above 256.</li><li>Flattening plus 1D RoPE breaks vertical adjacency. Axial 2D RoPE fixes this, and RoPE-Mixed adds learnable oblique frequencies that extrapolate better to higher resolutions.</li><li>An unrotated CLS token sits at position (0, 0). At a new resolution, recompute the grid and evaluate both extrapolation and interpolation.</li></ul><h4>References</h4><p>[1] <a href="https://arxiv.org/abs/2104.09864"><em>RoFormer: Enhanced Transformer with Rotary Position Embedding</em></a>, Su et al., 2021.<br>[2] <a href="https://arxiv.org/abs/1706.03762"><em>Attention Is All You Need</em></a>, Vaswani et al., 2017.<br>[3] <a href="https://arxiv.org/abs/1810.04805"><em>BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding</em></a>, Devlin et al., 2018.<br>[4] <a href="https://arxiv.org/abs/2010.11929"><em>An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale</em></a>, Dosovitskiy et al., 2020.<br>[5] <a href="https://arxiv.org/abs/1803.02155"><em>Self-Attention with Relative Position Representations</em></a>, Shaw et al., 2018.<br>[6] <a href="https://arxiv.org/abs/1910.10683"><em>Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer</em></a>, Raffel et al., 2019.<br>[7] <a href="https://arxiv.org/abs/2108.12409"><em>Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation</em></a>, Press et al., 2021.<br>[8] <a href="https://arxiv.org/abs/2302.13971"><em>LLaMA: Open and Efficient Foundation Language Models</em></a>, Touvron et al., 2023.<br>[9] <a href="https://arxiv.org/abs/2204.06745"><em>GPT-NeoX-20B: An Open-Source Autoregressive Language Model</em></a>, Black et al., 2022.<br>[10] GPT-J-6B: <a href="https://github.com/kingoflolz/mesh-transformer-jax">Mesh Transformer JAX repository</a>, Wang and Komatsuzaki, 2021.<br>[11] <a href="https://arxiv.org/abs/2410.06205"><em>Round and Round We Go! What makes Rotary Positional Encodings useful?</em></a>, Barbero et al., 2024.<br>[12] <a href="https://arxiv.org/abs/2306.15595"><em>Extending Context Window of Large Language Models via Positional Interpolation</em></a>, Chen et al., 2023.<br>[13] <a href="https://arxiv.org/abs/2309.00071"><em>YaRN: Efficient Context Window Extension of Large Language Models</em></a>, Peng et al., 2023.<br>[14] <a href="https://arxiv.org/abs/2403.13298"><em>Rotary Position Embedding for Vision Transformer</em></a>, Heo et al., ECCV 2024. Code: <a href="https://github.com/naver-ai/rope-vit">naver-ai/rope-vit</a>.<br>[15] <a href="https://arxiv.org/abs/2309.16588"><em>Vision Transformers Need Registers</em></a>, Darcet et al., 2023.<br>[16] <a href="https://arxiv.org/abs/2412.03603"><em>HunyuanVideo: A Systematic Framework For Large Video Generative Models</em></a>, Kong et al., 2024.<br>[17] <a href="https://arxiv.org/abs/2503.20314"><em>Wan: Open and Advanced Large-Scale Video Generative Models</em></a>, Team Wan, 2025.<br>[18] <a href="https://arxiv.org/abs/2409.12191"><em>Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution</em></a>, Wang et al., 2024.<br>[19] <a href="https://arxiv.org/abs/2511.21631"><em>Qwen3-VL Technical Report</em></a>, Bai et al., 2025.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=be76460daf06" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[20. How Transformers Work: Building One Module at a Time]]></title>
            <link>https://medium.com/@aminfadaeinejad.edu/20-how-transformers-work-building-one-module-at-a-time-a1e96e3e801c?source=rss-aff4faf055a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/a1e96e3e801c</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[transformers]]></category>
            <dc:creator><![CDATA[Amin Fadaeinejad]]></dc:creator>
            <pubDate>Thu, 20 Aug 2026 01:30:38 GMT</pubDate>
            <atom:updated>2026-08-20T01:30:38.513Z</atom:updated>
            <content:encoded><![CDATA[<p>Consider this sentence:</p><blockquote><em>The animal didn’t cross the street because it was too tired.</em></blockquote><p>What does “it” refer to, the animal or the street? You resolved that instantly by weighing “tired” against both candidates. A street is not the kind of thing that gets tired. An animal is.</p><p>That weighing is the entire idea behind attention. Not read left to right and hope the relevant word is still in memory,” but “look at every word at once and decide, for each one, how much it matters right now.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/1*v5TWJGEdC9mOwMScXBTJbw.gif" /></figure><p>This post builds a Transformer one module at a time. Each section is a piece of <a href="https://arxiv.org/abs/1706.03762"><em>Attention Is All You Need</em></a> (Vaswani et al., 2017), the code that implements it, and a pointer to the ManimGL scene in this repo that animates it.</p><h3>Setup</h3><pre>import math<br><br>import torch<br>import torch.nn as nn<br>import torch.nn.functional as F<br>torch.manual_seed(0)</pre><p>Notation used throughout:</p><ul><li>B : Batch size (8–512)</li><li>T : Sequence length in tokens (512–128k)</li><li>d_model : Model width: 512 in the base paper, 12288 in GPT-3 175B</li><li>h : Number of attention heads: 8 in the base paper, 96 in GPT-3 175B</li><li>d_head : Width per head, d_model // h, conventionally 64</li><li>d_ff : Feed-forward inner width, usually 4 * d_model, so 2048</li></ul><p>Tensors are (B, T, d_model) everywhere except inside attention, where they briefly become (B, h, T, d_head).</p><h3>1. Tokens and embeddings</h3><p>Text is not a sequence of words. It is a sequence of token IDs, integer indices into a fixed vocabulary. Modern tokenizers split on sub-words, so &quot;tokenizing&quot; might become [&quot;token&quot;, &quot;izing&quot;]. That keeps the vocabulary finite (typically 32k–200k entries) while still representing any string.</p><p>Each ID indexes one row of an embedding matrix. That row is the token’s vector, and it is learned by gradient descent like any other parameter.</p><pre>class TokenEmbedding(nn.Module):<br>    &quot;&quot;&quot;Vocabulary lookup table: token id -&gt; d_model vector.&quot;&quot;&quot;<br><br>    def __init__(self, vocab_size: int, d_model: int):<br>        super().__init__()<br>        self.embed = nn.Embedding(vocab_size, d_model)<br>        self.d_model = d_model<br><br>    def forward(self, ids: torch.Tensor) -&gt; torch.Tensor:<br>        # (B, T) -&gt; (B, T, d_model)<br>        # The sqrt(d_model) scaling is from section 3.4 of the paper: it puts<br>        # the embeddings on the same scale as the positional signal we are<br>        # about to add, which otherwise swamps them at initialisation.<br>        return self.embed(ids) * math.sqrt(self.d_model)</pre><p>Because these vectors are trained, geometry in that space ends up meaning something: cat and kitten land near each other, car lands far away. Nothing enforces this; it falls out of the objective.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/1*MJAbj5kgOuOmRcBBsUhF-w.gif" /></figure><h3>2. Positional encoding</h3><p>Here is the catch that makes attention work at all and the hole it leaves.</p><p>Attention, as we are about to define it, is a weighted sum over the whole sequence. Sums do not care about order. Feed in &quot;cat sat&quot; or &quot;sat cat&quot; and you get the same set of outputs, just permuted. The mechanism is permutation equivariant, and on its own it cannot tell subject from object.</p><p>So position gets added directly into the vector before the first layer:</p><pre>class SinusoidalPositionalEncoding(nn.Module):<br>    &quot;&quot;&quot;Fixed (non-learned) sinusoidal position signal from the 2017 paper.<br><br>    Dimension 2i is a sine of wavelength 2*pi*base^(2i/d_model), dimension<br>    2i+1 the matching cosine. Low dimensions oscillate fast, high dimensions<br>    barely move across the whole sequence, so sampling all of them at a<br>    position gives that position a unique fingerprint.<br>    &quot;&quot;&quot;<br><br>    def __init__(self, d_model: int, max_len: int = 5000, base: float = 10000.0):<br>        super().__init__()<br>        if d_model % 2 != 0:<br>            raise ValueError(f&quot;d_model must be even, got {d_model}&quot;)<br><br>        position = torch.arange(max_len).unsqueeze(1).float()      # (max_len, 1)<br>        # Computed in log space: base ** (2i / d_model) overflows float32 for<br>        # large d_model, exp(-2i * log(base) / d_model) does not.<br>        div_term = torch.exp(<br>            torch.arange(0, d_model, 2).float() * (-math.log(base) / d_model)<br>        )                                                          # (d_model/2,)<br><br>        pe = torch.zeros(max_len, d_model)<br>        pe[:, 0::2] = torch.sin(position * div_term)<br>        pe[:, 1::2] = torch.cos(position * div_term)<br>        # A buffer, not a parameter: saved with the model, never given a gradient.<br>        self.register_buffer(&quot;pe&quot;, pe.unsqueeze(0))                # (1, max_len, d)<br><br>    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:<br>        return x + self.pe[:, : x.size(1)]</pre><p>Two things worth knowing that the code alone does not say:</p><ul><li>Why sinusoids and not learned vectors? Both work, and the paper reports near-identical scores. Sinusoids extrapolate to sequences longer than any seen in training, and the offset between two positions is a linear function of their encodings a property a model can exploit for relative reasoning.</li><li>This is not what most models use today. GPT-style models used learned absolute embeddings; Llama, Mistral and friends use RoPE, which rotates <strong>Q</strong> and <strong>K</strong> by a position-dependent angle inside each attention head rather than adding anything to the residual stream. The sinusoidal version is still the clearest way to see why position has to be injected at all.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/1*EHKmK3YlmbFeIGji2DxSRA.gif" /></figure><h3>3. Attention, before the matrices</h3><p>Strip away the linear algebra and one attention layer does this: every token asks every other token how much it should listen, then takes a weighted average of what they offer.</p><p>Two steps. First, score every pair. Second, turn each row of scores into a probability distribution with a softmax:</p><pre>scores = torch.tensor([0.5, 2.6, 1.6, 0.9])     # &quot;sat&quot; scored against 4 tokens<br>weights = torch.softmax(scores, dim=-1)<br>print(weights, weights.sum())<br># tensor([0.0732, 0.5977, 0.2199, 0.1092]) tensor(1.)</pre><p>The row sums to exactly 1, which is what makes the output a genuine weighted average rather than an arbitrary rescaling.</p><p>Why softmax and not just take the maximum? Because argmax has zero gradient almost everywhere. Softmax is a differentiable soft-maximum, so the model can learn how much to attend. A hard pick would give it nothing to descend.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/1*QjDypL1LA2gFry1H06khig.gif" /></figure><h3>4. Queries, keys and values</h3><p>Now the actual mechanism. Each token is projected three ways through three learned matrices, giving it three different jobs:</p><ul><li>Query (Q): What am I looking for?</li><li>Key (K): What do I advertise to others?</li><li>Value (V): What do I actually pass on if selected?</li></ul><p>The query of one token is compared against the keys of all tokens; the resulting weights are applied to the values. The query and key drive routing; the value carries content. Keeping them separate is what lets a token advertise something different from what it consumes.</p><pre>Attention(Q, K, V) = softmax( Q Kᵀ / √d_k ) V</pre><pre>def scaled_dot_product_attention(<br>    q: torch.Tensor,            # (..., T_q, d_head)<br>    k: torch.Tensor,            # (..., T_k, d_head)<br>    v: torch.Tensor,            # (..., T_k, d_head)<br>    mask: torch.Tensor | None = None,   # broadcastable to (..., T_q, T_k), True = keep<br>    dropout: nn.Dropout | None = None,<br>):<br>    &quot;&quot;&quot;The whole formula, in five lines.&quot;&quot;&quot;<br>    d_k = q.size(-1)<br><br>    # 1. score every query against every key<br>    scores = q @ k.transpose(-2, -1) / math.sqrt(d_k)    # (..., T_q, T_k)<br><br>    # 2. blank out anything this query is not allowed to see<br>    if mask is not None:<br>        scores = scores.masked_fill(~mask, float(&quot;-inf&quot;))<br><br>    # 3. normalise each row into a distribution<br>    weights = torch.softmax(scores, dim=-1)<br>    if dropout is not None:<br>        weights = dropout(weights)<br><br>    # 4. weighted sum of the values<br>    return weights @ v, weights                          # (..., T_q, d_head)</pre><h3>Why divide by d_k^0.5</h3><p>This is the one line people skip, and it matters.</p><p>q · k is a sum of d_k products. If the components are independent with unit variance, the dot product has variance d_k — so it grows with model width. Push large values through a softmax, and it saturates: one weight goes to ~1, the rest to ~0, and the gradient through the softmax collapses toward zero. Dividing by √d_k restores unit variance and keeps the distribution soft enough to learn through.</p><p>You can watch it happen. Averaged over 500 random draws, here is the largest weight in an 8-key attention row as d_k grows:</p><pre>torch.manual_seed(0)<br>print(f&quot;{&#39;d_k&#39;:&gt;5}{&#39;unscaled&#39;:&gt;12}{&#39;scaled&#39;:&gt;10}&quot;)<br>for d_k in (16, 64, 256, 1024):<br>    unscaled = scaled = 0.0<br>    trials = 500<br>    for _ in range(trials):<br>        q, k = torch.randn(1, d_k), torch.randn(8, d_k)<br>        unscaled += float(torch.softmax(q @ k.T, dim=-1).max())<br>        scaled += float(torch.softmax(q @ k.T / math.sqrt(d_k), dim=-1).max())<br>    print(f&quot;{d_k:&gt;5}{unscaled / trials:&gt;12.3f}{scaled / trials:&gt;10.3f}&quot;)</pre><pre>d_k    unscaled    scaled<br>   16       0.760     0.356<br>   64       0.874     0.355<br>  256       0.934     0.363<br> 1024       0.969     0.358</pre><p>Unscaled, the row collapses toward one-hot as the model gets wider: 0.969 of all attention on a single key at d_k = 1024, with almost no gradient left. Scaled, it sits at ~0.355 regardless of width. That flat column is the entire reason for the √d_k.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/1*XFm8NrMaOZhop89Fy9UvJw.gif" /></figure><h3>5. Multi-head attention</h3><p>One attention map expresses one kind of relationship. Real sentences have several going at once: syntactic agreement, coreference, local adjacency.</p><p>So run several in parallel. Split d_model into h heads of width d_model / h, attend independently in each, concatenate, and mix the results with one more linear layer.</p><p>Crucially, the heads are narrower, not additional: 8 heads of width 64 cost about the same as 1 head of width 512. You get several relationship types for roughly the price of one.</p><pre>class MultiHeadAttention(nn.Module):<br>    def __init__(self, d_model: int, n_heads: int, dropout: float = 0.0):<br>        super().__init__()<br>        if d_model % n_heads != 0:<br>            raise ValueError(f&quot;d_model {d_model} not divisible by n_heads {n_heads}&quot;)<br>        self.n_heads = n_heads<br>        self.d_head = d_model // n_heads<br><br>        # One fused matrix per role, sliced into heads afterwards — cheaper<br>        # than n_heads separate small matmuls.<br>        self.w_q = nn.Linear(d_model, d_model)<br>        self.w_k = nn.Linear(d_model, d_model)<br>        self.w_v = nn.Linear(d_model, d_model)<br>        self.w_o = nn.Linear(d_model, d_model)<br>        self.dropout = nn.Dropout(dropout)<br><br>    def _split_heads(self, x: torch.Tensor) -&gt; torch.Tensor:<br>        b, t, _ = x.shape<br>        # (B, T, d_model) -&gt; (B, h, T, d_head)<br>        return x.view(b, t, self.n_heads, self.d_head).transpose(1, 2)<br><br>    def forward(self, query, key, value, mask=None):<br>        b, t_q, _ = query.shape<br><br>        q = self._split_heads(self.w_q(query))<br>        k = self._split_heads(self.w_k(key))<br>        v = self._split_heads(self.w_v(value))<br><br>        if mask is not None:<br>            # (B, T_q, T_k) -&gt; (B, 1, T_q, T_k) so it broadcasts over heads<br>            mask = mask.unsqueeze(1) if mask.dim() == 3 else mask<br><br>        out, weights = scaled_dot_product_attention(q, k, v, mask, self.dropout)<br><br>        # (B, h, T_q, d_head) -&gt; (B, T_q, d_model)<br>        out = out.transpose(1, 2).contiguous().view(b, t_q, -1)<br>        return self.w_o(out), weights</pre><p>Note that forward takes query, key and value separately. When all three are the same tensor, this is self-attention; when the query comes from the decoder and the key/value from the encoder, it is cross-attention. Same module, both jobs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/1*qaSfaojPP_1LOqTU-2xIEg.gif" /></figure><h3>6. The encoder block</h3><p>Attention mixes information across tokens. A position-wise feed-forward network then processes each token independently, with the same weights at every position, no interaction. Alternating the two is the whole architecture: gather context, then think about it.</p><pre>class FeedForward(nn.Module):<br>    &quot;&quot;&quot;Applied to each position separately and identically.&quot;&quot;&quot;<br><br>    def __init__(self, d_model: int, d_ff: int, dropout: float = 0.0):<br>        super().__init__()<br>        self.net = nn.Sequential(<br>            nn.Linear(d_model, d_ff),<br>            nn.ReLU(),<br>            nn.Dropout(dropout),<br>            nn.Linear(d_ff, d_model),<br>        )<br><br>    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:<br>        return self.net(x)</pre><p>Each sub-layer is wrapped in a residual connection followed by layer normalization:</p><pre>x = LayerNorm(x + Sublayer(x))</pre><p>The residual is not a detail. With a dozen-plus blocks stacked, the skip path gives gradients a route back to the early layers that doesn&#39;t pass through every intervening transformation; without it, deep stacks barely train. It also means each block edits a running representation rather than replacing it, which is where the “residual stream” framing comes from.</p><pre>class EncoderBlock(nn.Module):<br>    def __init__(self, d_model: int, n_heads: int, d_ff: int, dropout: float = 0.1):<br>        super().__init__()<br>        self.attn = MultiHeadAttention(d_model, n_heads, dropout)<br>        self.ff = FeedForward(d_model, d_ff, dropout)<br>        self.norm1 = nn.LayerNorm(d_model)<br>        self.norm2 = nn.LayerNorm(d_model)<br>        self.drop = nn.Dropout(dropout)<br><br>    def forward(self, x: torch.Tensor, mask: torch.Tensor | None = None):<br>        attn_out, weights = self.attn(x, x, x, mask)      # self-attention<br>        x = self.norm1(x + self.drop(attn_out))           # add &amp; norm<br>        x = self.norm2(x + self.drop(self.ff(x)))         # add &amp; norm<br>        return x, weights</pre><p>Post-norm vs pre-norm. The code above is post-norm, exactly as published. Nearly every model since does pre-norm x = x + Sublayer(LayerNorm(x)) because post-norm needs a learning-rate warmup to converge at depth, while pre-norm trains stably without one. One line moved, large practical difference.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/1*aJAArTWOOdlaTNCKAQvQqQ.gif" /></figure><h3>7. The decoder: masking and cross-attention</h3><p>The decoder generates one token at a time, feeding what it wrote back in. That creates an obligation: while predicting positiont, it must not be allowed to look at positions &gt; t. Otherwise training leaks the answer, and the model learns nothing useful for inference, where the future genuinely does not exist.</p><p>The fix is a causal mask: set the upper triangle of the score matrix to -inf before the softmax, so those positions receive exactly zero weight and the remaining ones renormalize to sum to 1.</p><pre>def causal_mask(t: int, device=None) -&gt; torch.Tensor:<br>    &quot;&quot;&quot;Lower-triangular (T, T) bool mask. True = may attend.&quot;&quot;&quot;<br>    return torch.tril(torch.ones(t, t, dtype=torch.bool, device=device))<br><br><br>def padding_mask(ids: torch.Tensor, pad_id: int = 0) -&gt; torch.Tensor:<br>    &quot;&quot;&quot;(B, T) ids -&gt; (B, 1, T) mask that hides padding from every query.&quot;&quot;&quot;<br>    return (ids != pad_id).unsqueeze(1)<br></pre><p>Masking before the softmax, rather than zeroing weights after, keeps the rows normalized. Zeroing afterwards would leave each row summing to less than 1.</p><p>The decoder block has three sub-layers rather than two. The middle one is cross-attention: queries from the decoder, keys and values from the encoder output. This is the only channel through which source information reaches the generated sequence.</p><pre>class DecoderBlock(nn.Module):<br>    def __init__(self, d_model: int, n_heads: int, d_ff: int, dropout: float = 0.1):<br>        super().__init__()<br>        self.self_attn = MultiHeadAttention(d_model, n_heads, dropout)<br>        self.cross_attn = MultiHeadAttention(d_model, n_heads, dropout)<br>        self.ff = FeedForward(d_model, d_ff, dropout)<br>        self.norm1 = nn.LayerNorm(d_model)<br>        self.norm2 = nn.LayerNorm(d_model)<br>        self.norm3 = nn.LayerNorm(d_model)<br>        self.drop = nn.Dropout(dropout)<br><br>    def forward(self, x, memory, self_mask=None, memory_mask=None):<br>        # 1. masked self-attention over what has been generated so far<br>        h, _ = self.self_attn(x, x, x, self_mask)<br>        x = self.norm1(x + self.drop(h))<br><br>        # 2. cross-attention: Q from the decoder, K/V from the encoder<br>        h, cross_weights = self.cross_attn(x, memory, memory, memory_mask)<br>        x = self.norm2(x + self.drop(h))<br><br>        # 3. position-wise feed-forward<br>        x = self.norm3(x + self.drop(self.ff(x)))<br>        return x, cross_weights</pre><p>Finally, the decoder output is projected to vocabulary size and softmaxed into a distribution over next tokens. Taking the top one every time is greedy decoding; sampling from the distribution instead is what makes outputs vary, and temperature scales how sharp that distribution is.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/1*U37tG3eQhVwlPZvOtIhGSQ.gif" /></figure><h3>8. Assembling the model</h3><pre>class Transformer(nn.Module):<br>    def __init__(<br>        self,<br>        src_vocab: int,<br>        tgt_vocab: int,<br>        d_model: int = 512,<br>        n_heads: int = 8,<br>        n_layers: int = 6,<br>        d_ff: int = 2048,<br>        dropout: float = 0.1,<br>        max_len: int = 5000,<br>    ):<br>        super().__init__()<br>        self.src_embed = TokenEmbedding(src_vocab, d_model)<br>        self.tgt_embed = TokenEmbedding(tgt_vocab, d_model)<br>        self.pos = SinusoidalPositionalEncoding(d_model, max_len)<br>        self.drop = nn.Dropout(dropout)<br><br>        self.encoder = nn.ModuleList(<br>            EncoderBlock(d_model, n_heads, d_ff, dropout) for _ in range(n_layers)<br>        )<br>        self.decoder = nn.ModuleList(<br>            DecoderBlock(d_model, n_heads, d_ff, dropout) for _ in range(n_layers)<br>        )<br>        self.out = nn.Linear(d_model, tgt_vocab)<br><br>    def encode(self, src_ids, src_mask=None):<br>        x = self.drop(self.pos(self.src_embed(src_ids)))<br>        for block in self.encoder:<br>            x, _ = block(x, src_mask)<br>        return x<br><br>    def decode(self, tgt_ids, memory, tgt_mask=None, memory_mask=None):<br>        x = self.drop(self.pos(self.tgt_embed(tgt_ids)))<br>        for block in self.decoder:<br>            x, _ = block(x, memory, tgt_mask, memory_mask)<br>        return x<br><br>    def forward(self, src_ids, tgt_ids, src_mask=None, memory_mask=None):<br>        memory = self.encode(src_ids, src_mask)<br>        t = tgt_ids.size(1)<br>        tgt_mask = causal_mask(t, tgt_ids.device)<br>        x = self.decode(tgt_ids, memory, tgt_mask, memory_mask)<br>        return self.out(x)          # (B, T, tgt_vocab) — logits, not probabilities</pre><p>Returning logits rather than probabilities is deliberate: nn.CrossEntropyLoss expects raw logits and folds the log-softmax in itself, which is numerically stabler than softmaxing then taking a log.</p><p>A shape check, plus a real test that the mask does its job:</p><pre>model = Transformer(src_vocab=1000, tgt_vocab=1000, d_model=64,<br>                    n_heads=4, n_layers=2, d_ff=256)<br>model.eval()<br><br>src = torch.randint(1, 1000, (2, 7))<br>tgt = torch.randint(1, 1000, (2, 5))<br>logits = model(src, tgt)<br>print(logits.shape)            # torch.Size([2, 5, 1000])<br><br># The causal mask must make position t independent of anything after it.<br>with torch.no_grad():<br>    base = model(src, tgt)<br>    poked = tgt.clone()<br>    poked[:, -1] = (poked[:, -1] + 1) % 1000      # change ONLY the last token<br>    after = model(src, poked)<br><br># every position except the last is untouched<br>print(torch.allclose(base[:, :-1], after[:, :-1], atol=1e-6))   # True</pre><p>If that printed False, the mask is leaking — the single most common way a from-scratch implementation is silently wrong, because it still trains and the loss still falls. It just cheats.</p><h3>9. Generation</h3><p>Inference re-runs the decoder once per token. The encoder runs once, and its output is reused, which is why encoder–decoder models are cheap on the source side and expensive on the target side.</p><pre>@torch.no_grad()<br>def greedy_decode(model, src_ids, start_id, end_id, max_new_tokens=20):<br>    model.eval()<br>    memory = model.encode(src_ids)<br>    tgt = torch.full((src_ids.size(0), 1), start_id, dtype=torch.long,<br>                     device=src_ids.device)<br><br>    for _ in range(max_new_tokens):<br>        mask = causal_mask(tgt.size(1), tgt.device)<br>        hidden = model.decode(tgt, memory, mask)<br>        next_id = model.out(hidden[:, -1]).argmax(-1, keepdim=True)<br>        tgt = torch.cat([tgt, next_id], dim=1)<br>        if (next_id == end_id).all():<br>            break<br>    return tgt</pre><p>Recomputing the full decoder stack for every new token is quadratic and wasteful — production implementations cache the keys and values of previous positions (the KV cache), turning each step into a single-token forward pass. That is an optimization, not a change to the mathematics, so it is left out here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/1*QBjo-hP1yFNM2FfUCWX1SA.gif" /><figcaption><em>A horizontal chain of labelled boxes: </em>text → tokens → embeddings → + position → self-attention → encoder × N → decoder → softmax → next word<em>, each with an arrow to the next.</em></figcaption></figure><h3>What actually matters</h3><p>Three properties explain why this architecture displaced what came before:</p><ol><li>Every token sees every other token in one step. An RNN needs O(T) sequential steps to connect two distant positions and degrades over that distance. Attention needs one, no matter the distance.</li><li>It parallelizes. Training has no sequential dependency across positions; the whole sequence is one matmul. That property turned model scale into an engineering budget rather than a wall-clock ceiling.</li><li>The routing is learned. Nothing hand-specifies that it should look at animal. The weights that produce that behaviour emerge from gradient descent on next-token prediction.</li></ol><p>The cost: attention is O(T²) in sequence length, in both time and memory. Every long-context technique since sparse patterns, sliding windows, linear attention, FlashAttention&#39;s tiling, and state-space hybrids is an attack on that quadratic term.</p><h3>The complete file</h3><p>Everything above, in dependency order, runnable as one script:</p><pre>import math<br><br>import torch<br>import torch.nn as nn<br><br>torch.manual_seed(0)<br><br><br>class TokenEmbedding(nn.Module):<br>    def __init__(self, vocab_size: int, d_model: int):<br>        super().__init__()<br>        self.embed = nn.Embedding(vocab_size, d_model)<br>        self.d_model = d_model<br><br>    def forward(self, ids):<br>        return self.embed(ids) * math.sqrt(self.d_model)<br><br><br>class SinusoidalPositionalEncoding(nn.Module):<br>    def __init__(self, d_model: int, max_len: int = 5000, base: float = 10000.0):<br>        super().__init__()<br>        if d_model % 2 != 0:<br>            raise ValueError(f&quot;d_model must be even, got {d_model}&quot;)<br>        position = torch.arange(max_len).unsqueeze(1).float()<br>        div_term = torch.exp(<br>            torch.arange(0, d_model, 2).float() * (-math.log(base) / d_model)<br>        )<br>        pe = torch.zeros(max_len, d_model)<br>        pe[:, 0::2] = torch.sin(position * div_term)<br>        pe[:, 1::2] = torch.cos(position * div_term)<br>        self.register_buffer(&quot;pe&quot;, pe.unsqueeze(0))<br><br>    def forward(self, x):<br>        return x + self.pe[:, : x.size(1)]<br><br><br>def scaled_dot_product_attention(q, k, v, mask=None, dropout=None):<br>    d_k = q.size(-1)<br>    scores = q @ k.transpose(-2, -1) / math.sqrt(d_k)<br>    if mask is not None:<br>        scores = scores.masked_fill(~mask, float(&quot;-inf&quot;))<br>    weights = torch.softmax(scores, dim=-1)<br>    if dropout is not None:<br>        weights = dropout(weights)<br>    return weights @ v, weights<br><br><br>class MultiHeadAttention(nn.Module):<br>    def __init__(self, d_model: int, n_heads: int, dropout: float = 0.0):<br>        super().__init__()<br>        if d_model % n_heads != 0:<br>            raise ValueError(f&quot;d_model {d_model} not divisible by n_heads {n_heads}&quot;)<br>        self.n_heads = n_heads<br>        self.d_head = d_model // n_heads<br>        self.w_q = nn.Linear(d_model, d_model)<br>        self.w_k = nn.Linear(d_model, d_model)<br>        self.w_v = nn.Linear(d_model, d_model)<br>        self.w_o = nn.Linear(d_model, d_model)<br>        self.dropout = nn.Dropout(dropout)<br><br>    def _split_heads(self, x):<br>        b, t, _ = x.shape<br>        return x.view(b, t, self.n_heads, self.d_head).transpose(1, 2)<br><br>    def forward(self, query, key, value, mask=None):<br>        b, t_q, _ = query.shape<br>        q = self._split_heads(self.w_q(query))<br>        k = self._split_heads(self.w_k(key))<br>        v = self._split_heads(self.w_v(value))<br>        if mask is not None:<br>            mask = mask.unsqueeze(1) if mask.dim() == 3 else mask<br>        out, weights = scaled_dot_product_attention(q, k, v, mask, self.dropout)<br>        out = out.transpose(1, 2).contiguous().view(b, t_q, -1)<br>        return self.w_o(out), weights<br><br><br>class FeedForward(nn.Module):<br>    def __init__(self, d_model: int, d_ff: int, dropout: float = 0.0):<br>        super().__init__()<br>        self.net = nn.Sequential(<br>            nn.Linear(d_model, d_ff),<br>            nn.ReLU(),<br>            nn.Dropout(dropout),<br>            nn.Linear(d_ff, d_model),<br>        )<br><br>    def forward(self, x):<br>        return self.net(x)<br><br><br>class EncoderBlock(nn.Module):<br>    def __init__(self, d_model, n_heads, d_ff, dropout=0.1):<br>        super().__init__()<br>        self.attn = MultiHeadAttention(d_model, n_heads, dropout)<br>        self.ff = FeedForward(d_model, d_ff, dropout)<br>        self.norm1 = nn.LayerNorm(d_model)<br>        self.norm2 = nn.LayerNorm(d_model)<br>        self.drop = nn.Dropout(dropout)<br><br>    def forward(self, x, mask=None):<br>        attn_out, weights = self.attn(x, x, x, mask)<br>        x = self.norm1(x + self.drop(attn_out))<br>        x = self.norm2(x + self.drop(self.ff(x)))<br>        return x, weights<br><br><br>class DecoderBlock(nn.Module):<br>    def __init__(self, d_model, n_heads, d_ff, dropout=0.1):<br>        super().__init__()<br>        self.self_attn = MultiHeadAttention(d_model, n_heads, dropout)<br>        self.cross_attn = MultiHeadAttention(d_model, n_heads, dropout)<br>        self.ff = FeedForward(d_model, d_ff, dropout)<br>        self.norm1 = nn.LayerNorm(d_model)<br>        self.norm2 = nn.LayerNorm(d_model)<br>        self.norm3 = nn.LayerNorm(d_model)<br>        self.drop = nn.Dropout(dropout)<br><br>    def forward(self, x, memory, self_mask=None, memory_mask=None):<br>        h, _ = self.self_attn(x, x, x, self_mask)<br>        x = self.norm1(x + self.drop(h))<br>        h, cross_weights = self.cross_attn(x, memory, memory, memory_mask)<br>        x = self.norm2(x + self.drop(h))<br>        x = self.norm3(x + self.drop(self.ff(x)))<br>        return x, cross_weights<br><br><br>def causal_mask(t: int, device=None):<br>    return torch.tril(torch.ones(t, t, dtype=torch.bool, device=device))<br><br><br>def padding_mask(ids, pad_id: int = 0):<br>    return (ids != pad_id).unsqueeze(1)<br><br><br>class Transformer(nn.Module):<br>    def __init__(self, src_vocab, tgt_vocab, d_model=512, n_heads=8,<br>                 n_layers=6, d_ff=2048, dropout=0.1, max_len=5000):<br>        super().__init__()<br>        self.src_embed = TokenEmbedding(src_vocab, d_model)<br>        self.tgt_embed = TokenEmbedding(tgt_vocab, d_model)<br>        self.pos = SinusoidalPositionalEncoding(d_model, max_len)<br>        self.drop = nn.Dropout(dropout)<br>        self.encoder = nn.ModuleList(<br>            EncoderBlock(d_model, n_heads, d_ff, dropout) for _ in range(n_layers)<br>        )<br>        self.decoder = nn.ModuleList(<br>            DecoderBlock(d_model, n_heads, d_ff, dropout) for _ in range(n_layers)<br>        )<br>        self.out = nn.Linear(d_model, tgt_vocab)<br><br>    def encode(self, src_ids, src_mask=None):<br>        x = self.drop(self.pos(self.src_embed(src_ids)))<br>        for block in self.encoder:<br>            x, _ = block(x, src_mask)<br>        return x<br><br>    def decode(self, tgt_ids, memory, tgt_mask=None, memory_mask=None):<br>        x = self.drop(self.pos(self.tgt_embed(tgt_ids)))<br>        for block in self.decoder:<br>            x, _ = block(x, memory, tgt_mask, memory_mask)<br>        return x<br><br>    def forward(self, src_ids, tgt_ids, src_mask=None, memory_mask=None):<br>        memory = self.encode(src_ids, src_mask)<br>        tgt_mask = causal_mask(tgt_ids.size(1), tgt_ids.device)<br>        x = self.decode(tgt_ids, memory, tgt_mask, memory_mask)<br>        return self.out(x)<br><br><br>@torch.no_grad()<br>def greedy_decode(model, src_ids, start_id, end_id, max_new_tokens=20):<br>    model.eval()<br>    memory = model.encode(src_ids)<br>    tgt = torch.full((src_ids.size(0), 1), start_id, dtype=torch.long,<br>                     device=src_ids.device)<br>    for _ in range(max_new_tokens):<br>        mask = causal_mask(tgt.size(1), tgt.device)<br>        hidden = model.decode(tgt, memory, mask)<br>        next_id = model.out(hidden[:, -1]).argmax(-1, keepdim=True)<br>        tgt = torch.cat([tgt, next_id], dim=1)<br>        if (next_id == end_id).all():<br>            break<br>    return tgt<br><br><br>if __name__ == &quot;__main__&quot;:<br>    model = Transformer(src_vocab=1000, tgt_vocab=1000, d_model=64,<br>                        n_heads=4, n_layers=2, d_ff=256)<br>    model.eval()<br><br>    src = torch.randint(1, 1000, (2, 7))<br>    tgt = torch.randint(1, 1000, (2, 5))<br>    print(&quot;logits:&quot;, model(src, tgt).shape)<br><br>    with torch.no_grad():<br>        base = model(src, tgt)<br>        poked = tgt.clone()<br>        poked[:, -1] = (poked[:, -1] + 1) % 1000<br>        after = model(src, poked)<br>    print(&quot;causal mask holds:&quot;, torch.allclose(base[:, :-1], after[:, :-1], atol=1e-6))<br><br>    print(&quot;generated:&quot;, greedy_decode(model, src, start_id=1, end_id=2,<br>                                      max_new_tokens=5).shape)</pre><h3>Further reading</h3><ul><li><a href="https://arxiv.org/abs/1706.03762">Attention Is All You Need</a> — the original paper; still short and readable.</li><li><a href="https://nlp.seas.harvard.edu/annotated-transformer/">The Annotated Transformer</a> — Harvard NLP’s line-by-line reimplementation.</li><li><a href="https://transformer-circuits.pub/2021/framework/index.html">A Mathematical Framework for Transformer Circuits</a> — where the “residual stream” reading of the architecture comes from.</li><li><a href="https://arxiv.org/abs/2104.09864">RoFormer</a> — rotary position embeddings, what most current models actually use instead of section 2.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a1e96e3e801c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[19. When to use SAM, and when not to]]></title>
            <link>https://medium.com/@aminfadaeinejad.edu/when-to-use-sam-and-when-not-to-14374cb9c401?source=rss-aff4faf055a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/14374cb9c401</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[segmentation]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Amin Fadaeinejad]]></dc:creator>
            <pubDate>Sat, 08 Aug 2026 23:21:24 GMT</pubDate>
            <atom:updated>2026-08-08T23:21:38.870Z</atom:updated>
            <content:encoded><![CDATA[<p>Sooner or later, every team that touches segmentation problems asks the same question. “We need segmentation. Can we just use SAM?”</p><p>The demo page makes it look like the answer is always yes. You click on an object, you get a clean mask, and the whole problem seems solved. In production, the answer is more often “yes, but not where you think,” and sometimes it is a clear no.</p><p>I have used the Segment Anything family in real pipelines, including a video object removal system built on SAM 2 masklets. This post is the decision guide I wish someone had given me before I started, so I would appreciate it if you could support my page to grow.</p><h3>What SAM actually gives you</h3><p>The original SAM (Meta, 2023) is a promptable segmentation model. You pass a point, a box, or a coarse mask, and the mask decoder returns a binary mask. It was trained on SA-1B, roughly 1.1B masks over 11M images.</p><p>The important property is this: SAM is class-agnostic. It gives you the mask, not the label. Almost every good and bad decision people make with SAM comes back to that one sentence.</p><p>SAM 2 (2024) extended the same idea to video. It added a streaming memory bank, so a mask given on one frame propagates through the sequence as a masklet. The image encoder was replaced by Hiera, and the checkpoints range from tiny (about 39M parameters) to large (about 224M).</p><p>SAM 3, released on 19 November 2025, changed the interface more than the internals. It introduced Promptable Concept Segmentation: you give a short noun phrase like “yellow school bus”, or an image exemplar, and the model returns masks and instance IDs for every match in the image or video. Architecturally, it is a DETR-based detector plus a SAM 2-style tracker sharing one vision encoder, around 848M parameters in total. Meta reports about 30 ms per image with more than 100 detected objects on an H200, and it fits on a 16 GB GPU. Meta also shipped SAM 3D alongside it for single-image 3D reconstruction. That is a different problem, but the 2D and 3D sides are more connected than they look, and I come back to it near the end.</p><p>So with SAM 3, the “no labels” limitation is partly gone. Partly is the operative word. Open-vocabulary is not the same thing as your taxonomy, and I will come back to that.</p><h3>The question that decides most cases</h3><p>Before comparing benchmarks, answer this:</p><blockquote>Do you know your classes in advance, and will that list stay stable?</blockquote><p>If the answer is yes, the list is short, and you need high throughput, you are usually better off training a specialist. A <strong>YOLO-seg variant</strong>, a <strong>Mask2Former head</strong>, or a plain U-Net will be smaller, faster, cheaper per frame, and easier to debug. You will also own the failure modes instead of inheriting someone else’s.</p><p>If the answer is no, because the label set is open, changes per customer, or is defined by the user at runtime, then a foundation model is doing something a specialist cannot do at all.</p><p>There is a second question that people skip:</p><blockquote>Where does the prompt come from?</blockquote><p>SAM 1 and SAM 2 do not find objects. They segment what you point at. If there is no human in the loop and no detector producing boxes, you do not have a system yet; you have half of one. SAM 3 removes this constraint when a noun phrase is enough to describe your target, and Grounded SAM-style pipelines (a detector such as Grounding DINO feeding boxes into SAM) remove it when the phrase is not enough.</p><h3>Where SAM is clearly the right choice</h3><p>Annotation is the strongest case, and in my experience the highest return on investment. A box-prompted mask is often good enough to skip manual polygon drawing entirely, and a human reviewer only fixes the bad ones. You then train a small task-specific model on the result. Most teams that use SAM in production are actually doing this: SAM offline, small model online.</p><p>Interactive tools are the second case. Anything with a human in the loop suits SAM’s design, because a person can correct a wrong mask with one more click. Photo editors, VFX rotoscoping, e-commerce cutouts, and annotation interfaces for domain experts all fall here.</p><p>Video object tracking and removal is the third. SAM 2 propagating a masklet across a shot is genuinely convenient, and it saved me a large amount of work in a removal pipeline where the mask feeds an inpainting model. The segmentation was never the hard part after that. The inpainting was.</p><p>Long-tail and open vocabulary is the fourth. If your users can ask for “The blue Book on the second shelf”, no fixed taxonomy will cover them. SAM 3 concept prompts or a detector plus SAM is the only reasonable route.</p><p>Finally, prototyping. If you have no labelled data and you need to know whether the product idea works at all, zero-shot masks let you build the demo this week instead of after a three-month annotation contract. Just be honest inside the team that this is a prototype, not the architecture.</p><h3>Where SAM is the wrong tool</h3><p>A fixed, small taxonomy at scale. If you segment three classes on ten million frames a month, a foundation model is a very expensive way to solve a solved problem. Distill it or train a specialist.</p><p>Real-time inference on edge or CPU. SAM 3 is an 848M parameter GPU model. Even SAM 2 tiny expects a decent accelerator. If your constraint is a Jetson, a phone, or a CPU-only industrial box at 30 FPS, this family is <strong>not</strong> the answer.</p><p>Domain shift. This is the failure people underestimate the most. SAM was trained on natural images, and performance drops on medical modalities, microscopy, thermal, SAR and satellite imagery, and industrial X-ray. The medical literature is consistent about it: zero-shot SAM struggles with weak boundaries, low contrast, and small structures, and it usually sits below a fully trained task-specific model. The situation is messier than “just use MedSAM” too, since MedSAM has been reported to underperform plain SAM and SAM 2 on some ultrasound datasets despite being fine-tuned on medical data. If your images do not look like photographs, budget for fine-tuning and for evaluation, and do not promise anything based on the public demo.</p><p>Semantic or panoptic output under your own class definitions. SAM 3 gives you open-vocabulary instances, which is not the same as a consistent, auditable label taxonomy with your business rules attached. If your downstream system needs “damage type B” and not “scratch-like thing”, you need a classifier that was trained on your definition of B.</p><p>Measurement-grade boundaries. SAM has an inherent granularity ambiguity: given a point on a person, the correct answer might be the person, the jacket, or the logo on the jacket. For interactive use, this is fine, since the user disambiguates. For automated metrology, where a two-pixel boundary shift changes the reported area, it is a real risk. Validate against your own tolerance, not against mIoU on COCO.</p><p>Licensing. SAM 3 is released under Meta’s custom SAM License, not MIT or Apache 2.0. It is free and allows commercial use, but with conditions attached, including an acceptable use policy. Read the terms before your product ships, not after. This is a boring point that has killed real projects.</p><h3>Do the cost calculation before the meeting</h3><p>Very few teams actually compute this, and it takes five minutes.</p><p>Take 1M frames per day at 30 ms per frame. That is 30,000 seconds, so about 8.3 GPU-hours per day in the best case, before you account for batching inefficiency, decoding, and I/O. Now price that against a 20M parameter specialist running at 3 ms on a much cheaper card. The per-image cost of SAM 3 is not high in absolute terms. Amortized over a year of continuous inference, the gap between a foundation model and a distilled specialist is usually one order of magnitude, sometimes two.</p><p>That number is what your decision should be based on, not on which model has better qualitative results in a blog post.</p><h3>Hardware realities that change the arithmetic</h3><p><strong>[Important Note]: The single most useful thing to know about SAM’s architecture is that the cost is almost entirely in the image encoder.</strong></p><p>The mask decoder is around 4 ms. So if you are running several prompts on the same image, encode once, cache the embedding, and run the decoder as many times as you like. Interactive tools that re-encode on every click are throwing away an order of magnitude for nothing. The same idea applies to batch labelling: the encoder is what you batch, the decoder is almost free.</p><pre>import torch<br>from sam2.sam2_image_predictor import SAM2ImagePredictor<br><br>predictor = SAM2ImagePredictor.from_pretrained(&quot;facebook/sam2.1-hiera-large&quot;)<br>with torch.inference_mode(), torch.autocast(&quot;cuda&quot;, dtype=torch.bfloat16):<br>    predictor.set_image(image)            # encoder runs once, this is the expensive part<br>    for box in candidate_boxes:           # decoder only, a few ms each<br>        masks, scores, _ = predictor.predict(box=box, multimask_output=False)</pre><p>Two lines, and on an image with twenty candidate boxes it is the difference between twenty encoder passes and one.</p><p>Beyond that, the usual PyTorch levers apply, and they are worth taking before you conclude the model is too expensive. Run the encoder under bfloat16 autocast. Compile it (SAM 2 ships a compile_image_encoder option, and the video predictor has optimized configs). Make sure attention is going through the fused SDPA or FlashAttention kernel rather than a naive implementation, since a ViT encoder at 1024 by 1024 is attention-bound. For a fixed deployment shape, export the encoder to TensorRT or ONNX, which is exactly what NanoSAM does on top of MobileSAM to reach real-time on a Jetson. Custom Triton kernels are rarely where you should start here, because the standard fused kernels already cover the hot paths; that effort pays off more in the memory attention of SAM 2, which is less well served by stock kernels.</p><h3>The VRAM trap in SAM 2 video</h3><p>The streaming memory bank sounds cheap because it is bounded: SAM 2 attends to a small number of recent frame memories plus the prompted frames and object pointers, so the memory attention cost does not grow without limit as the clip gets longer. That part is fine.</p><p>What does grow is the inference state. The video predictor caches encoded features for the frames it has processed, and it holds the video tensor itself. At 1024 by 1024 in half precision, a thousand frames is already several GB before any model weights are loaded. Track ten objects instead of one and the per-frame memory features scale with it, because objects are batched along their own dimension. Teams usually discover this when a demo that worked on a 200-frame clip runs out of memory on a real shot.</p><p>Three things keep it under control. Use the offloading flags in init_state, which move the video and the inference state to CPU and trade PCIe traffic for VRAM. Process long sequences in overlapping chunks and re-prompt at each boundary, using the last good mask as the new prompt, which also limits drift. And cap the memory bank length explicitly rather than trusting the default if you are tracking many objects at once.</p><p>The first one is a three-argument change and it is usually enough on its own:</p><pre>import torch<br>from sam2.build_sam import build_sam2_video_predictor<br><br>predictor = build_sam2_video_predictor(model_cfg, checkpoint, device=&quot;cuda&quot;)<br>state = predictor.init_state(<br>    video_path=frames_dir,<br>    offload_video_to_cpu=True,     # decoded frames stay in host memory<br>    offload_state_to_cpu=True,     # per-frame features stay in host memory<br>    async_loading_frames=True,<br>)<br>with torch.inference_mode(), torch.autocast(&quot;cuda&quot;, dtype=torch.bfloat16):<br>    predictor.add_new_points_or_box(state, frame_idx=0, obj_id=1, box=first_box)<br>    for frame_idx, obj_ids, mask_logits in predictor.propagate_in_video(state):<br>        yield frame_idx, (mask_logits[0, 0] &gt; 0).cpu().numpy()</pre><p>When the clip is long enough that even offloading is not enough, chunk it and carry the mask across the boundary:</p><pre>CHUNK = 200<br>prompt_mask = first_mask<br>for start in range(0, n_frames, CHUNK):<br>    state = predictor.init_state(video_path=chunk_dir(start), offload_state_to_cpu=True)<br>    predictor.add_new_mask(state, frame_idx=0, obj_id=1, mask=prompt_mask)<br>    for frame_idx, _, mask_logits in predictor.propagate_in_video(state):<br>        prompt_mask = (mask_logits[0, 0] &gt; 0)      # last mask becomes the next prompt<br>        write_mask(start + frame_idx, prompt_mask)<br>    predictor.reset_state(state)<br>    del state<br>    torch.cuda.empty_cache()</pre><p>The reset_state and empty_cache at the end matter more than they look. Without them, the states accumulate across iterations, and you get the same out-of-memory error one chunk later.</p><p>If none of that is enough, EdgeTAM and EfficientTAM attack this directly by compressing the memory representation itself, which is why they scale better on long clips than a smaller encoder alone would.</p><h3>If SAM is not worth the money, what do you use instead?</h3><p>There are two different situations here, and they have different answers. Either you keep the promptable interface and pay much less for it, or you drop the promptable interface completely.</p><h4>Keep the interface, shrink the model</h4><p>MobileSAM replaces the ViT-H image encoder with a TinyViT and trains it through decoupled distillation, so SAM’s original mask decoder is reused unchanged. The whole model is under 10M parameters, roughly 60 times smaller than SAM, and runs at about 10 ms per image on a single GPU (8 ms in the encoder, 4 ms in the decoder). Mask quality drops by a few mIoU points. I treat this as the default first replacement, because the API matches SAM and switching is close to a one-line change.</p><p>EdgeSAM is distilled with a prompt-in-the-loop scheme, so the student model sees the same kind of prompts it will get at inference time instead of only matching encoder features. The authors report more than 37x speedup over SAM on a 2080 Ti and more than 7x over MobileSAM and EfficientSAM on an iPhone 14, and on COCO with one or two refinement points it can even pass SAM. Choose it when the target is a phone or an NPU. Read the license before commercial use.</p><p>EfficientSAM uses SAMI, a masked image pretraining scheme where the lightweight encoder learns to reconstruct SAM’s features. Accuracy is better than MobileSAM and FastSAM at a comparable size. The catch is the training recipe: 400 epochs on ImageNet-1K plus 5 epochs on SA-1B. Use the released weights and do not plan to reproduce it.</p><p>FastSAM is not a distillation of SAM at all, which people often miss. It is a YOLOv8-seg detection branch with a YOLACT-style mask branch that generates all masks in one pass, and the prompt only selects among them afterwards. It has 68M parameters and about 40 ms per image, so MobileSAM beats it on both size and speed, and its masks agree with SAM’s less closely. It also comes with Ultralytics AGPL-3.0 licensing, which is a real constraint if you ship closed source.</p><p>For video, the two worth knowing are EdgeTAM and EfficientTAM. EdgeTAM (Meta, CVPR 2025) starts from a useful diagnosis: in SAM 2, the memory attention blocks, not only the image encoder, are the latency bottleneck, which explains why compressing the encoder alone never gave enough speedup. It adds a 2D Spatial Perceiver that compresses the frame-level memories into a fixed set of learnable queries, and reaches 16 FPS on an iPhone 15 Pro Max, about 22 times faster than SAM 2, at 87.7 J&amp;F on DAVIS 2017. EfficientTAM (ICCV 2025) goes back to a plain ViT-Tiny or ViT-Small encoder with an efficient memory cross-attention. It is around 1.6x faster and 2.4x smaller than SAM 2 with almost the same accuracy (74.5 against 74.7 on SA-V test), and the Mobile variant is about 4.6x faster and 4.5x smaller, running near 10 FPS on the same phone.</p><h3>Drop the prompt and train a specialist</h3><p>If the class list is fixed, a promptable model is a feature you are paying for and not using.</p><p>The YOLO-seg family (v8-seg, YOLO11-seg and later) gives you instance masks at a few million parameters, in real time, on hardware you already have. You need labelled data, and the licensing unless you buy a commercial license. RTMDet-Ins is the alternative I reach for when the license is a problem, since MMDetection is Apache-2.0 and the speed-accuracy trade-off is close. SegFormer with a MiT-B0 backbone or DeepLabv3+ covers semantic segmentation when instances do not matter. nnU-Net remains the honest answer for medical volumes, and it still beats most foundation-model pipelines while configuring itself from your dataset. Mask2Former is the choice when you need panoptic output and can afford the compute.</p><h3>Keep open vocabulary, pay less for it</h3><p>If the reason you wanted SAM 3 was text prompts rather than mask quality, YOLO-World and YOLOE do open-vocabulary detection and segmentation from text at real-time speed. The accuracy ceiling is lower and rare concepts suffer, but the cost per frame is not comparable.</p><h3>Sometimes no learned model at all</h3><p>Fixed camera, controlled lighting, high contrast against the background. Thresholding, watershed, GrabCut, or plain background subtraction solve a surprising number of industrial inspection and document tasks. They are deterministic, they run on a CPU, and you can explain a failure to a customer in one sentence. Spend an afternoon on this before renting a GPU.</p><h3>Quick reference</h3><p>Model Size and speed Best used when <strong>MobileSAM</strong> Under 10M params; ~10 ms per image on GPU You want SAM’s interface at a fraction of the cost <strong>EdgeSAM</strong> &gt;37x faster than SAM on a 2080 Ti; mobile oriented Phone or NPU deployment with point or box prompts <strong>EfficientSAM</strong> Ti and S variants; more accurate than MobileSAM Accuracy matters more than the last millisecond <strong>FastSAM</strong> 68M params; ~40 ms per image; AGPL-3.0 You mainly need “segment everything” proposals <strong>EdgeTAM</strong> 16 FPS on iPhone 15 Pro Max; 22x faster than SAM 2 Video tracking directly on device <strong>EfficientTAM</strong> ~1.6x faster and 2.4x smaller than SAM 2 Server-side video at lower cost, near SAM 2 accuracy <strong>YOLO-seg, RTMDet-Ins</strong> A few million params; real time Classes are fixed and you have labeled data <strong>nnU-Net</strong> Task dependent Medical volumes <strong>YOLO-World, YOLOE</strong> Real time Text prompts without the SAM 3 bill <strong>Classical CV</strong> Negligible Controlled scene, high contrast</p><p>One warning about all of the distilled variants. They inherit SAM’s behaviour, including its weaknesses. A smaller model does not fix domain shift and it does not fix mask granularity ambiguity. It only makes the same behaviour cheaper. If SAM’s masks are wrong on your data, MobileSAM’s masks will be wrong faster.</p><h3>The part people in 3D should not skip</h3><p>If you work in 3D or 4D reconstruction, SAM is not a side topic. It has quietly become the default source of object priors for a whole class of pipelines, because almost nothing else gives you dense, open-set object masks without a taxonomy.</p><p>The pattern is the same everywhere. Run SAM on the posed input views, lift the 2D masks into the 3D representation, and optimize a per-primitive or per-point identity alongside the appearance. In Gaussian Splatting, this shows up as segmentation and grouping methods that attach a learnable identity feature to each Gaussian and supervise it with SAM masks through the differentiable rasterizer. In NeRF it appeared earlier as mask-supervised feature fields. For point clouds and RGB-D scans, SAM masks projected from multiple views give you instance proposals in domains where labelled 3D data barely exists.</p><p>The failure mode is worth stating clearly, because it is the thing that costs you a week. SAM is per-image and class-agnostic, so mask identity is not consistent across views. The mask index for a chair in view 12 has nothing to do with the mask index for the same chair in view 13. Every one of these pipelines needs an association step, usually IoU or Hungarian matching between projected masks, a contrastive identity loss, or a 3D consensus vote. This is where most of the engineering time goes, not in the segmentation itself.</p><p>SAM 2 changes the economics here more than people expect. If your input views are frames of a camera trajectory rather than an unordered image set, masklet propagation gives you consistent IDs for free along the sequence, and you only need association across the loop closure instead of across every pair. For 4D and dynamic scenes, this is close to essential. SAM 3D goes a step further and produces the 3D structure directly from a single image, which is useful as an initialization or a prior, though it is not a replacement for multi-view reconstruction when you actually have multiple views.</p><h3>The architecture most teams converge to</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/914/1*NEC7-wt0Fpx0ll7_fjBNnA.png" /></figure><p>After a few months, the pipelines I see tend to look the same:</p><p>SAM runs offline as a labelling engine and a pseudo-label generator. A small supervised model runs online. A detector or a text prompt supplies the geometric prompt where a human cannot. SAM 2 or SAM 3 handles the video propagation when temporal consistency is needed, and a lighter tracker takes over when it is not.</p><p>This is less elegant than “one model does everything”, and it is cheaper, faster, and easier to certify.</p><h3>A checklist you can run in one meeting</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tA8CMNMowOmKzVECP1W27g.png" /></figure><h3>What I would tell a team today</h3><p>Use SAM to get to your first labelled dataset and your first working demo. Use it in anything where a person is in the loop. Use SAM 2 or SAM 3 when the temporal or open-vocabulary behaviour is the actual product requirement.</p><p>Then, once the task stops changing, train the small model and move SAM out of the serving path. Foundation models are excellent at giving you a starting point. They are an expensive way to keep doing the same thing forever.</p><p>And if your data is medical, industrial, or otherwise far from natural images, run your own evaluation before you commit to anything. The zero-shot numbers you read will not transfer.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=14374cb9c401" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[18. Per-Scene Optimization vs. Feed-Forward Inference: Two Ways to Reconstruct a 3D Scene]]></title>
            <link>https://medium.com/@aminfadaeinejad.edu/18-per-scene-optimization-vs-feed-forward-inference-two-ways-to-reconstruct-a-3d-scene-c1c748f5590a?source=rss-aff4faf055a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/c1c748f5590a</guid>
            <category><![CDATA[3d-reconstruction]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Amin Fadaeinejad]]></dc:creator>
            <pubDate>Mon, 03 Aug 2026 06:13:26 GMT</pubDate>
            <atom:updated>2026-08-03T06:13:26.347Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mp3vDW-fiInIvhU1UnMo8g.png" /><figcaption>Figure 1. The same reconstruction problem, solved two ways. Per-scene optimization runs SfM, then fits one scene with tens of thousands of gradient steps, and throws the parameters away afterwards. Feed-forward inference replaces the whole chain with a single pass through a frozen network whose training cost was paid once.</figcaption></figure><p>Two families of methods currently dominate 3D reconstruction from images, and they are built on opposite assumptions.</p><p>The first family treats reconstruction as an inverse rendering problem, solved separately for every scene. Neural Radiance Fields (NeRF, 2020) replaced mesh-based pipelines with a continuous volumetric field stored in the weights of an MLP. 3D Gaussian Splatting (3DGS, 2023) removed the rendering bottleneck of NeRF by returning to an explicit, primitive-based representation combined with a custom tile-based CUDA rasterizer. Both methods share the same working principle: gradient descent is used to fit one scene, and the resulting parameters are useful for that scene only.</p><p>The second family treats reconstruction as a prediction problem. DUSt3R, MASt3R, VGGT, pixelSplat, MVSplat and VolSplat train a network on large multi-view datasets so that the geometry of a new scene can be predicted in a single forward pass. There is no optimization component involved in the feedforward manner.</p><p>The two families are usually compared on speed alone, which is the least interesting difference between them. What follows is a more careful comparison: the formulations, the architectures, the failure modes, and the assumptions that are often left implicit in papers and blog posts. One clarification before starting, because it is frequently stated incorrectly: only a subset of feed-forward methods work without camera poses. DUSt3R, MASt3R and VGGT do estimate cameras. pixelSplat, MVSplat, DepthSplat, GS-LRM and VolSplat all require known poses and intrinsics at inference time.</p><h3>Part 1: Per-Scene Optimization</h3><p>Both NeRF and 3DGS formulate reconstruction as inverse rendering. If a differentiable renderer can project a 3D representation into a 2D image, then the rendering can be compared with the captured photographs and the error can be backpropagated into the representation.</p><h4>1.1 Neural Radiance Fields</h4><p>NeRF represents a scene as a continuous function <strong>F_Θ</strong> parameterized by an MLP. The network maps a 3D position <strong>x = (x, y, z)</strong> and a viewing direction <strong>d = (θ, φ)</strong> to a volume density σ and a view-dependent colour <strong>c = (r, g, b).</strong></p><p>To render a pixel, a ray <strong>r(t) = o + td </strong>is cast from the camera center. The expected colour of the pixel follows the volume rendering integral:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/332/0*HHaXeRThbJnADVwI" /></figure><p>where <strong>T(t)</strong> is the accumulated transmittance, that is, the probability that the ray reaches <strong>t</strong> without being absorbed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/276/0*Z-tzMS7PGQdYDv27" /></figure><p>The integral is approximated with numerical quadrature over stratified samples. In discrete form, with <strong>δ_i​</strong> the distance between adjacent samples:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/0*kkNkWWYToyH9cD-L" /></figure><p>This is the same alpha compositing that 3DGS later uses, with:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/181/0*UgMZiTQeRkHsgRE-" /></figure><p>The cost is dominated by MLP evaluations. Rendering a 1024 × 1024 image requires roughly one million rays, and the original NeRF samples 64 coarse plus 128 fine positions per ray through hierarchical sampling. This gives on the order of 10⁸ network evaluations for a single frame. Instant-NGP reduced this cost substantially by replacing most of the MLP capacity with a multiresolution hash encoding, but the ray marching itself remains the structural bottleneck: quality is tied to the number of samples along each ray.</p><h4>1.2 3D Gaussian Splatting</h4><p>3DGS abandons the continuous field and represents the scene with a set of anisotropic 3D Gaussians. Each Gaussian i is defined by:</p><ol><li><strong>Position μ_i: </strong>The center in world space.</li><li><strong>Covariance Σ_i:</strong> The shape and the spatial extent of the ellipsoid.</li><li><strong>Opacity α_i:</strong> A scalar passed through a sigmoid that represents the transparency.</li><li><strong>Spherical harmonics coefficients: </strong>The view-dependent colour, up to degree 3, which gives 48 coefficients per Gaussian.</li></ol><p>The covariance must stay positive semi-definite during gradient descent, so it is not optimized directly. It is factorized into a scaling matrix <strong><em>S</em></strong> and a rotation matrix <strong><em>R</em></strong> obtained from a normalized quaternion:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/134/0*VZXCNgl8MZ1InHM_" /></figure><p>The spatial influence of a Gaussian at a point <strong><em>x</em></strong> is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/368/0*LP1I0yo3FGuqSSOF" /></figure><p>Rendering avoids ray marching completely. The 3D Gaussians are projected to the image plane with the EWA splatting formulation, in which the projective transform is locally approximated by its Jacobian <strong>J</strong>. With <strong>W</strong> the viewing transformation, the 2D covariance is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/170/0*OD5lAl7kPx4wUKer" /></figure><p>The screen is divided into 16 × 16 tiles. Gaussians are duplicated into the tiles they overlap and sorted once with a single GPU radix sort on a key that combines the tile index with the view-space depth of the Gaussian center. Each tile is then composited front to back:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/228/0*GofCLSdEz3B2Fz9Y" /></figure><p>Here <strong>α’_i</strong> is not simply the learned opacity. It is the learned opacity multiplied by the projected 2D Gaussian evaluated at the pixel position:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/373/0*RrT7_9-fUkNRIT1z" /></figure><p>One detail is worth keeping in mind, because it explains a common artifact. The sort uses the depth of the Gaussian center, so the ordering is global per tile and not exact per pixel. When two large overlapping Gaussians change their relative order between neighbouring viewpoints, the rendering shows popping. This is a known limitation of the original rasterizer and the motivation behind later work on per-pixel sorting and on 2D Gaussian primitives.</p><h4>1.3 The optimization loop</h4><p>The pseudo-code below follows the reference implementation. The details that are usually omitted in summaries are the ones that actually make the optimization converge: the learning-rate schedule on positions, the periodic opacity reset, and the progressive activation of the spherical-harmonics bands.</p><pre># Pseudo-code: the 3DGS optimization pipeline (per scene)<br>def optimize_scene(images, cameras, sfm_point_cloud, max_steps=30_000):<br><br>    # 1. Initialization from the sparse SfM point cloud produced by COLMAP.<br>    #    Random initialization also works on many scenes, with some quality loss.<br>    gaussians = initialize_from_points(sfm_point_cloud)<br><br>    # 2. Adam with per-parameter-group learning rates.<br>    #    The position learning rate is multiplied by the scene extent and<br>    #    decays exponentially from 1.6e-4 to 1.6e-6 over the full schedule.<br>    optimizer = torch.optim.Adam([<br>        {&#39;params&#39;: gaussians.positions,     &#39;lr&#39;: 1.6e-4 * scene_extent},<br>        {&#39;params&#39;: gaussians.features_dc,   &#39;lr&#39;: 2.5e-3},<br>        {&#39;params&#39;: gaussians.features_rest, &#39;lr&#39;: 1.25e-4},   # 2.5e-3 / 20<br>        {&#39;params&#39;: gaussians.opacities,     &#39;lr&#39;: 5.0e-2},<br>        {&#39;params&#39;: gaussians.scaling,       &#39;lr&#39;: 5.0e-3},<br>        {&#39;params&#39;: gaussians.rotation,      &#39;lr&#39;: 1.0e-3},<br>    ], eps=1e-15)<br><br>    for step in range(max_steps):<br>        # Activate one more SH band every 1000 steps, up to degree 3.<br>        if step % 1000 == 0:<br>            gaussians.increase_sh_degree(max_degree=3)<br><br>        cam, gt_image = get_random_training_view(images, cameras)<br><br>        rendered, viewspace_points, visibility, radii = rasterize(gaussians, cam)<br><br>        # L1 combined with D-SSIM, lambda_dssim = 0.2<br>        l1 = l1_loss(rendered, gt_image)<br>        loss = (1.0 - 0.2) * l1 + 0.2 * (1.0 - ssim(rendered, gt_image))<br><br>        loss.backward()<br><br>        # 3. Adaptive Density Control: heuristic, gradient-driven, non-differentiable.<br>        if 500 &lt; step &lt; 15_000:<br>            gaussians.accumulate_viewspace_gradients(viewspace_points, visibility)<br>            if step % 100 == 0:<br>                # Clone under-reconstructed Gaussians (small, high view-space gradient)<br>                gaussians.clone(grad_threshold=2e-4)<br>                # Split over-reconstructed ones into two, scale divided by 1.6<br>                gaussians.split(grad_threshold=2e-4, size_threshold=0.01 * scene_extent)<br>                # Prune transparent and oversized primitives<br>                gaussians.prune(min_opacity=0.005, max_screen_size=20)<br>            if step % 3000 == 0:<br>                # Opacity reset: forces the optimizer to re-justify every primitive<br>                # and removes floaters that survive only because they are opaque.<br>                gaussians.reset_opacity(value=0.01)<br><br>        optimizer.step()<br>        optimizer.zero_grad(set_to_none=True)<br><br>    return gaussians</pre><p>Adaptive Density Control deserves attention because it is the least principled part of the method. The number of primitives is not learned; it is controlled by thresholds on view-space gradients, opacity, and screen-space radius. These thresholds are dataset-dependent, and much of the follow-up work on 3DGS consists of replacing them with better criteria.</p><h4>1.4 What actually limits per-scene optimization</h4><p><strong>Time:</strong> 3DGS reaches its final quality after 30k iterations, which takes roughly 20 to 40 minutes per scene on an A100-class GPU for the Mip-NeRF 360 scenes. NeRF variants without hash grids take hours. This is per scene, and it must be repeated for every new capture.</p><p><strong>Dependency on camera poses:</strong> Optimization requires accurate extrinsics and intrinsics before training begins, and these normally come from an SfM pipeline such as COLMAP. COLMAP is fragile on textureless surfaces, on strong reflections, on repetitive structures and on captures with few images. The point cloud is a softer requirement than the poses: the 3DGS paper shows that random initialization is usable on several scenes. Poses are the hard requirement, and if SfM fails or drifts, the optimization fails with it. Pose-free variants exist (BARF, Nope-NeRF, COLMAP-Free 3DGS), but they add their own optimization cost and are less robust than a working SfM run.</p><p><strong>Lack of Generalization: </strong>The parameters fitted on one room say nothing about the next room. The representation contains no prior about geometry, materials or semantics, which is exactly why sparse-view captures degrade so badly: with three input images, nothing in the objective prevents a solution that explains the training views with incorrect geometry. For NeRF, this is usually described as the shape-radiance ambiguity, since view-dependent radiance can compensate for wrong density. For 3DGS, the failure is more direct: density control creates primitives that fit the few training views and produce floaters elsewhere. Regularized variants such as RegNeRF, FreeNeRF, DNGaussian and FSGS reduce the problem but do not remove it.</p><p><strong>Storage:</strong> Each reconstructed scene is a separate asset. A 3DGS scene of a medium room typically contains one to five million primitives, which is a few hundred megabytes before compression.</p><h3>Part 2: Feed-Forward Reconstruction</h3><p>Feed-forward methods learn a mapping from a set of images to a 3D representation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/234/0*qLLY3A3RoHYpbjlo" /></figure><p>The training data determines what the model can do, and the datasets are not interchangeable. Objaverse is synthetic and object-centric, and it is used by the LRM family. RealEstate10K, ACID, ScanNet and DL3DV are real scene-level video datasets. Their camera poses are themselves produced by SfM, which means that feed-forward models inherit the biases and the errors of COLMAP through their supervision. This is rarely discussed, and it matters when evaluating pose accuracy claims.</p><p>Two branches are worth separating: models that predict geometry, and models that predict renderable Gaussians.</p><h4>2.1 Geometry prediction: DUSt3R, MASt3R and VGGT</h4><p>Classical two-view geometry estimates depth by searching for matches along epipolar lines, which requires calibrated cameras. DUSt3R removes that requirement and regresses geometry directly.</p><p>Given two uncalibrated and unposed images, DUSt3R predicts dense pointmaps together with confidence maps:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/0*vv6BH6zLvb-slT1K" /></figure><p>Both pointmaps are expressed in the coordinate frame of the first camera, which is the key design decision: the relative pose is implicit in the output rather than being an input.</p><p>The architecture is a Siamese ViT-Large encoder with shared weights, initialized from CroCo (cross-view completion) pretraining. It is followed by two ViT-Base decoders, one per view. The decoders are intertwined but do not share weights; each decoder block applies self-attention within its own view, then cross-attention to the tokens of the other view, then an MLP. DPT heads produce the pointmaps and the confidences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/460/0*uQn-o5eEi4kTZT8g" /></figure><p>Two properties of DUSt3R are commonly overlooked.</p><p>First, the prediction is defined up to scale. The training loss normalizes the pointmaps by the average distance of the points to the origin, so the network is scale-invariant by construction. A single pair gives correct shape, not metric size.</p><p>Second, DUSt3R is a two-view model. For <strong>N</strong> images, it runs on pairs and then solves a global alignment optimization in pointmap space, which recovers one similarity transform per pair and produces a globally consistent reconstruction. That stage is fast compared with COLMAP, but it is still an optimization, so calling DUSt3R “a single forward pass for a full scene” is not accurate.</p><p><strong>MASt3R</strong> (“Grounding Image Matching in 3D with MASt3R”) extends DUSt3R with a local feature head trained with an InfoNCE matching loss, plus a fast reciprocal nearest-neighbour matching scheme. It also produces metric pointmaps, which resolve the scale ambiguity of DUSt3R, and MASt3R-SfM replaces the quadratic pair enumeration with a scalable retrieval-based scheme.</p><p><strong>VGGT</strong> (“Visual Geometry Grounded Transformer”, CVPR 2025) removes the alignment stage. It processes many frames jointly with a stack of 24 blocks that alternate frame-wise self-attention and global self-attention over all tokens of all frames. Notably, it uses no cross-attention; the ablation in the paper shows that alternating self-attention outperforms a cross-attention design. Images are tokenized with DINOv2, a learned camera token is appended per frame, and separate heads predict camera extrinsics and intrinsics, depth maps, pointmaps and tracking features. The model has about 1.2 billion parameters and was trained on 64 A100 GPUs for nine days. One useful empirical finding: composing the predicted depth maps with the predicted cameras gives more accurate pointmaps than the direct pointmap head.</p><p>The cost of removing the alignment stage is memory. Global attention is quadratic in the total number of tokens, that is <strong>O((N⋅L)²)</strong> per block for <strong>N</strong> frames with <strong>L</strong> tokens each, and this dominates both runtime and memory for long sequences. This is why an entire line of follow-up work (FastVGGT and similar) focuses on making attention cheaper rather than improving the geometry.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wA8ej_PS6WfvrMLSwmcUdA.png" /><figcaption>Figure 2: pixelSplat Architecture. Source: CVPR</figcaption></figure><p>Scale is the recurring subtlety in this branch. DUSt3R and VGGT predict geometry up to an unknown similarity transform, RealEstate10K poses are themselves defined up to scale, and only metric-supervised models such as MASt3R output real-world units. Any pipeline that mixes these outputs must explicitly fix the scale.</p><h4>2.2 Gaussian prediction: pixel-aligned and voxel-aligned</h4><p>Pointmaps are sufficient for geometry, but not for novel view synthesis. A second branch regresses 3D Gaussian parameters <strong>(μ, Σ, α, SH)</strong> directly.</p><p>A note on terminology, since the naming is confusing: <strong>LRM</strong> (“Large Reconstruction Model for Single Image to 3D”) predicts a triplane NeRF, not Gaussians. The Gaussian version is <strong>GS-LRM</strong>, which predicts per-pixel Gaussians from 2 to 4 posed images in about 0.23 seconds on a single A100.</p><h4>Pixel-aligned architectures</h4><p>In pixel-aligned methods, the network predicts one or a few Gaussians per pixel of each input image. The Gaussian center is obtained by unprojecting the pixel with a predicted depth, so the 3D structure is rigidly tied to the 2D grid.</p><p><strong>pixelSplat</strong> (CVPR 2024) takes an image pair with known poses. Features come from a ResNet-50 and a DINO-pretrained ViT-B/8, and are refined by two rounds of epipolar cross-attention. The epipolar encoder is what allows the model to resolve the scale ambiguity of RealEstate10K, and the ablation is unambiguous: removing it drops PSNR from 26.09 to 19.89.</p><p>The depth is not regressed directly. pixelSplat predicts a discrete probability distribution over depth along each ray and samples the primitive location from it, using a reparameterization trick to keep the operation differentiable. The reason is specific and worth stating: direct regression of primitive positions falls into local minima, because a primitive only receives gradient where it already lies. Removing the probabilistic sampling costs about 1.5 dB in the paper’s own ablation.</p><p><strong>MVSplat</strong> replaces the epipolar transformer with a plane-sweep cost volume and a Swin Transformer backbone, which is both cheaper and more accurate. <strong>DepthSplat</strong> adds monocular depth features from a pretrained backbone. Both remain pixel-aligned.</p><p>The structural weakness of the family is multi-view consistency. If a pixel in image 1 and a pixel in image 2 observe the same surface point, the network can place two slightly different Gaussians, which produces blur and floaters near occlusion boundaries. In addition, the number of primitives is fixed at <strong>H×W×N</strong> regardless of scene content, so simple surfaces are over-represented, and fine structures are under-represented. The consequence is visible in benchmarks: pixelSplat runs out of memory beyond a handful of input views, and the pixel-aligned methods lose PSNR as views are added, which is the opposite of the expected behaviour.</p><h4>Voxel-aligned architectures</h4><p><strong>VolSplat</strong> (“Rethinking Feed-Forward 3D Gaussian Splatting with Voxel-Aligned Prediction”, ECCV 2026) decouples the primitives from the pixel grid. The pipeline is more specific than “a 3D CNN over a cost volume”, so it is worth stating precisely:</p><ul><li>A weight-sharing ResNet backbone extracts per-view features, refined by local-window cross-view attention with the two nearest views.</li><li>Per-view cost volumes are built by plane sweeping, and a depth module fuses them with monocular features from a Depth Anything V2 backbone to regress one depth map per view.</li><li>The predicted depths unproject the features into world space, producing an <strong>H×W×N</strong> point cloud with an attached feature vector per point.</li><li>The points are voxelized, and the features inside each voxel are average-pooled. The occupied voxels are the ones that receive points; there is no separate occupancy head.</li><li>A <strong>sparse</strong> 3D U-Net predicts a residual per voxel, which is added to the pooled features.</li><li>One Gaussian is decoded per occupied voxel. Its center is the voxel centroid plus a bounded offset, limited to three times the voxel size by a shifted sigmoid.</li></ul><p>This design fixes both weaknesses at once. Multi-view evidence is fused in a shared 3D container before any Gaussian is created, so duplicate primitives cannot appear, and the number of primitives follows the geometry rather than the image resolution. Because the grid is sparse, the memory does not scale with the cube of the grid resolution, which is the usual objection to volumetric methods and does not apply here.</p><pre># Pseudo-code: voxel-aligned feed-forward splatting (VolSplat-style)<br># Camera poses and intrinsics are REQUIRED inputs for this family.<br>@torch.no_grad()<br>def feedforward_voxel_splat(images, cameras, voxel_size):<br><br>    # 1. Per-view 2D features + cross-view attention with nearest neighbours<br>    feats = resnet_backbone(images)                       # [N, C, H/p, W/p]<br>    feats = cross_view_window_attention(feats, cameras)<br><br>    # 2. Plane-sweep cost volume, fused with monocular depth features<br>    cost_volumes = build_cost_volumes(feats, cameras, n_depth_hypotheses=128)<br>    mono = depth_anything_backbone(images)<br>    depths = depth_head(cost_volumes, mono)               # [N, H, W]<br><br>    # 3. Unproject features into world space<br>    points, point_feats = unproject(depths, feats, cameras)   # ~N*H*W points<br><br>    # 4. Voxelize; occupancy is defined by which voxels receive points<br>    voxel_idx = torch.round(points / voxel_size).long()<br>    voxel_feats = average_pool_by_index(point_feats, voxel_idx)<br><br>    # 5. Sparse 3D U-Net predicts a residual (not a full re-encoding)<br>    voxel_feats = voxel_feats + sparse_unet3d(voxel_feats, voxel_idx)<br><br>    # 6. One Gaussian per occupied voxel, centred near the voxel centroid<br>    centroids = voxel_idx * voxel_size<br>    offsets   = 3.0 * voxel_size * (sigmoid(offset_head(voxel_feats)) - 0.5)<br>    return GaussianModel(<br>        positions = centroids + offsets,<br>        scales    = exp(scale_head(voxel_feats)),<br>        rotations = normalize_quaternion(rotation_head(voxel_feats)),<br>        opacities = sigmoid(opacity_head(voxel_feats)),<br>        sh_coeffs = sh_head(voxel_feats),<br>    )</pre><h3>Part 3: Comparison</h3><h4>3.1 Computational cost</h4><p>Per-scene optimization costs <strong>O(S×N_{rays})</strong> for NeRF and <strong>O(S×N_G)</strong> for 3DGS, where <strong>S </strong>is the number of steps. In practice, this is minutes to hours per scene, and it is paid again for every capture.</p><p>Feed-forward inference is constant with respect to scene complexity, but it is not constant in general, and describing it as <strong>O(1)</strong> is misleading. Two terms grow:</p><ul><li><strong>The number of input views.</strong> Global attention in VGGT is quadratic in the total token count. Pixel-aligned splatting methods produce <strong>H×W×N</strong> primitives, so memory grows linearly with the number of views and pixelSplat already goes out of memory at 12 views of 256 × 256.</li><li><strong>The input resolution.</strong> Both the token count and the primitive count scale with it.</li></ul><p>Typical numbers: GS-LRM reports 0.23 s for 2 to 4 posed images on an A100; MVSplat reports around 0.04 s for a two-view pair at 256 × 256; VGGT reconstructs a small set of frames in under a second. These are all measured at low resolution, which is a real constraint rather than a detail.</p><h4>3.2 Sparse views and unobserved regions</h4><p>Per-scene optimization degrades severely with three or four images unless it is regularized, because nothing constrains the geometry outside the training views.</p><p>Feed-forward models are much better in this regime, but their advantage is often described incorrectly. A model such as DUSt3R, VGGT, MVSplat or VolSplat does not invent the back of a chair it has never seen. What the learned prior provides is a strong bias about plausible surfaces, monocular depth cues and matching, so that few views are enough to constrain the geometry that <em>is</em> observed. Genuine completion of unobserved regions requires a generative prior, which is a different class of model: multi-view diffusion approaches such as Zero-1-to-3, Instant3D and CAT3D, or object-level LRM variants trained on Objaverse, which do produce plausible unseen sides but only within the distribution of their training data. Reconstruction and generation are still separate problems, and blurring them in a discussion of DUSt3R or VGGT is a mistake.</p><h4>3.3 Fidelity and resolution</h4><p>Per-scene optimization still produces the best absolute quality on dense captures. Because the representation overfits the observed pixels, it recovers high-frequency detail such as wood grain, fabric texture and specular highlights, and it scales to 4K.</p><p>Feed-forward quality is bounded by the resolution of the feature maps, by the training resolution, and by the number of primitives the architecture can emit. Most scene-level feed-forward models are trained at 256 × 256, so textures tend to be smoothed. Comparing their PSNR against per-scene 3DGS numbers is not meaningful: the benchmarks differ in baseline, view count, and resolution. Within the feed-forward setting, the comparison is fair, and the current ordering on RealEstate10K with 6 input views is roughly VolSplat 31.30 dB, DepthSplat 30.52 dB, TranSplat 29.62 dB, MVSplat 29.13 dB, pixelSplat 28.95 dB.</p><h4>3.4 Summary table</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2sL6k0l6U6Iyf6lZMA2Heg.png" /></figure><h3>Part 4: The Hybrid Pipeline</h3><p>The two families are already converging in practice, and the useful combination is feed-forward initialization followed by short per-scene optimization.</p><p>The bottleneck when deploying 3DGS on casual captures is not the optimizer; it is COLMAP. Handheld video with motion blur, low texture or a short baseline often produces an unstable camera track, and 3DGS fails downstream. Replacing SfM with a learned model removes that failure mode.</p><p><strong>InstantSplat</strong> is the clearest example of this pattern. It runs DUSt3R with global alignment to obtain poses, intrinsics and a dense point cloud, downsamples the over-parameterized point cloud with confidence-aware farthest point sampling, and then jointly optimizes Gaussian attributes and camera parameters with pose regularization. The reported split is about 20.6 s for the initialization and 16.7 s for the optimization, so under a minute in total, compared with 32 percent higher SSIM and 80 percent lower absolute trajectory error against pose-free baselines on Tanks and Temples. The same recipe now appears with VGGT in place of DUSt3R.</p><p>Three points matter when building such a pipeline:</p><ul><li><strong>The initial poses are approximate and must remain trainable.</strong> They are a good starting point, not ground truth, which is why joint optimization of cameras and primitives is part of the design rather than an optional extra.</li><li><strong>Scale must be fixed explicitly.</strong> If the geometry comes from a scale-invariant model, the reconstruction has no physical units, which matters for robotics and for any measurement task. A metric model such as MASt3R, or a known reference length, is required.</li><li><strong>The dense predicted point cloud needs pruning.</strong> One point per pixel per view is far more primitives than the optimizer needs, and keeping all of them slows convergence instead of accelerating it.</li></ul><h3>Part 5: Conclusion</h3><p>Per-scene optimization is not obsolete. It still produces the highest fidelity on dense, well-posed captures, and it will keep that position as long as feed-forward models are trained at low resolution.</p><p>What has changed is the role of the classical front end. Camera estimation, dense matching and initial geometry are now solved better and faster by learned models than by SfM on the difficult inputs, and this is where feed-forward methods have already won. The reconstruction stage is a more open question: pixel-aligned splatting has clear structural limits, voxel-aligned prediction addresses them, and both are still constrained by training resolution and by the cost of attention over many views.</p><p>For engineers working in this area, the questions worth following are concrete: how to make global attention over many frames affordable, how to decouple primitive density from the input grid, how to obtain metric scale without a calibration target, and how to combine a learned prior with a short optimization without inheriting the fragility of either. The generation of unobserved geometry remains a separate problem, and it should be treated as such.</p><h3>References</h3><ol><li>Mildenhall et al. <em>NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis</em>. ECCV 2020. <a href="https://arxiv.org/abs/2003.08934">https://arxiv.org/abs/2003.08934</a></li><li>Müller et al. <em>Instant Neural Graphics Primitives with a Multiresolution Hash Encoding</em>. SIGGRAPH 2022. <a href="https://arxiv.org/abs/2201.05989">https://arxiv.org/abs/2201.05989</a></li><li>Kerbl et al. <em>3D Gaussian Splatting for Real-Time Radiance Field Rendering</em>. SIGGRAPH 2023. <a href="https://arxiv.org/abs/2308.04079">https://arxiv.org/abs/2308.04079</a></li><li>Zwicker et al. <em>EWA Volume Splatting</em>. IEEE Visualization 2001. <a href="https://doi.org/10.1109/VISUAL.2001.964490">https://doi.org/10.1109/VISUAL.2001.964490</a></li><li>Schönberger and Frahm. <em>Structure-from-Motion Revisited</em> (COLMAP). CVPR 2016. <a href="https://doi.org/10.1109/CVPR.2016.445">https://doi.org/10.1109/CVPR.2016.445</a></li><li>Barron et al. <em>Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields</em>. CVPR 2022. <a href="https://arxiv.org/abs/2111.12077">https://arxiv.org/abs/2111.12077</a></li><li>Weinzaepfel et al. <em>CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completion</em>. NeurIPS 2022. <a href="https://arxiv.org/abs/2210.10716">https://arxiv.org/abs/2210.10716</a></li><li>Wang et al. <em>DUSt3R: Geometric 3D Vision Made Easy</em>. CVPR 2024. <a href="https://arxiv.org/abs/2312.14132">https://arxiv.org/abs/2312.14132</a></li><li>Leroy et al. <em>Grounding Image Matching in 3D with MASt3R</em>. ECCV 2024. <a href="https://arxiv.org/abs/2406.09756">https://arxiv.org/abs/2406.09756</a></li><li>Wang et al. <em>VGGT: Visual Geometry Grounded Transformer</em>. CVPR 2025. <a href="https://arxiv.org/abs/2503.11651">https://arxiv.org/abs/2503.11651</a></li><li>Charatan et al. <em>pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction</em>. CVPR 2024. <a href="https://arxiv.org/abs/2312.12337">https://arxiv.org/abs/2312.12337</a></li><li>Chen et al. <em>MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images</em>. ECCV 2024. <a href="https://arxiv.org/abs/2403.14627">https://arxiv.org/abs/2403.14627</a></li><li>Xu et al. <em>DepthSplat: Connecting Gaussian Splatting and Depth</em>. CVPR 2025. <a href="https://arxiv.org/abs/2410.13862">https://arxiv.org/abs/2410.13862</a></li><li>Hong et al. <em>LRM: Large Reconstruction Model for Single Image to 3D</em>. ICLR 2024. <a href="https://arxiv.org/abs/2311.04400">https://arxiv.org/abs/2311.04400</a></li><li>Zhang et al. <em>GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting</em>. ECCV 2024. <a href="https://arxiv.org/abs/2404.19702">https://arxiv.org/abs/2404.19702</a></li><li>Wang et al. <em>VolSplat: Rethinking Feed-Forward 3D Gaussian Splatting with Voxel-Aligned Prediction</em>. ECCV 2026. <a href="https://arxiv.org/abs/2509.19297">https://arxiv.org/abs/2509.19297</a></li><li>Fan et al. <em>InstantSplat: Sparse-view Gaussian Splatting in Seconds</em>. 2024. <a href="https://arxiv.org/abs/2403.20309">https://arxiv.org/abs/2403.20309</a></li><li>Yang et al. <em>Depth Anything V2</em>. NeurIPS 2024. <a href="https://arxiv.org/abs/2406.09414">https://arxiv.org/abs/2406.09414</a></li><li>Gao et al. <em>CAT3D: Create Anything in 3D with Multi-View Diffusion Models</em>. NeurIPS 2024. <a href="https://arxiv.org/abs/2405.10314">https://arxiv.org/abs/2405.10314</a></li><li>Zhou et al. <em>Stereo Magnification: Learning View Synthesis using Multiplane Images</em> (RealEstate10K). SIGGRAPH 2018. <a href="https://arxiv.org/abs/1805.09817">https://arxiv.org/abs/1805.09817</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1c748f5590a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[17. Unlocking Fast LLM Inference: A Deep Dive into KV Caching]]></title>
            <link>https://medium.com/@aminfadaeinejad.edu/17-unlocking-fast-llm-inference-a-deep-dive-into-kv-caching-648ea1674099?source=rss-aff4faf055a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/648ea1674099</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Amin Fadaeinejad]]></dc:creator>
            <pubDate>Sat, 01 Aug 2026 22:08:35 GMT</pubDate>
            <atom:updated>2026-08-01T22:08:35.581Z</atom:updated>
            <content:encoded><![CDATA[<p>Large Language Models (LLMs) have reshaped the AI landscape, but moving them from training to production exposes a severe efficiency bottleneck. The Transformer architecture introduced by Vaswani et al. (2017) [1] is highly parallel during training, where the full target sequence is available, and every position can be processed at once. Generation at inference time is a different regime: it is strictly sequential, and the properties that make training efficient no longer apply.</p><p>In this post, I go through the cost structure of Transformer decoding, the tensor mechanics of KV Caching, the exact memory arithmetic of the cache, and the architectural solutions that keep inference scalable today (MQA, GQA, and PagedAttention).</p><h3>1. The Autoregressive Bottleneck and the Two Inference Phases</h3><p>Inference splits into two phases with fundamentally different cost profiles: the <strong>Prefill Phase</strong> and the <strong>Decode Phase</strong>.</p><p><strong>Prefill Phase (Time-To-First-Token, TTFT):</strong> The user provides a prompt. The model processes the entire prompt in a single forward pass, which maps to large matrix-matrix multiplications with high arithmetic intensity. This phase is <strong>compute-bound</strong>: it is limited by the FLOP/s throughput of the GPU.</p><p><strong>Decode Phase (Time-Per-Output-Token, TPOT):</strong> The model returns one token at a time. The prediction of token <strong><em>t</em></strong> is conditioned on tokens <em>0</em> to <strong><em>t-1</em></strong>. Each step is a matrix-vector product per layer, so arithmetic intensity is low and the phase is <strong>memory-bandwidth-bound</strong>: it is limited by how fast weights and cached activations can be moved from HBM into the compute units.</p><p>Without any optimization, naive generation recomputes the hidden states of the whole sequence at every decode step. It is worth being precise about the cost here, because this figure is often quoted incorrectly. A single forward pass over a sequence of length <strong><em>N </em></strong>costs <strong>O(N² d)</strong> in the attention operation, where <strong><em>d</em></strong> is the hidden dimension. But naive generation performs one such pass per generated token: the pass at step <strong><em>t</em></strong> costs <strong>O(t² d)</strong>, and summing over <strong>t = 1… N</strong> yields</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/192/0*RT_3159Ya_TaDivI" /></figure><p>So uncached autoregressive generation is cubic in the sequence length, not quadratic. The overwhelming majority of that work is redundant: it re-derives, at every step, key and value representations that were already computed and have not changed.</p><p>Without optimization, standard generation looks like this:</p><ol><li>Feed [Token 1] → Predict [Token 2]</li><li>Feed [Token 1, Token 2] → Predict [Token 3]</li><li>Feed [Token 1, Token 2, Token 3] → Predict [Token 4]</li></ol><p>Do you see the problem? To predict [Token 4], the model recomputes the attention representations for Token 1, Token 2, and Token 3 completely from scratch. As the sequence length <strong><em>N </em></strong>grows, the number of matrix multiplications grows quadratically. This recomputation of past states is the autoregressive bottleneck.</p><h3>2. How KV Caching Works</h3><p>The Self-Attention mechanism projects the hidden states entering a layer into three sets of vectors: Query (Q), Key (K), and Value (V). Only the first layer operates directly on token embeddings; all subsequent layers project the output of the layer below.</p><p>The observation that makes KV caching possible is an invariance property. Because the decoder uses causal masking, the hidden state at position <strong><em>i</em></strong> is a function of positions <strong><em>0</em></strong> to <strong><em>i</em></strong> only. Appending tokens at positions <strong><em>i+1, i+2,…</em></strong>therefore cannot change the hidden state at position <strong><em>i</em></strong>, and consequently cannot change <strong><em>K_i</em></strong> or <strong><em>V_i</em></strong> either. These tensors are computed once and remain valid for the entire remaining generation.</p><p>Rather than recomputing them, we allocate a buffer in GPU memory, the KV Cache, and store them there.</p><p>Consider the decode step for token <strong><em>t</em></strong> with batch size 1, written per attention head with head dimension <strong><em>d_h</em></strong>:</p><ul><li><strong>Current token Query.</strong> We compute <strong><em>Q_t ∈ R^{1 </em></strong>×<strong><em> 1 </em></strong>× <strong><em>d_h}</em></strong> for each of the <strong><em>n_q</em></strong> query heads.</li><li><strong>Current token Key/Value.</strong> We compute <strong><em>K_t, V_t ∈ R^{1 × 1 × d_h}</em></strong> for each of the <strong><em>n_{kv}</em></strong> key/value heads. Under standard Multi-Head Attention <strong><em>n_{kv} = n_q</em></strong>, so the concatenated KV width equals the model dimension <strong><em>d</em></strong>. Under MQA and GQA it does not, which is exactly the lever exploited in Section 6.</li><li><strong>Cache retrieval.</strong> We read the cached keys <strong><em>K_{1:t-1}</em></strong> and values <strong><em>V_{1:t-1}</em></strong> from the buffer.</li><li><strong>Concatenation.</strong> We write <strong><em>K_t</em></strong> and <strong><em>V_t</em></strong> into the cache. The effective tensors for this step are <strong><em>K_{total}, V_{total} ∈ R^{1 × t × d_h}</em></strong> per head.</li><li><strong>Attention scores.</strong> The product <strong><em>Q_t K^T_{total}</em></strong> produces a score vector of shape <strong><em>1 × t</em></strong>, describing how strongly the new token attends to each past position.</li></ul><p>With this scheme, the attention cost of a single decode step drops from <strong>O(t² d)</strong> to <strong>O(t , d)</strong>, and the cumulative cost of generating <strong><em>N</em></strong> tokens drops from <strong>O(N³ d)</strong> to <strong>O(N² d)</strong>.</p><p><strong>[Important Note]: </strong>One implementation detail is easy to get wrong. If the model uses RoPE, the rotation is position-dependent and must be applied before the key is written to the cache. Storing pre-rotation keys and rotating on read would re-introduce per-step work and, in a sliding-window setting, would silently change the encoded positions.</p><h3>3. Visualizing the Flow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0GHIYtLxacxNXOeXcPOCZA.png" /><figcaption>Figure 1: Standard autoregressive generation without caching. Every decode step recomputes <em>K</em> and <em>V</em> for the full prefix.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bKqr-z77epE7hKuPgw4NOA.png" /><figcaption>Figure 2: Autoregressive generation with a KV cache. Only the new token is projected; earlier <em>K</em> and <em>V</em> are read from the buffer.</figcaption></figure><h3>4. Pseudocode: Inside the Attention Head</h3><p>The two functions below are deliberately simplified to a single head, with shapes (batch, seq, dim). A production implementation carries an explicit head axis, (batch, n_heads, seq, head_dim).</p><h4>4.1 Standard Forward Pass (No Cache)</h4><pre>import torch<br>import torch.nn.functional as F<br>import math<br><br>def standard_attention(query_states, key_states, value_states):<br>    &quot;&quot;&quot;<br>    Standard causal attention. query_states, key_states and value_states<br>    all have shape (batch_size, seq_len, hidden_dim).<br>    &quot;&quot;&quot;<br>    d_k = query_states.size(-1)<br>    seq_len = query_states.size(-2)<br>    # (B, S, d) x (B, d, S) -&gt; (B, S, S)<br>    scores = torch.matmul(query_states, key_states.transpose(-2, -1)) / math.sqrt(d_k)<br>    # Causal mask: position i must not attend to any position j &gt; i.<br>    # Omitting this makes the layer bidirectional and breaks autoregressive<br>    # decoding, so it is not optional.<br>    causal_mask = torch.triu(<br>        torch.ones(seq_len, seq_len, dtype=torch.bool, device=scores.device),<br>        diagonal=1,<br>    )<br>    scores = scores.masked_fill(causal_mask, float(&quot;-inf&quot;))<br>    attn_weights = F.softmax(scores, dim=-1)<br>    # (B, S, S) x (B, S, d) -&gt; (B, S, d)<br>    output = torch.matmul(attn_weights, value_states)<br>    return output</pre><h4>4.2 Cached Forward Pass (Decode Step)</h4><pre>def cached_attention_decode_step(<br>    new_query,<br>    new_key,<br>    new_value,<br>    past_key_cache,<br>    past_value_cache<br>):<br>    &quot;&quot;&quot;<br>    new_query, new_key, new_value: shape (batch_size, 1, hidden_dim) — the new token only.<br>    past_key_cache, past_value_cache: shape (batch_size, seq_len_so_far, hidden_dim)No causal mask is required here. The query is a single position at the end of<br>    the sequence, so attending to the full cache is already causal by construction.<br>    &quot;&quot;&quot;<br>    d_k = new_query.size(-1)<br>    # 1. Update the cache along the sequence dimension (dim=-2)<br>    #    Resulting shape: (batch_size, seq_len_so_far + 1, hidden_dim)<br>    #<br>    #    Note: torch.cat allocates a new tensor at every step, which makes the<br>    #    cache update O(N^2) in memory traffic over a full generation. Production<br>    #    engines pre-allocate a fixed buffer and write in place at index t.<br>    updated_key_cache = torch.cat([past_key_cache, new_key], dim=-2)<br>    updated_value_cache = torch.cat([past_value_cache, new_value], dim=-2)<br>    # 2. Score the single new token against all cached keys<br>    # (B, 1, d) x (B, d, S+1) -&gt; (B, 1, S+1)<br>    scores = torch.matmul(new_query, updated_key_cache.transpose(-2, -1)) / math.sqrt(d_k)<br>    attn_weights = F.softmax(scores, dim=-1)<br>    # 3. Compute the output for the new token<br>    # (B, 1, S+1) x (B, S+1, d) -&gt; (B, 1, d)<br>    output = torch.matmul(attn_weights, updated_value_cache)<br>    # Return the output plus the updated caches, which stay resident in VRAM<br>    return output, updated_key_cache, updated_value_cache<br></pre><h3>5. The Memory Wall: Sizing the KV Cache</h3><p>KV caching removes the redundant computation, but it converts the problem into a memory problem. Two distinct constraints are involved, and they should not be confused:</p><ul><li><strong>Capacity.</strong> The cache occupies VRAM that is not available for anything else, which caps the achievable batch size and the maximum context length.</li><li><strong>Bandwidth.</strong> The entire cache must be read from HBM at every decode step, which places a hard floor under TPOT regardless of available FLOP/s.</li></ul><p>The section below quantifies the first of these. The size of the cache in bytes per token is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*_CuwGBtn81YYjGjA" /></figure><p>The leading factor of 2 accounts for storing <strong><em>K</em></strong> and <strong><em>V</em></strong>. Note that this stores activations, not parameters, so the precision here is the activation dtype.</p><p><strong>Example: Llama2-7B</strong></p><ul><li>Layers: 32</li><li>KV heads: 32 (Llama 2 7B uses standard MHA, not GQA)</li><li>Head dimension: 128</li><li>Precision: FP16, i.e. 2 bytes per element</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/0*fRIm7Ev4qXcCR6Be" /></figure><p>For a 4,096-token sequence, a single request therefore requires exactly 2 GiB of VRAM for its cache alone. A batched serving deployment holding 32 concurrent requests at that length needs roughly 64 GiB, considerably more than the ~13.5 GiB occupied by the FP16 model weights themselves. At this point the cache, not the checkpoint, is the dominant term in the memory budget.</p><h3>6. Modern Architectural Solutions</h3><p>Because the cache scales linearly with layers, KV heads, sequence length and batch size, several lines of work attack it directly.</p><p><strong>Multi-Query Attention (MQA).</strong> Introduced by Shazeer (2019) in <em>“Fast Transformer Decoding: One Write-Head is All You Need”</em> [2]. MQA keeps <strong><em>n_q</em></strong> query heads but reduces the model to a <strong>single</strong> shared Key/Value head. This divides the cache footprint by a factor of <strong><em>n_q</em></strong> — 32× for the Llama 2 7B configuration above. The trade-off is a measurable loss in quality and, as later work reported, some training instability.</p><p><strong>Grouped-Query Attention (GQA).</strong> Proposed by Ainslie et al. (2023) in <em>“GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints”</em> [3]. GQA interpolates between MHA and MQA: the <strong><em>n_q</em></strong> query heads are partitioned into <strong><em>G</em></strong> groups, and each group shares one Key/Value head, so the cache shrinks by a factor of <strong><em>n_q / G</em></strong>. A common configuration is 32 query heads with 8 KV heads, giving a 4× reduction. GQA is now the default in production models Llama 2 34B and 70B, all sizes of Llama 3, and Mistral 7B. It is worth being accurate here, since it is a frequent source of confusion: Llama 2 7B and 13B do <strong>not</strong> use GQA; they retain standard MHA.</p><p><strong>PagedAttention (vLLM).</strong> Traditional serving systems reserve a contiguous VRAM block sized for the maximum possible sequence length of each request. Since the true length is unknown in advance, this produces heavy internal fragmentation, plus external fragmentation as requests of differing sizes are allocated and freed. Kwon et al. (2023), in <em>“Efficient Memory Management for Large Language Model Serving with PagedAttention”</em> [4], measured that existing systems wasted <strong>60–80%</strong> of KV cache memory this way. Their solution treats cache blocks the way an operating system treats virtual memory pages: fixed-size blocks allocated on demand in non-contiguous memory, indexed through a block table. This brings waste below 4%, and the block table additionally enables prefix sharing between requests that begin with the same system prompt.</p><p><strong>Multi-head Latent Attention (MLA).</strong> A more recent direction, used in DeepSeek-V2, which takes a different route: instead of sharing KV heads, it projects <strong><em>K</em></strong> and <strong><em>V</em></strong> down into a low-rank latent vector that is cached in place of the full tensors, and reconstructs them on read. This achieves larger reductions than GQA at comparable quality, at the cost of a more complex kernel.</p><h3>Closing</h3><p>In modern LLM engineering, the KV cache is not an optional optimization. It defines the cost model of the decode phase; it sets the ceiling on batch size and context length, and every serving system in production is organized around managing it. Understanding its arithmetic is a prerequisite for reasoning about inference performance at all.</p><h3>References</h3><ol><li>Vaswani, A., et al. (2017). <em>“Attention Is All You Need.”</em> Advances in Neural Information Processing Systems (NeurIPS).</li><li>Shazeer, N. (2019). <em>“Fast Transformer Decoding: One Write-Head is All You Need.”</em> arXiv:1911.02150.</li><li>Ainslie, J., et al. (2023). <em>“GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints.”</em> arXiv:2305.13245.</li><li>Kwon, W., et al. (2023). <em>“Efficient Memory Management for Large Language Model Serving with PagedAttention.”</em> Proceedings of the 29th Symposium on Operating Systems Principles (SOSP).</li><li>Touvron, H., et al. (2023). <em>“Llama 2: Open Foundation and Fine-Tuned Chat Models.”</em> arXiv:2307.09288.</li><li>DeepSeek-AI (2024). <em>“DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model.”</em> arXiv:2405.04434.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=648ea1674099" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[16. V-JEPA 2 in Practice: Loading, Feature Extraction, and Fine-Tuning the Encoder]]></title>
            <link>https://medium.com/@aminfadaeinejad.edu/16-v-jepa-2-in-practice-loading-feature-extraction-and-fine-tuning-the-encoder-85aafe60fb0a?source=rss-aff4faf055a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/85aafe60fb0a</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Amin Fadaeinejad]]></dc:creator>
            <pubDate>Tue, 28 Jul 2026 06:12:20 GMT</pubDate>
            <atom:updated>2026-07-28T06:12:20.574Z</atom:updated>
            <content:encoded><![CDATA[<p>In my previous article, I described what a Joint Embedding Predictive Architecture is and why V-JEPA 2 <a href="https://arxiv.org/abs/2506.09985">[1]</a> predicts in a latent space instead of at the pixel level. That post was conceptual. Many readers asked the obvious follow-up question: how do I actually run this thing?</p><p>This is that follow-up. I keep the prose short and the code long. By the end, you will be able to:</p><ol><li>Set up a clean environment for V-JEPA 2,</li><li>Load the pretrained encoder and extract video and image embeddings,</li><li>Run the released video-classification checkpoint,</li><li>Attach and train an attentive probe on top of a frozen encoder for your own labels.</li><li>Understand where the action-conditioned world model (V-JEPA 2-AC) fits in.</li></ol><p>I assume you are comfortable with pytorch and the transformers library, so I do not stop to explain tensors or DataLoader. Every snippet below is written to run as-is on a single GPU, and I point out the two or three places where people typically get stuck.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sjdCgcHR4ss3GOtaBsjS4w.gif" /><figcaption>Figure 1: JEPA architecture</figcaption></figure><h3>1. Environment setup</h3><p>V-JEPA 2 is integrated into Hugging Face transformers, so for inference and probe-based fine-tuning you do not need Meta&#39;s original repository at all. You only need it if you want the pre-training loops or the robot world model.</p><p>One thing you will need is <strong>torchcodec</strong>, which the official examples use for video decoding <a href="https://huggingface.co/docs/transformers/en/model_doc/vjepa2">[3]</a><a href="https://github.com/pytorch/torchcodec">[9]</a>, and it relies on FFmpeg.</p><pre># CUDA-enabled PyTorch is strongly recommended<br>pip install -U torch torchvision<br><br># transformers has native V-JEPA 2 support from recent releases;<br># install from source if your version is older than the model release<br>pip install -U transformers<br><br># video decoding used by the official snippets<br>pip install -U torchcodec<br><br># used by the original Meta checkpoints / torch.hub path<br>pip install -U timm einops</pre><p>A quick sanity check that the classes are importable:</p><pre>from transformers import (<br>    AutoModel,<br>    AutoVideoProcessor,<br>    AutoModelForVideoClassification,<br>    VJEPA2Config,<br>)<br>print(&quot;V-JEPA 2 classes are available.&quot;)</pre><p>If the import fails, your transformers version predates the model. Install from the main branch and retry.</p><h3>2. Choosing a checkpoint</h3><p>Before writing any code, it helps to know which checkpoints exist, because the naming convention encodes the important properties. All of them live under the official V-JEPA 2 collection on the Hub <a href="https://huggingface.co/collections/facebook/v-jepa-2-6841bad8413014e185b497a6">[6]</a>.</p><p>The pattern is vjepa2-{encoder}-fpc{N}-{resolution}-[{task}]:</p><ul><li><strong>encoder</strong>: vitl (ViT-L, ~300M), vith (ViT-H), vitg (ViT-g, the 1B-scale encoder),</li><li><strong>fpc</strong>: <em>frames per clip</em> used during pre-training,</li><li><strong>resolution</strong>: input crop size (256 or 384),</li><li><strong>task</strong>: present only on fine-tuned heads, e.g. ssv2 for Something-Something v2.</li></ul><p>Here is a quick insight on the models:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oyzQqXHWrTXgDbVI1aviTQ.png" /></figure><p>One detail worth internalizing: frames_per_clip in the config records how the model was pre-trained; it does <strong>not</strong> force you to feed exactly that many frames at inference<a href="https://huggingface.co/docs/transformers/en/model_doc/vjepa2"> [3]</a>. You can sample 16 frames for a fpc64 encoder without any problem. I return to this point in the fine-tuning section, because it is exactly the freedom that makes attentive probing cheap.</p><h3>3. Loading the encoder and extracting features</h3><p>This is the core operation. Everything else is built on top of it. We load the encoder, decode a handful of frames, pass them through the video processor, and read the hidden states.</p><pre>import numpy as np<br>import torch<br>from torchcodec.decoders import VideoDecoder<br>from transformers import AutoVideoProcessor, AutoModel<br><br>hf_repo = &quot;facebook/vjepa2-vitl-fpc64-256&quot;<br>processor = AutoVideoProcessor.from_pretrained(hf_repo)<br>model = AutoModel.from_pretrained(<br>    hf_repo,<br>    device_map=&quot;auto&quot;,<br>    attn_implementation=&quot;sdpa&quot;,   # or &quot;flash_attention_2&quot; if installed<br>)<br>model.eval()<br># Decode some frames. Any .mp4 works; this is the sample from the docs.<br>video_url = (<br>    &quot;https://huggingface.co/datasets/nateraw/kinetics-mini/&quot;<br>    &quot;resolve/main/val/archery/-Qz25rXdMjE_000014_000024.mp4&quot;<br>)<br>vr = VideoDecoder(video_url)<br>frame_idx = np.arange(0, 64)                     # a simple contiguous sampling<br>video = vr.get_frames_at(indices=frame_idx).data # tensor of shape T x C x H x W<br>inputs = processor(video, return_tensors=&quot;pt&quot;).to(model.device)<br>with torch.no_grad():<br>    outputs = model(**inputs)<br># Encoder output: this is what you use as a video representation<br>encoder_outputs = outputs.last_hidden_state<br>print(&quot;encoder:&quot;, encoder_outputs.shape)<br># Predictor output: the JEPA predictor branch<br>predictor_outputs = outputs.predictor_output.last_hidden_state<br>print(&quot;predictor:&quot;, predictor_outputs.shape)</pre><h4>Reading the shapes</h4><p>The encoder output has shape (batch, num_tokens, hidden_size). It is worth understanding where num_tokens comes from, because it explains the memory cost.</p><p>V-JEPA 2 patchifies video into <strong>tubelets</strong>: it groups tubelet_size frames (default 2) together and splits each into patch_size × patch_size spatial patches (default 16). So for a ViT-L encoder with 64 frames at 256×256:</p><pre>temporal tokens = 64 / 2   = 32<br>spatial tokens  = (256/16) × (256/16) = 16 × 16 = 256<br>num_tokens      = 32 × 256 = 8192<br>hidden_size     = 1024      (ViT-L)</pre><p>That is 8192 tokens per clip. This is why V-JEPA 2 is heavier than an image ViT and why frame sampling matters so much for fine-tuning. If you want a single vector per clip rather than a token sequence, mean-pool over the token dimension, but note that the recommended way to summarise a clip is the attentive pooler, which we train in Section 6.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o8Rp4pkBbm-z8Cy5F6lFIw.png" /><figcaption><strong>Figure 2.</strong> How frames become tokens: tubelet_size frames are merged, then split into spatial patches. This is the source of the 8192 tokens above.</figcaption></figure><p>V-JEPA 2 also represents still images: you simply repeat the image along the temporal axis so it looks like a static clip <a href="https://huggingface.co/facebook/vjepa2-vitl-fpc64-256">[5]</a>.</p><pre>from transformers.image_utils import load_image<br><br>image = load_image(<br>    &quot;https://huggingface.co/datasets/merve/coco/&quot;<br>    &quot;resolve/main/val2017/000000000285.jpg&quot;<br>)<br>pixel_values = processor(image, return_tensors=&quot;pt&quot;).to(model.device)[&quot;pixel_values_videos&quot;]<br>pixel_values = pixel_values.repeat(1, 16, 1, 1, 1)   # repeat the frame 16 times<br>with torch.no_grad():<br>    image_embeddings = model.get_vision_features(pixel_values)<br>print(&quot;image embedding:&quot;, image_embeddings.shape)</pre><p>get_vision_features is a convenience method that returns exactly the encoder features. In current transformers it calls forward(..., skip_predictor=True), so the predictor is not run, but note this is version-dependent.</p><h3>4. Running the released classifier</h3><p>If your task happens to be Something-Something v2, or you just want to see the model produce human-readable labels, use the fine-tuned checkpoint that already ships with a classification head <a href="https://huggingface.co/docs/transformers/en/model_doc/vjepa2">[3].</a></p><pre>import numpy as np<br>import torch<br>from torchcodec.decoders import VideoDecoder<br>from transformers import AutoModelForVideoClassification, AutoVideoProcessor<br><br>hf_repo = &quot;facebook/vjepa2-vitl-fpc16-256-ssv2&quot;<br>model = AutoModelForVideoClassification.from_pretrained(hf_repo, device_map=&quot;auto&quot;)<br>processor = AutoVideoProcessor.from_pretrained(hf_repo)<br>model.eval()<br>video_url = (<br>    &quot;https://huggingface.co/datasets/nateraw/kinetics-mini/&quot;<br>    &quot;resolve/main/val/bowling/-WH-lxmGJVY_000005_000015.mp4&quot;<br>)<br>vr = VideoDecoder(video_url)<br># Sample as many frames as the classifier was trained on (fpc16 -&gt; 16).<br># CAUTION: the official docs snippet writes np.arange(0, frames_per_clip, 8),<br># which for fpc16 evaluates to [0, 8] -&gt; only 2 frames. That under-feeds a<br># 16-frame classifier and quietly degrades predictions. Sample the full clip:<br>total_frames = vr.metadata.num_frames<br>frame_idx = np.linspace(0, total_frames - 1, model.config.frames_per_clip).astype(int)<br>video = vr.get_frames_at(indices=frame_idx).data<br>inputs = processor(video, return_tensors=&quot;pt&quot;).to(model.device)<br>with torch.no_grad():<br>    logits = model(**inputs).logits<br>print(&quot;Top-5 predictions:&quot;)<br>top5 = logits.topk(5).indices[0]<br>probs = torch.softmax(logits, dim=-1).topk(5).values[0]<br>for idx, p in zip(top5, probs):<br>    print(f&quot;  {model.config.id2label[idx.item()]}: {p:.2f}&quot;)</pre><p>This is the whole inference path: decode → process → forward → argmax. One thing to flag, since this is a common trap: the frame-sampling line above is corrected relative to the official docs snippet, which samples only two frames for this checkpoint. The interesting work begins when the labels are your own.</p><h3>5. The fine-tuning philosophy: probe, do not retrain</h3><p>Here is the single most important design decision, and it is different from what most people expect coming from generative video models.</p><p>Meta does <strong>not</strong> fine-tune the encoder end-to-end for downstream understanding. Instead, they freeze the encoder and train a small attentive probe on top of the frozen features <a href="https://arxiv.org/abs/2506.09985">[1].</a> The self-supervised pre-training already produced strong representations; the probe only has to learn how to read them for a specific task.</p><p>The probe architecture, from Appendix C of the paper <a href="https://arxiv.org/abs/2506.09985">[1]</a>, is compact:</p><ul><li>Four transformer blocks, each with 16 attention heads,</li><li>The first three blocks use standard self-attention,</li><li>The final block uses cross-attention with a learnable query token (the paper describes learnable query tokens in general; the transformers pooler instantiates exactly one),</li><li>A residual connection adds the query back after cross-attention,</li><li>A final linear layer produces the class logits.</li></ul><p>In transformers, this is exactly what VJEPA2ForVideoClassification implements: a VJEPA2Model backbone, a VJEPA2AttentivePooler, and an nn.Linear classifier on top. So we do not have to hand-write the probe; we reuse the class and simply freeze the backbone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/996/1*S_NSU_l9zK-Rm00eHeMbAg.png" /><figcaption><strong>Figure 3.</strong> Frozen encoder (grey, no gradient) feeding an attentive pooler and a linear head (coloured, trainable). Only the coloured part is updated during fine-tuning. <em>Where to get it:</em> the components are described in Appendix C of the paper<a href="https://arxiv.org/abs/2506.09985"> [1]</a>, but there is no ready-made figure. Redraw it: a locked encoder block, then three self-attention blocks, then one cross-attention block with a query token, then a linear layer.</figcaption></figure><p>Two practical consequences of freezing:</p><ol><li><strong>Memory.</strong> You do not store optimizer state or activations’ gradients for a 300M–1B parameter backbone. Fine-tuning becomes feasible on a single mid-range GPU.</li><li><strong>Data efficiency.</strong> With a frozen backbone, you are fitting a small head, so you overfit far less on small datasets.</li></ol><p>If you genuinely need to adapt the backbone (for a large domain shift, say), I discuss the parameter-efficient option at the end of Section 6.</p><h3>6. Fine-tuning V-JEPA 2 on your own video labels</h3><p>Now the full worked example. The task: video classification on a custom dataset of .mp4 files, each with an integer label. The recipe is attentive probing on a frozen encoder, following Meta&#39;s protocol.</p><h4>6.1 Build the classification model from a base encoder</h4><p>We load AutoModelForVideoClassification from a base encoder checkpoint (not from a task checkpoint), specify our own number of labels, and let transformers attach a fresh, randomly initialized attentive pooler and classifier head.</p><pre>import torch<br>from transformers import AutoModelForVideoClassification, AutoVideoProcessor<br><br>BACKBONE = &quot;facebook/vjepa2-vitl-fpc64-256&quot;<br>id2label = {0: &quot;class_a&quot;, 1: &quot;class_b&quot;, 2: &quot;class_c&quot;}   # your labels<br>label2id = {v: k for k, v in id2label.items()}<br>processor = AutoVideoProcessor.from_pretrained(BACKBONE)<br>model = AutoModelForVideoClassification.from_pretrained(<br>    BACKBONE,<br>    num_labels=len(id2label),<br>    id2label=id2label,<br>    label2id=label2id,<br>    ignore_mismatched_sizes=True,   # no-op from a base ckpt (the head is just initialized);<br>                                    # only matters when starting from a task head (e.g. ssv2) with a different num_labels<br>)</pre><p>You will see a warning that the pooler and classifier weights were newly initialized. That warning is expected and correct: those are exactly the parameters we are about to train.</p><p>Now freeze the backbone. The backbone lives undermodel.vjepa2; the trainable head is the pooler plus the classifier.</p><pre># Freeze the encoder (and predictor): no gradients for the backbone<br>for param in model.vjepa2.parameters():<br>    param.requires_grad = False<br><br>trainable = [n for n, p in model.named_parameters() if p.requires_grad]<br>print(f&quot;Trainable parameter groups: {len(trainable)}&quot;)<br>print(&quot;Example:&quot;, trainable[:4])<br>n_total = sum(p.numel() for p in model.parameters())<br>n_train = sum(p.numel() for p in model.parameters() if p.requires_grad)<br>print(f&quot;Training {n_train/1e6:.1f}M of {n_total/1e6:.1f}M parameters &quot;<br>      f&quot;({100*n_train/n_total:.1f}%).&quot;)</pre><p>One optional memory note: the classification head runs the backbone withskip_predictor=True, so the JEPA predictor is never exercised for this task. On tight VRAM you can drop it after loading (model.vjepa2.predictor = None) as an inference-time trick. Reattach or reload it beforesave_pretrained, since a None submodule will not serialize cleanly.</p><h4>6.2 The dataset and the one preprocessing trap</h4><p>This is where most people hit a wall. The video processor does not batch cleanly if you hand it a stacked (B, T, C, H, W) tensor; there is an open issue about exactly this failure mode when people try to preprocess a whole batch at once <a href="https://github.com/huggingface/transformers/issues/42251">[10]</a>.</p><p>The robust pattern I use: call the processor per-sample inside__getitem__, return the ready pixel_values_videos for one clip, and stack them in a custom collate function. The processor then only ever sees a single video, which is the case it handles reliably.</p><pre>import numpy as np<br>from torch.utils.data import Dataset, DataLoader<br>from torchcodec.decoders import VideoDecoder<br><br>def sample_indices(num_frames_total, num_frames_wanted, frame_step):<br>    &quot;&quot;&quot;Uniform sampling with a fixed step, clamped to the clip length.&quot;&quot;&quot;<br>    span = num_frames_wanted * frame_step<br>    if num_frames_total &lt;= span:<br>        idx = np.linspace(0, max(num_frames_total - 1, 0),<br>                          num_frames_wanted).astype(np.int64)<br>    else:<br>        start = np.random.randint(0, num_frames_total - span + 1)<br>        idx = start + np.arange(0, span, frame_step)<br>    return idx[:num_frames_wanted]<br><br>class VideoClsDataset(Dataset):<br>    def __init__(self, samples, processor, num_frames=16, frame_step=4):<br>        # samples: list of (path_to_mp4, integer_label)<br>        self.samples = samples<br>        self.processor = processor<br>        self.num_frames = num_frames<br>        self.frame_step = frame_step<br>    def __len__(self):<br>        return len(self.samples)<br>    def __getitem__(self, i):<br>        path, label = self.samples[i]<br>        vr = VideoDecoder(path)<br>        total = vr.metadata.num_frames<br>        idx = sample_indices(total, self.num_frames, self.frame_step)<br>        video = vr.get_frames_at(indices=idx).data          # T x C x H x W, uint8<br>        proc = self.processor(video, return_tensors=&quot;pt&quot;)   # single clip only<br>        pixel_values = proc[&quot;pixel_values_videos&quot;][0]       # drop the batch dim<br>        return pixel_values, torch.tensor(label, dtype=torch.long)<br><br>def collate(batch):<br>    videos = torch.stack([b[0] for b in batch], dim=0)<br>    labels = torch.stack([b[1] for b in batch], dim=0)<br>    return {&quot;pixel_values_videos&quot;: videos, &quot;labels&quot;: labels}<br></pre><p>Sampling only 16 frames here, rather than the 64 the encoder was pre-trained with, is deliberate. It keeps the token count and the memory footprint low, and because frames_per_clip does not constrain inference, the encoder handles it without complaint.</p><h4>6.3 The training loop</h4><p>A plain PyTorch loop with mixed precision. Because the backbone is frozen, we run it under torch.no_grad()-like conditions automatically (no grad is stored for frozen params), and the optimizer only touches the probe.</p><pre>from torch.optim import AdamW<br><br>device = &quot;cuda&quot;<br>model.to(device)<br># Replace with your real split<br>train_samples = [(&quot;path/to/clip_0.mp4&quot;, 0), (&quot;path/to/clip_1.mp4&quot;, 1)]  # ...<br>val_samples   = [(&quot;path/to/val_0.mp4&quot;, 2)]                              # ...<br>train_ds = VideoClsDataset(train_samples, processor, num_frames=16, frame_step=4)<br>val_ds   = VideoClsDataset(val_samples,   processor, num_frames=16, frame_step=4)<br>train_loader = DataLoader(train_ds, batch_size=4, shuffle=True,<br>                          num_workers=4, collate_fn=collate)<br>val_loader   = DataLoader(val_ds, batch_size=4, shuffle=False,<br>                          num_workers=4, collate_fn=collate)<br># Optimize only the parameters that require grad (pooler + classifier)<br>optim = AdamW(<br>    [p for p in model.parameters() if p.requires_grad],<br>    lr=1e-3, weight_decay=0.05,<br>)<br>EPOCHS = 20   # the paper uses ~20 epochs for probe training on most tasks<br>for epoch in range(EPOCHS):<br>    model.train()<br>    # keep the frozen backbone in eval mode so its norm layers stay fixed<br>    model.vjepa2.eval()<br>    running = 0.0<br>    for step, batch in enumerate(train_loader):<br>        batch = {k: v.to(device) for k, v in batch.items()}<br>        optim.zero_grad(set_to_none=True)<br>        with torch.autocast(device_type=&quot;cuda&quot;, dtype=torch.bfloat16):<br>            out = model(**batch)          # loss is computed because labels are passed<br>            loss = out.loss<br>        loss.backward()   # bf16 has fp32 exponent range, so no GradScaler is needed<br>        optim.step()<br>        running += loss.item()<br>        if step % 20 == 0:<br>            print(f&quot;epoch {epoch} step {step} loss {loss.item():.4f}&quot;)<br>    # simple validation accuracy<br>    model.eval()<br>    correct = total = 0<br>    with torch.no_grad():<br>        for batch in val_loader:<br>            batch = {k: v.to(device) for k, v in batch.items()}<br>            logits = model(pixel_values_videos=batch[&quot;pixel_values_videos&quot;]).logits<br>            preds = logits.argmax(-1)<br>            correct += (preds == batch[&quot;labels&quot;]).sum().item()<br>            total += batch[&quot;labels&quot;].numel()<br>    print(f&quot;epoch {epoch}: train_loss={running/len(train_loader):.4f} &quot;<br>          f&quot;val_acc={correct/max(total,1):.3f}&quot;)</pre><p>A few notes from experience:</p><ul><li>Keep the backbone in eval() even during training (model.vjepa2.eval()), so its normalization statistics do not drift. Only the probe should be in training mode.</li><li>The classifier learning rate is the single most sensitive hyperparameter. The paper sweeps values roughly in the range 1e-4 to 5e-3 with weight decay between 0.01 and 0.8<a href="https://arxiv.org/abs/2506.09985"> [1].</a> Start at 1e-3 and sweep from there.</li><li>Because we pass labels in the batch, VJEPA2ForVideoClassification computes the cross-entropy loss internally and returns it as out.loss. You do not build the loss yourself.</li></ul><h4>6.4 Saving and reloading</h4><pre>model.save_pretrained(&quot;./vjepa2-vitl-myprobe&quot;)<br>processor.save_pretrained(&quot;./vjepa2-vitl-myprobe&quot;)<br><br># later<br>reload = AutoModelForVideoClassification.from_pretrained(&quot;./vjepa2-vitl-myprobe&quot;)</pre><h4>6.5 Parameter-efficient fine-tuning: LoRA, QLoRA, DoRA</h4><p>The frozen probe is the right default, and I want to be explicit that this is not a lazy choice: it is the exact protocol Meta uses to report their state-of-the-art numbers <a href="https://arxiv.org/abs/2506.09985">[1]</a>, because the self-supervised pre-training already did the heavy lifting. So before adding adapters, confirm the probe has genuinely plateaued; it is a strong baseline and should be your point of comparison.</p><p>That said, when there is a real domain shift (egocentric surgery footage, synthetic renders, a sensor the encoder never saw during pre-training), the frozen features leave accuracy on the table, and you want the backbone to move cheaply. This is where LoRA and its variants belong. The mental model is the one you already use for diffusion transformers: freeze the base weights, inject low-rank adapters into the attention projections, train only those.</p><p>There is one V-JEPA 2-specific subtlety that trips people up. In an LLM, you attach LoRA around a head that already exists. Here the classification head does not exist in the base checkpoint: the attentive pooler and the linear classifier are freshly initialized. So the correct configuration is LoRA on the backbone plus a fully-trained head, expressed through modules_to_save. Forget this and your randomly initialized probe stays random; nothing learns.</p><p>I checked the module names against the transformers source so the targets below actually match. The encoder attention (VJEPA2RopeAttention) exposes query, key, value, and proj. The catch is that the <strong>predictor</strong> is built from the sameVJEPA2Layer, so vjepa2.predictor.layer.N.attention.{query,key,value} exist as well. A bare target_modules=[&quot;query&quot;, &quot;key&quot;, &quot;value&quot;] would therefore attach adapters to the predictor too, and because the classification head runs the backbone withskip_predictor=True, those predictor adapters receive no gradients: inflated print_trainable_parameters() counts, wasted memory, and &quot;unused parameter&quot; errors under DDP. The fix is to scope the match to the <strong>encoder</strong> with a regex, as below. The attentive pooler is safe either way, since it uses q_proj/k_proj/v_proj, not query/key/value.</p><pre>pip install -U peft bitsandbytes</pre><pre>import torch<br>from transformers import AutoModelForVideoClassification, BitsAndBytesConfig<br>from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training<br><br>BACKBONE = &quot;facebook/vjepa2-vitg-fpc64-256&quot;   # the 1B encoder is where PEFT earns its keep<br># (optional) 4-bit backbone -&gt; this makes it QLoRA. Drop the two 4-bit lines<br># below for ordinary (bf16) LoRA.<br>bnb = BitsAndBytesConfig(<br>    load_in_4bit=True,<br>    bnb_4bit_quant_type=&quot;nf4&quot;,<br>    bnb_4bit_compute_dtype=torch.bfloat16,<br>    bnb_4bit_use_double_quant=True,<br>)<br>model = AutoModelForVideoClassification.from_pretrained(<br>    BACKBONE,<br>    num_labels=len(id2label),<br>    id2label=id2label,<br>    label2id=label2id,<br>    ignore_mismatched_sizes=True,<br>    quantization_config=bnb,          # remove for non-quantized LoRA<br>    device_map=&quot;auto&quot;,<br>)<br>model = prepare_model_for_kbit_training(model)   # only needed in the 4-bit case<br>lora = LoraConfig(<br>    r=16,<br>    lora_alpha=32,<br>    lora_dropout=0.05,<br>    bias=&quot;none&quot;,<br>    # scope to the ENCODER only: the predictor reuses the same layer class, so a<br>    # bare [&quot;query&quot;,&quot;key&quot;,&quot;value&quot;] would also (wrongly) adapt vjepa2.predictor<br>    target_modules=r&quot;.*vjepa2\.encoder\.layer\.\d+\.attention\.(query|key|value)&quot;,<br>    # the head is freshly initialised and MUST be trained in full, not LoRA-adapted<br>    modules_to_save=[&quot;pooler&quot;, &quot;classifier&quot;],<br>    use_rslora=True,     # optional: rank-stabilised scaling, steadier at higher r<br>    # use_dora=True,     # optional: DoRA, usually better at low rank<br>)<br>model = get_peft_model(model, lora)<br>model.print_trainable_parameters()</pre><p>To also adapt the attention output projection and the MLP, widen the same encoder-scoped regex (scoping still matters, because the bare name proj is shared with the predictor and the patch-embedding Conv3d):</p><pre>target_modules = (<br>    r&quot;.*vjepa2\.encoder\.layer\.\d+\.&quot;<br>    r&quot;(attention\.(query|key|value|proj)|mlp\.(fc1|fc2))&quot;<br>)</pre><p>Extra knobs, each one line in LoraConfig:</p><ul><li>use_rslora=True: rank-stabilized LoRA, more stable when you push r higher.</li><li>use_dora=True: DoRA, which splits the update into magnitude and direction and usually beats plain LoRA at low rank, at a small speed cost.</li><li>Tighten the regex to the later blocks only (e.g. layers 17–23 for ViT-L, the ones Meta reads from <a href="https://arxiv.org/abs/2506.09985">[1]</a>) if you want to adapt even less.</li></ul><p><strong>Memory.</strong> For the ViT-g (1B) encoder on a single GPU, the block above is already the QLoRA recipe: 4-bit backbone viaBitsAndBytesConfig, then adapters on top. Add gradient checkpointing; the V-JEPA 2 encoder layers subclassGradientCheckpointingLayer, so it is supported:</p><pre>model.gradient_checkpointing_enable()<br>model.enable_input_require_grads()   # required: frozen inputs + gradient checkpointing</pre><p><strong>Training.</strong> The loop from Section 6.3 is unchanged. PEFT has already set requires_grad for you, so AdamW([p for p in model.parameters() if p.requires_grad], lr=2e-4) trains the adapters and the head together. Adapters like a slightly higher learning rate than a bare probe, so start around 2e-4. To be precise, put the head (pooler, classifier) and the LoRA adapters in separate parameter groups with their own learning rates.</p><p>Saving writes only the adapters and the trained head, so checkpoints are a few megabytes rather than gigabytes:</p><pre>model.save_pretrained(&quot;./vjepa2-vitg-lora&quot;)   # adapter + head only</pre><p><strong>Decision summary</strong>, in the order I actually try things:</p><ol><li><strong>Frozen encoder + attentive probe</strong>: always start here; often already competitive with the state of the art.</li><li><strong>LoRA / DoRA on the backbone + fully-trained head</strong>: when the probe plateaus or the domain shifts.</li><li><strong>QLoRA (4-bit backbone + LoRA)</strong>: the same as (2) for the ViT-g on limited VRAM.</li><li><strong>Full fine-tuning</strong>: only with a large labelled dataset; otherwise you overwrite the pre-training and lose the very thing that made V-JEPA 2 worth using.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/841/1*-_twrk3nn8j4vmrI7XRAzw.png" /><figcaption><strong>Figure 4.</strong> V-JEPA 2 downstream results for the largest (ViT-g) configuration: 77.3 top-1 on Something-Something v2 and 39.7 recall-at-5 on Epic-Kitchens-100 action anticipation <a href="https://arxiv.org/abs/2506.09985">[1]</a>[<a href="https://ai.meta.com/blog/v-jepa-2-world-model-benchmarks/">2]</a>. The ViT-L encoder used in the code examples here scores somewhat lower; see the per-model results table in the repository<a href="https://github.com/facebookresearch/vjepa2"> [4]</a></figcaption></figure><h3>7. The world model: V-JEPA 2-AC and planning</h3><p>Everything above treats V-JEPA 2 as a video encoder. But the headline capability of V-JEPA 2 is that it is a world model: after pre-training, Meta post-trains an action-conditioned predictor, V-JEPA 2-AC, on 62 hours of unlabeled robot trajectories from the DROID dataset <a href="https://arxiv.org/abs/2506.09985">[1]</a><a href="https://arxiv.org/abs/2403.12945">[8]</a>. The encoder is frozen; only a new action-conditioned predictor is learned.</p><p>At deployment, the model plans by prediction. Given a goal image, the predictor imagines the latent consequence of candidate actions, scores each by how close the predicted latent is to the goal latent (an energy), and executes the best action in a model-predictive-control loop, replanning at every step <a href="https://ai.meta.com/research/publications/revisiting-feature-prediction-for-learning-visual-representations-from-video/">[7]</a>. This is what lets it perform reach, grasp, and pick-and-place on robot arms it has never seen, with only a monocular RGB camera and no task-specific reward <a href="https://arxiv.org/abs/2506.09985">[1]</a><a href="https://ai.meta.com/blog/v-jepa-2-world-model-benchmarks/">[2]</a>.</p><p>The AC checkpoint is trained from the ViT-g encoder and is loaded through torch.hub from Meta&#39;s repository rather than through transformers <a href="https://github.com/facebookresearch/vjepa2">[4]</a>:</p><pre>import torch<br>encoder, ac_predictor = torch.hub.load(<br>    &quot;facebookresearch/vjepa2&quot;, &quot;vjepa2_ac_vit_giant&quot;<br>)</pre><p>For the actual planning and energy computation, follow energy_landscape_example.ipynb in the official repository<a href="https://github.com/facebookresearch/vjepa2"> [4]</a>; the cost function and the sampling of candidate actions are the part that matters, and it is easier to read their notebook than to reproduce it here. I treat this as an advanced appendix rather than a core step, because most readers will use V-JEPA 2 as an encoder first.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TOJBCG7GtYdhq7XBn_Wppw.png" /><figcaption><strong>Figure 5.</strong> Model-predictive control with V-JEPA 2-AC: from the current observation and a goal image, the predictor rolls out candidate actions in latent space and picks the lowest-energy one.</figcaption></figure><h3>8. A note on V-JEPA 2.1</h3><p>Note that Meta released an improved recipe, V-JEPA 2.1 <a href="https://huggingface.co/papers/2603.14482">[11]</a>, focused on higher-quality and temporally consistent dense features <a href="https://github.com/facebookresearch/vjepa2">[4].</a> It introduces a dense predictive loss where both visible and masked tokens contribute to the objective, deep self-supervision at several intermediate layers, and multi-modal tokenizers. It also ships new encoder sizes: ViT-B (~80M) and ViT-G (~2B) alongside ViT-L and ViT-g, all at 384 resolution. If your downstream task is dense (segmentation-like or tracking-like), it is worth trying a 2.1 encoder before committing to the original 2.0 one.</p><p>One practical point that is easy to get wrong: the loading code is not the same as for 2.0. V-JEPA 2.1 changes the architecture (a corrected RoPE, learnable modality embeddings, hierarchical feature extraction with per-layer norms, a separate image patch embedding), and the VJEPA2Model in transformers does not support these. As of this writing, the 2.1 checkpoints load only through Meta&#39;s torch.hub interface; there is an open feature request for transformers support (issue #45496), and the only Hub-hosted weights are unofficial community conversions loaded with trust_remote_code=True.</p><pre>import torch<br># V-JEPA 2.1 loads through torch.hub, NOT through transformers VJEPA2Model<br>encoder = torch.hub.load(&quot;facebookresearch/vjepa2&quot;, &quot;vjepa2_1_vit_large_384&quot;)<br># other entrypoints: vjepa2_1_vit_base_384, vjepa2_1_vit_giant_384, vjepa2_1_vit_gigantic_384</pre><h3>9. Common failure modes, in one place</h3><p>To save you the debugging I already did:</p><ul><li>torchcodec cannot open a file. FFmpeg is missing or too old. Install it at the system level first.</li><li>Processor crashes on a batch tensor. Do not feed the processor a (B, T, C, H, W) stack; process one clip at a time in the dataset and stack afterwards, as in Section 6.2 <a href="https://github.com/huggingface/transformers/issues/42251">[10]</a>.</li><li>Out-of-memory during fine-tuning. Reduce the number of sampled frames before reducing batch size; token count grows with frames, and 16 frames is usually enough for a probe.</li><li>The probe does not learn. Confirm the backbone is actually frozen (requires_grad == False) and in eval() mode, then sweep the classifier learning rate; it is by far the most sensitive knob.</li></ul><h3>Final Thoughts</h3><p>The pattern to remember is simple: V-JEPA 2 gives you a strong frozen representation, and you learn a light attentive probe on top of it. That is the whole reason the model is practical: you get most of the value of a 1B-parameter video model without paying to fine-tune one. Start with feature extraction, confirm the shapes make sense, then train a probe on your labels. Only reach for backbone adaptation, or for the action-conditioned world model, once the frozen baseline has told you what it can already do.</p><p>If you build something on top of this, I would like to hear about it.</p><h3>References</h3><p>[1] M. Assran et al., “V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning,” arXiv:2506.09985, 2025. <a href="https://arxiv.org/abs/2506.09985">https://arxiv.org/abs/2506.09985</a></p><p>[2] Meta AI, “Introducing the V-JEPA 2 world model and new benchmarks for physical reasoning,” June 2025. <a href="https://ai.meta.com/blog/v-jepa-2-world-model-benchmarks/">https://ai.meta.com/blog/v-jepa-2-world-model-benchmarks/</a></p><p>[3] Hugging Face, “V-JEPA 2, Transformers documentation.” <a href="https://huggingface.co/docs/transformers/en/model_doc/vjepa2">https://huggingface.co/docs/transformers/en/model_doc/vjepa2</a></p><p>[4] facebookresearch/vjepa2, official PyTorch code and models. <a href="https://github.com/facebookresearch/vjepa2">https://github.com/facebookresearch/vjepa2</a></p><p>[5] Model card, facebook/vjepa2-vitl-fpc64-256. <a href="https://huggingface.co/facebook/vjepa2-vitl-fpc64-256">https://huggingface.co/facebook/vjepa2-vitl-fpc64-256</a></p><p>[6] V-JEPA 2 collection on the Hugging Face Hub. <a href="https://huggingface.co/collections/facebook/v-jepa-2-6841bad8413014e185b497a6">https://huggingface.co/collections/facebook/v-jepa-2-6841bad8413014e185b497a6</a></p><p>[7] A. Bardes et al., “Revisiting Feature Prediction for Learning Visual Representations from Video” (V-JEPA), 2024. <a href="https://ai.meta.com/research/publications/revisiting-feature-prediction-for-learning-visual-representations-from-video/">https://ai.meta.com/research/publications/revisiting-feature-prediction-for-learning-visual-representations-from-video/</a></p><p>[8] A. Khazatsky et al., “DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset,” arXiv:2403.12945, 2024. <a href="https://arxiv.org/abs/2403.12945">https://arxiv.org/abs/2403.12945</a></p><p>[9] torchcodec, PyTorch video decoding library. <a href="https://github.com/pytorch/torchcodec">https://github.com/pytorch/torchcodec</a></p><p>[10] “Finetuning issue with V-JEPA 2,” huggingface/transformers issue #42251. <a href="https://github.com/huggingface/transformers/issues/42251">https://github.com/huggingface/transformers/issues/42251</a></p><p>[11] L. Mur-Labadia et al., “V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning,” 2026. <a href="https://huggingface.co/papers/2603.14482">https://huggingface.co/papers/2603.14482</a></p><p>[12] “Add V-JEPA 2.1 inference support,” huggingface/transformers issue #45496. <a href="https://github.com/huggingface/transformers/issues/45496">https://github.com/huggingface/transformers/issues/45496</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85aafe60fb0a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[15. Can a Language Model Hear? A Technical Tour of Audio & Music Understanding LLMs]]></title>
            <link>https://medium.com/@aminfadaeinejad.edu/15-can-a-language-model-hear-a-technical-tour-of-audio-music-understanding-llms-826fdd8a13c6?source=rss-aff4faf055a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/826fdd8a13c6</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Amin Fadaeinejad]]></dc:creator>
            <pubDate>Sun, 28 Jun 2026 07:16:43 GMT</pubDate>
            <atom:updated>2026-06-28T07:16:43.094Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>A hands-on comparison of five open audio-language models such as Qwen2-Audio, Audio Flamingo 3, Gemma 4, MOSS-Audio, and MU-LLaMA on the exact same task: listen to a piece of music and describe it. Every model below ships with runnable code in </em><a href="https://github.com/aminfadaei116/published-tutorials/tree/main/Music-Understanding"><em>here</em></a><em> so you can reproduce the results yourself.</em></blockquote><h4>Why “Audio LLMs” are interesting</h4><p>Large language models reason over text. But a huge amount of human information, speech, music, and ambient sound, never gets written down. An audio-language model (audio LLM) bolts an ear onto an LLM: it takes raw waveform and text as input and produces text as output. Ask it “What instruments do you hear?” or “Transcribe this and summarize the mood,” and it answers in natural language.</p><p>Music is the hardest stress test for these models. Speech recognition mostly needs to map sound to words. Music understanding needs the model to reason about simultaneous events, multiple instruments, harmony, rhythm, timbre, production, and verbalize abstract perceptual qualities. It’s where the differences between architectures show up most.</p><p>This post walks through five open models that approach the problem in three fundamentally different ways, gives you sample code and then puts them head-to-head on the same audio clips.</p><h3>The three architectures</h3><h4>1. Encoder → adapter → frozen LLM</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*M5bQ1YOSguDc2xT2EcqTVA.png" /></figure><p>The dominant pattern. A pretrained audio encoder (often a Whisper or a self-supervised music encoder like MERT) turns the waveform into a sequence of embeddings. A small trainable adapter/projection maps those embeddings into the LLM’s token-embedding space, where they’re spliced in alongside the text tokens. The LLM itself is frozen or lightly fine-tuned. Cheap to train, modular, but capped by the encoder’s resolution.</p><blockquote><em>Examples here: Qwen2-Audio, Audio Flamingo 3, MU-LLaMA.</em></blockquote><h4>2. Natively multimodal (encoder-free / unified)</h4><p>No separate bolted-on encoder. Audio (and image, and video) is tokenized and fed through the same transformer stack that handles text, trained jointly from early on. One set of weights does everything. Tighter integration, but audio quality is bound by what a general-purpose model can spare for it.</p><blockquote><em>Example here: Gemma 4.</em></blockquote><h4>3. Discrete audio tokens</h4><p>The waveform is quantized into discrete acoustic tokens (think “words for sound”) by a neural audio codec, and the LLM treats them as just more vocabulary. This makes the model symmetric: it can both <em>read</em> audio tokens (understanding) and <em>write</em> them (generation/speech synthesis). Powerful and unifying, but the tokenizer is now part of your fidelity budget.</p><blockquote><em>Example here: MOSS-Audio.</em></blockquote><p>Keep these three methods in mind, they explain most of the behavioural differences you’ll see at the end.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MUXSgzIt3N173PCu4prepA.png" /></figure><p>All five take a waveform plus a text question and return text. What differs is how they hear, how fast they run, and how much they hallucinate. Let’s run each one.</p><h3>1. Qwen2-Audio (The reliable generalist)</h3><p>Alibaba’s Qwen2-Audio-7B-Instruct pairs a Whisper-style audio encoder with the Qwen2 7B LLM via an adapter. It was trained on a broad mix of speech, sound, and music, and it works pretty well in my personal experience.</p><p>One thing to keep in mind you must decode the audio yourself with librosa at 16 kHz mono and hand the samples to the processor the audio_url In the chat template, there is only a placeholder. In transformers 5.x the keyword is audio= (it used to be audios=).</p><pre>import librosa, torch<br>from transformers import AutoProcessor, Qwen2AudioForConditionalGeneration<br><br>MODEL_ID = &quot;Qwen/Qwen2-Audio-7B-Instruct&quot;<br>processor = AutoProcessor.from_pretrained(MODEL_ID)<br>model = Qwen2AudioForConditionalGeneration.from_pretrained(<br>    MODEL_ID, torch_dtype=torch.bfloat16, device_map=&quot;cuda:0&quot;).eval()<br><br>waveform, _ = librosa.load(&quot;song.wav&quot;, sr=16000, mono=True)   # decode yourself!<br>conversation = [{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: [<br>    {&quot;type&quot;: &quot;audio&quot;, &quot;audio_url&quot;: &quot;song.wav&quot;},               # placeholder only<br>    {&quot;type&quot;: &quot;text&quot;, &quot;text&quot;: &quot;What instruments do you hear?&quot;},<br>]}]<br>text = processor.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False)<br>inputs = processor(text=text, audio=[waveform], sampling_rate=16000,<br>                   return_tensors=&quot;pt&quot;, padding=True).to(model.device)<br>out = model.generate(**inputs, max_new_tokens=512)<br>print(processor.batch_decode(out[:, inputs.input_ids.size(1):], skip_special_tokens=True)[0])</pre><p>▶ Full script: <a href="https://github.com/aminfadaei116/published-tutorials/blob/main/Music-Understanding/code/qwen2_audio.py">code/qwen2_audio.py</a> — run with python qwen2_audio.py --audio song.wav -q &quot;your question&quot;.</p><h3>2. Audio Flamingo 3 (Fastest of them all)</h3><p>NVIDIA’s audio-flamingo-3-hf uses an AF-Whisper encoder feeding the LLM through Flamingo-style cross-attention. It&#39;s built for long audio (up to ~10 minutes, internally chunked into 30s windows) and, as you&#39;ll see, it&#39;s dramatically faster than everything else here.</p><p>AF3 Loads it in float32, not bfloat16. The audio tower keeps some submodules (LayerNorm) in fp32, so a bf16 load throws dtype-mismatch errors. The upside: the processor reads the file straight from the path field, no manual librosa decode needed.</p><pre>import torch<br>from transformers import AudioFlamingo3ForConditionalGeneration, AutoProcessor<br><br>MODEL_ID = &quot;nvidia/audio-flamingo-3-hf&quot;<br>processor = AutoProcessor.from_pretrained(MODEL_ID)<br>model = AudioFlamingo3ForConditionalGeneration.from_pretrained(<br>    MODEL_ID, torch_dtype=torch.float32, device_map=&quot;cuda:0&quot;).eval()   # fp32!<br>conversation = [{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: [<br>    {&quot;type&quot;: &quot;text&quot;, &quot;text&quot;: &quot;Describe the genre, tempo, and mood.&quot;},<br>    {&quot;type&quot;: &quot;audio&quot;, &quot;path&quot;: &quot;song.wav&quot;},                              # processor loads it<br>]}]<br>inputs = processor.apply_chat_template(<br>    conversation, tokenize=True, add_generation_prompt=True,<br>    return_dict=True, return_tensors=&quot;pt&quot;).to(model.device)<br>out = model.generate(**inputs, max_new_tokens=512)<br>print(processor.batch_decode(out[:, inputs[&quot;input_ids&quot;].shape[1]:], skip_special_tokens=True)[0])</pre><p>▶ Full script: <a href="https://github.com/aminfadaei116/published-tutorials/blob/main/Music-Understanding/code/audio_flamingo3.py">code/audio_flamingo3.py</a>.</p><h3>3. Gemma 4 (Encoder Free)</h3><p>Google’s gemma-4-E4B-it is natively multimodal text, image, audio, and video, all flow through one encoder-free transformer. There&#39;s no separate audio tower; it&#39;s the same weights that do everything. Audio understanding is a side effect of a general model, which makes it a fascinating point of comparison.</p><p>Two things to keep in mind: (1) audio is capped at ~30 seconds; trim longer clips first; (2) Gemma wants the text before the audio in the content list (the opposite of Qwen2-Audio). It’s also gated: accept the license and huggingface-cli login, and install pillow + torchvision (vision deps of the unified model) even though you&#39;re only using audio.</p><pre>import torch<br>from transformers import AutoProcessor, Gemma4ForConditionalGeneration<br><br>MODEL_ID = &quot;google/gemma-4-E4B-it&quot;<br>processor = AutoProcessor.from_pretrained(MODEL_ID)<br>model = Gemma4ForConditionalGeneration.from_pretrained(<br>    MODEL_ID, dtype=&quot;auto&quot;, device_map=&quot;cuda:0&quot;).eval()<br>messages = [{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: [<br>    {&quot;type&quot;: &quot;text&quot;, &quot;text&quot;: &quot;What instruments do you hear?&quot;},   # text FIRST<br>    {&quot;type&quot;: &quot;audio&quot;, &quot;audio&quot;: &quot;song_30s.wav&quot;},                  # &lt;= 30 seconds<br>]}]<br>inputs = processor.apply_chat_template(<br>    messages, tokenize=True, add_generation_prompt=True,<br>    return_dict=True, return_tensors=&quot;pt&quot;).to(model.device)<br>n = inputs[&quot;input_ids&quot;].shape[-1]<br>out = model.generate(**inputs, max_new_tokens=512)<br>print(processor.batch_decode(out[:, n:], skip_special_tokens=True)[0])</pre><p>▶ Full script: <a href="https://github.com/aminfadaei116/published-tutorials/blob/main/Music-Understanding/code/gemma4_audio.py">code/gemma4_audio.py</a>.</p><h3>4. MOSS-Audio</h3><p>OpenMOSS’s MOSS-Audio-8B-Instruct represents audio as discrete tokens from a neural codec, so the same model can both understand <em>and</em> generate audio. It&#39;s not a plainAutoModel: it ships custom modelling code in its GitHub repo and loads weights from a local directory with trust_remote_code=True.</p><pre>import sys, torch<br>from src.modeling_moss_audio import MossAudioModel<br>from src.processing_moss_audio import MossAudioProcessor<br><br>MODEL_PATH = &quot;third_party/MOSS-Audio/weights/MOSS-Audio-8B-Instruct&quot;   # local!<br>model = MossAudioModel.from_pretrained(<br>    MODEL_PATH, trust_remote_code=True, dtype=&quot;auto&quot;, device_map=&quot;cuda:0&quot;).eval()<br>processor = MossAudioProcessor.from_pretrained(<br>    MODEL_PATH, trust_remote_code=True, enable_time_marker=True)<br># load mono audio at the processor&#39;s mel sample rate (see full script for soundfile loader)<br>inputs = processor(text=&quot;What instruments do you hear?&quot;,<br>                   audios=[raw_audio], return_tensors=&quot;pt&quot;).to(model.device)<br>inputs[&quot;audio_data&quot;] = inputs[&quot;audio_data&quot;].to(model.dtype)<br>inputs[&quot;audio_input_mask&quot;] = inputs[&quot;input_ids&quot;] == processor.audio_token_id<br>out = model.generate(**inputs, max_new_tokens=1024, do_sample=True,<br>                     temperature=1.0, top_p=1.0, top_k=50, use_cache=True)<br>print(processor.decode(out[0, inputs[&quot;input_ids&quot;].shape[1]:], skip_special_tokens=True))</pre><p>▶ Full script (incl. the soundfile loader): <a href="https://github.com/aminfadaei116/published-tutorials/blob/main/Music-Understanding/code/moss_audio.py">code/moss_audio.py</a>.</p><h3>5. MU-LLaMA (Music Specialist)</h3><p>The only model here is built specifically for music. MU-LLaMA freezes a MERT-v1–330M music encoder (a self-supervised model trained on music, running at 24 kHz) and adapts its features into LLaMA-7B. In principle, the most “music-native” encoder of the bunch.</p><p>Tips: It’s a research repo, not a Hub model.</p><pre>import llama<br><br>model = llama.load(&quot;mu_ckpts/checkpoint.pth&quot;, &quot;mu_ckpts/LLaMA&quot;,<br>                   mert_path=&quot;m-a-p/MERT-v1-330M&quot;, knn=False,<br>                   llama_type=&quot;7B&quot;, device=&quot;cuda:0&quot;).eval()<br>wav, sr = torchaudio.load(&quot;song.wav&quot;)                      # MERT wants 24 kHz mono<br>wav = torchaudio.functional.resample(wav.mean(0, keepdim=True), sr, 24000)<br>inputs = {&quot;Audio&quot;: [torch.stack([wav.squeeze(0)], dim=0), 1]}<br>prompts = [model.tokenizer.encode(llama.format_prompt(&quot;What instruments do you hear?&quot;),<br>                                  bos=True, eos=False)]<br>with torch.cuda.amp.autocast():<br>    out = model.generate(inputs, prompts, max_gen_len=512, temperature=0.6, top_p=0.8,<br>                         cache_size=100, cache_t=20.0, cache_weight=0.0)<br>print(out[0].strip())</pre><p>▶ Full script: <a href="https://github.com/aminfadaei116/published-tutorials/blob/main/Music-Understanding/code/mu_llama.py">code/mu_llama.py</a>.</p><h3>Head-to-head: the benchmark</h3><p>We ran all five models with the same prompt (a detailed 8-point “Expert Music Analyst” brief, see <a href="https://github.com/aminfadaei116/published-tutorials/blob/main/Music-Understanding/code/qwen2_audio.py">code/qwen2_audio.py</a> the <a href="https://github.com/aminfadaei116/published-tutorials/blob/main/Music-Understanding/results/full_report.md">full report</a>) over 10 clips spanning 10 GTZAN genres (blues, classical, country, disco, hip-hop, jazz, metal, pop, reggae, rock) on a single GPU. The raw per-clip outputs are in <a href="https://github.com/aminfadaei116/published-tutorials/blob/main/Music-Understanding/results">results/</a>.</p><h3>Speed &amp; accuracy</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TEmcRp8S-wUwm1phfboZOg.png" /></figure><p>(Genre accuracy = did the model’s free-text description name the right genre or a synonym. Latency includes the generation of a full multi-paragraph analysis; it scales with how much each model chooses to write.)</p><p>Two main results:</p><ul><li>Audio Flamingo 3 is the sweet spot. Good genre accuracy and faster than the other strong models.</li><li>The music-specialist encoder didn’t win. MU-LLaMA is fast, but its output is often wrong on genre, a reminder that a great encoder (MERT) doesn’t help if the adapter + LLM can’t verbalize what it heard.</li></ul><h3>Inference Test</h3><p>Nothing illustrates the differences more than one audio clip across all five. Here’s a blues track, prompt: <em>“give a detailed structured analysis.”</em></p><blockquote><em>Qwen2-Audio </em><strong><em>(33s)</em></strong><em>: “Features a prominent guitar solo with accompaniment from acoustic rhythm guitar and possibly harmonica… The genre is blues, specifically Chicago blues… tempo around 109 BPM… in the key of F minor… The production quality is low, indicating it may be an amateur or vintage recording.” (Thorough, hedged, mostly right — and note it commits to BPM/key.)</em></blockquote><blockquote><em>Audio Flamingo 3 </em><strong><em>(3s):</em></strong><em> “A blues track featuring a soulful male vocalist… guitar, piano, and drums… moderate 4/4… key of F minor… raw, authentic quality characteristic of traditional blues.” (Concise, correct, fast.)</em></blockquote><blockquote><em>Gemma 4 </em><strong><em>(36s):</em></strong><em> “…highly polished, modern-sounding track rooted in late 1990s/early 2000s R&amp;B/Pop… programmed beats… synthesized bass…” (Confident, detailed, fluent prose — and wrong: it heard R&amp;B, not blues. The encoder-free model writes beautifully but mishears.)</em></blockquote><blockquote><em>MOSS-Audio </em><strong><em>(11s)</em></strong><em>: “A high-energy instrumental blues-rock piece… warm overdriven electric guitar riff… shuffle drum pattern… ~147 BPM in 4/4… major-key tonality.” (Vivid and structured — but invents a specific artist attribution, a classic discrete-token hallucination.)</em></blockquote><blockquote><em>MU-LLaMA </em><strong><em>(2s):</em></strong><em> “The music is described as having a jazzy feel, with a strong emphasis on the trumpet and saxophone.” (One sentence, and wrong on both genre and instruments.)</em></blockquote><p>The pattern repeats on other clips.</p><h3>What this tells us</h3><p>(A) Architecture predicts personality:</p><ul><li><strong>Encoder + adapter (Qwen2-Audio, AF3): </strong>Models are the most reliable music listeners — a dedicated audio encoder is hard to beat for <em>perception</em>.</li><li><strong>Encoder-free native (Gemma 4):</strong> Writes the most fluent, well-structured prose but is the most likely to confidently hallucinate — its audio fidelity is bounded by a general-purpose model’s spare capacity.</li><li><strong>Discrete-token (MOSS-Audio) models:</strong> Are vivid and great for generation-style tasks, but quantization can invent plausible-sounding details (artist names, specific BPMs) that aren’t there.</li></ul><p>(B) A specialist encoder is necessary but not sufficient. MU-LLaMA has the most music-specific front-end (MERT) yet the weakest descriptions the adapter and LLM matter as much as the ears.</p><p>(C) Confidence ≠ correctness. Every model states the BPM, key, and era with absolute certainty. Some are right, some are fabrications. For any real application, verify the structured claims (BPM, key) with a signal-processing tool; trust the models for vibe, instrumentation, and the gist of the genre.</p><p>(D) Pick for your constraint. Need throughput and accuracy? Audio Flamingo 3. Want a Swiss-army multimodal model and can tolerate misses? Gemma 4. Building something that also generates audio? MOSS-Audio’s token approach is the right shape.</p><h3>Check the code in the repo</h3><p>Everything is in this folder:</p><ul><li><a href="https://github.com/aminfadaei116/published-tutorials/blob/main/Music-Understanding/code">code/</a> A standalone, runnable script per model, plus <a href="https://github.com/aminfadaei116/published-tutorials/blob/main/Music-Understanding/code/SETUP.md">code/SETUP.md</a></li><li><a href="https://github.com/aminfadaei116/published-tutorials/blob/main/Music-Understanding/results">results/</a> The raw per-clip JSON for all five models and the <a href="https://github.com/aminfadaei116/published-tutorials/blob/main/Music-Understanding/results/full_report.md">full report</a> with every description.</li></ul><pre># example: one model, one clip, your own question<br>cd code<br>conda activate persona<br>python audio_flamingo3.py --audio /path/to/your/song.wav \<br>    -q &quot;What instruments do you hear?&quot; \<br>    -q &quot;What is the tempo and mood?&quot;</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=826fdd8a13c6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[14. Fine-Tuning a Vision-Language Model with LoRA and QLoRA: A Hands-On Guide]]></title>
            <link>https://medium.com/@aminfadaeinejad.edu/fine-tuning-a-vision-language-model-with-lora-and-qlora-a-hands-on-guide-3b22d090f425?source=rss-aff4faf055a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b22d090f425</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Amin Fadaeinejad]]></dc:creator>
            <pubDate>Sat, 27 Jun 2026 04:04:53 GMT</pubDate>
            <atom:updated>2026-06-27T19:40:19.222Z</atom:updated>
            <content:encoded><![CDATA[<p>A vision-language model (VLM) takes images and text as input and produces text as output. Modern open VLMs, Qwen2.5-VL, Llama 3.2 Vision, Gemma 3, InternVL are built by bolting a vision encoder onto a language model, so they inherit the same problem as plain LLMs: full fine-tuning updates billions of weights and needs more memory than most people have.</p><p>The fix is the same one that works for text. <strong>LoRA (Low-Rank Adaptation)</strong> freezes the base model and trains only a small pair of low-rank matrices next to the original weights. QLoRA loads the frozen base in 4-bit, so a 3B–7B VLM fits comfortably on a single 16 GB GPU. If you have read the text-only guide in the LoRA-Fine Tuning <a href="https://medium.com/@aminfadaeinejad.edu/fine-tuning-an-llm-with-lora-and-qlora-a-hands-on-guide-441ea09360c5?sharedUserId=aminfadaeinejad.edu">blog</a>, the method here is identical. This guide focuses on the four things that are genuinely different for vision:</p><ol><li>You load an image-text-to-text model and use a processor (tokenizer + image processor), not just a tokenizer.</li><li>Your dataset carries images alongside the text.</li><li>You almost always freeze the vision encoder and adapt only the language model.</li><li>A handful of trainer settings are VLM-specific; most importantly, packing must be off, and you should train with completion-only loss so the hundreds of image tokens don’t pollute the loss.</li></ol><blockquote><em>This guide is code-first. The example fine-tunes </em><em>Qwen/Qwen2.5-VL-3B-Instruct to answer questions about charts (the ChartQA dataset), a task where the base model is mediocre, and a small adapter gives a visible lift. Visit </em><a href="https://github.com/aminfadaei116/published-tutorials/tree/main/LoRA-FT-Vision"><em>https://github.com/aminfadaei116/published-tutorials/tree/main/LoRA-FT-Vision</em></a><em> to see the code.</em></blockquote><h3>1. Idea</h3><p>LoRA is unchanged from the text case. A linear layer computes h = W0 · x; LoRA freezes W0 and adds a low-rank update beside it:</p><pre>h = W0 · x + (alpha / r) · B · A · x</pre><p>A is (r, k), B is (d, r), the rank r is small (8–64), and only A and B train. B starts at zero, so the model begins exactly at the base behaviour.</p><p>What is new for a VLM is where you put the adapters. A VLM has two stacks:</p><ul><li>Vision Encoder (ViT) that turns image patches into a sequence of visual tokens, plus a small projector that maps them into the language model’s embedding space</li><li>A language model that consumes the visual tokens combined with the text tokens and generates the answer.</li></ul><p>The standard, well-tested recipe is to freeze the vision encoder and adapt only the language model’s projections (q_proj, k_proj, v_proj, o_proj and the MLP gate_proj, up_proj, down_proj). The vision encoder already produces good features; what you usually want to teach is how the language model reasons about those features for your task. Adapting the vision tower as well is possible (--tune-vision in the code) But it costs more memory and rarely helps for instruction-style tasks.</p><p>QLoRA adds one thing: the frozen language-model weights are stored in 4-bit (NF4), while the LoRA matrices and the (small) vision encoder stay in 16-bit.</p><h3>2. Environment</h3><p>The vision stack moves fast. Qwen2.5-VL needs transformers &gt;= 4.49, and native VLM support in TRL&#39;s SFTTrainer (the on-the-fly image collator) needs a recent trl. Install recent versions:</p><pre>pip install -U &quot;transformers&gt;=4.49&quot; &quot;trl&gt;=0.14&quot; &quot;peft&gt;=0.13&quot; \<br>               &quot;datasets&gt;=3.0&quot; &quot;accelerate&gt;=1.0&quot; &quot;bitsandbytes&gt;=0.44&quot; \<br>               qwen-vl-utils torchvision pillow requests tensorboard</pre><pre>import torch<br>from datasets import load_dataset<br>from transformers import (<br>    AutoModelForImageTextToText,<br>    AutoProcessor,<br>    BitsAndBytesConfig,<br>)<br>from peft import LoraConfig<br>from trl import SFTConfig, SFTTrainer<br><br>MODEL_ID = &quot;Qwen/Qwen2.5-VL-3B-Instruct&quot;<br># Smaller alternative: &quot;Qwen/Qwen2-VL-2B-Instruct&quot;</pre><p>AutoModelForImageTextToText is the modern, architecture-agnostic class for image+text→text models; it replaces the older per-model classes like Qwen2_5_VLForConditionalGeneration and works for Llama-Vision, Gemma 3, etc.</p><h3>3. Load the base VLM in 4-bit</h3><p>This is the “Q” in QLoRA. The key VLM-specific detail is in the last line: keep the vision encoder out of 4-bit. The vision tower is small and quantizing it measurably hurts quality, so we skip it (and lm_head).</p><pre>bnb_config = BitsAndBytesConfig(<br>    load_in_4bit=True,<br>    bnb_4bit_quant_type=&quot;nf4&quot;,<br>    bnb_4bit_use_double_quant=True,<br>    bnb_4bit_compute_dtype=torch.bfloat16,<br>    llm_int8_skip_modules=[&quot;visual&quot;, &quot;lm_head&quot;],  # keep vision tower in 16-bit<br>)<br><br>model = AutoModelForImageTextToText.from_pretrained(<br>    MODEL_ID,<br>    quantization_config=bnb_config,<br>    dtype=torch.bfloat16,<br>    device_map=&quot;auto&quot;,<br>    # attn_implementation=&quot;flash_attention_2&quot;,  # add if flash-attn is installed<br>)<br>model.config.use_cache = False   # required with gradient checkpointing</pre><p>The skip-module name (&quot;visual&quot;) is Qwen&#39;s name for its vision tower. For other models, check print(model) Llama-Vision calls it vision_model, Gemma 3 vision_tower. On Ampere or newer GPUs, use bfloat16; on a T4 use torch.float16 everywhere instead.</p><p><strong>Version note!!:</strong> dtype= is the current argument name. If you get TypeError about an unexpected keyword dtype, your Transformers is older use torch_dtype=torch.bfloat16 instead (it means the same thing). This applies everywhere dtype= appears below (Sections 10 and 11).</p><h3>4. The processor</h3><p>A VLM does not use a bare tokenizer. It uses a processor that bundles the tokenizer with an image processor. The processor also owns the chat template, which, for a VLM, knows how to splice image placeholder tokens into the prompt.</p><pre>processor = AutoProcessor.from_pretrained(<br>    MODEL_ID,<br>    # Bound how many visual tokens each image becomes. Each 28x28 patch is one<br>    # token, so a large image can explode into thousands of tokens and blow up<br>    # memory. These caps keep it sane.<br>    min_pixels=4 * 28 * 28,<br>    max_pixels=512 * 28 * 28,<br>)</pre><p>min_pixels/max_pixels are the single most important memory knob for VLM training. A 1280×960 photo at full resolution is ~1500 visual tokens <em>per image</em>; capping max_pixels brings that down dramatically with little accuracy loss for most tasks. (These two arguments are Qwen-VL&#39;s interface; other processors expose size={&quot;longest_edge&quot;: ...} or similar.)</p><h3>5. Prepare the dataset</h3><p>This is where vision differs most from text. Each example needs the image(s) plus the conversation. We use ChartQA, which has animage, aquery, and a label.</p><p>The format that matters: produce each example as {&quot;images&quot;, &quot;prompt&quot;, &quot;completion&quot;}. TRL recognizes this prompt-completion format and computes the loss only on the completion (the answer) masking the prompt, which is where all the image-placeholder tokens live.</p><pre>raw = load_dataset(&quot;HuggingFaceM4/ChartQA&quot;, split=&quot;train&quot;)<br><br>def to_chat(example):<br>    answer = example[&quot;label&quot;]<br>    if isinstance(answer, list):<br>        answer = answer[0]<br>    return {<br>        &quot;images&quot;: [example[&quot;image&quot;]],                                # list of PIL images<br>        &quot;prompt&quot;: [{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: example[&quot;query&quot;]}],   # plain text is fine<br>        &quot;completion&quot;: [{&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: str(answer)}],<br>    }<br>dataset = raw.map(to_chat, remove_columns=raw.column_names)<br># Inspect one example before training.<br>print(dataset[0][&quot;prompt&quot;])<br>print(dataset[0][&quot;completion&quot;])<br>print(dataset[0][&quot;images&quot;][0].size)</pre><p>Two things to note:</p><ul><li>The message content is a plain string. You do not hand-insert an {&quot;type&quot;: &quot;image&quot;} block — TRL&#39;s vision collator calls prepare_multimodal_messages for you and injects the image placeholder before the first user turn, then matches it to the image in &quot;images&quot;.</li><li>Why prompt-completion and not a single messages list? With a flat messages field, TRL falls back to language-modelling loss over every token, including the hundreds of image-placeholder tokens. In practice, that sends the training loss to absurd values (we measured ~16 with ~6% token accuracy) because the model is being asked to &quot;predict&quot; image tokens. Switching to prompt-completion (which masks the prompt) drops the loss to a healthy ~0.2–0.7. This is the most common silent mistake in VLM SFT.</li></ul><p>Sanity check. The print(...) output above, next to the chart, describes — dataset[0] from ChartQA, with its prompt/completion and image size:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uhcb28kwhrjgM155.png" /></figure><h3>6. The LoRA configuration</h3><p>Adapt the language model’s attention and MLP projections, and exclude the vision tower so it stays frozen.</p><pre>lora_config = LoraConfig(<br>    r=16,<br>    lora_alpha=32,                # alpha = 2*r is a safe default<br>    lora_dropout=0.05,<br>    bias=&quot;none&quot;,<br>    task_type=&quot;CAUSAL_LM&quot;,<br>    target_modules=[<br>        &quot;q_proj&quot;, &quot;k_proj&quot;, &quot;v_proj&quot;, &quot;o_proj&quot;,   # attention<br>        &quot;gate_proj&quot;, &quot;up_proj&quot;, &quot;down_proj&quot;,      # MLP<br>    ],<br>    exclude_modules=r&quot;.*visual.*&quot;,   # freeze the vision encoder (Qwen names it &quot;visual&quot;)<br>)</pre><p>Why exclude_modules? Qwen2.5-VL&#39;s vision blocks also contain layers named gate_proj/up_proj/down_proj, so a bare target_modules The list would attach adapters inside the vision tower, too. The exclude regex keeps it frozen, which is the standard recipe. (Drop the exclude — or pass --tune-vision In the code, if you specifically want to adapt vision as well.)</p><p>You do not need to call prepare_model_for_kbit_training or get_peft_model yourself; passing peft_config to SFTTrainer does it for you.</p><p>A useful check after the trainer is built: trainer.model.print_trainable_parameters() should report well under 1% (we see trainable params: 29,933,568 || all params: 3,784,556,544 || trainable%: 0.79). If it is much higher, you exclude_modules didn&#39;t take it, and you&#39;re training the vision tower by accident.</p><h3>7. Training configuration and trainer</h3><p>SFTConfig are the same class as the text case. Three settings are VLM-specific and flagged below.</p><pre>sft_config = SFTConfig(<br>    output_dir=&quot;qwen2.5-vl-3b-chartqa-lora&quot;,<br>    num_train_epochs=1,<br>    per_device_train_batch_size=2,      # images are memory-heavy; start small<br>    gradient_accumulation_steps=8,      # effective batch size = 16<br>    gradient_checkpointing=True,<br>    gradient_checkpointing_kwargs={&quot;use_reentrant&quot;: False},<br>    learning_rate=2e-4,<br>    lr_scheduler_type=&quot;cosine&quot;,<br>    warmup_ratio=0.03,<br>    logging_steps=5,<br>    save_strategy=&quot;epoch&quot;,<br>    bf16=True,                          # fp16=True on older GPUs<br>    max_length=2048,                    # prompt + completion token budget<br>    packing=False,                      # REQUIRED for VLMs (TRL raises otherwise)<br>    report_to=&quot;tensorboard&quot;,<br>)<br><br>trainer = SFTTrainer(<br>    model=model,<br>    args=sft_config,<br>    train_dataset=dataset,<br>    peft_config=lora_config,<br>    processing_class=processor,         # passing a *processor* turns on VLM mode<br>)<br>trainer.model.print_trainable_parameters()</pre><p>The two things to internalize:</p><ul><li>processing_class=processor. Because you pass a processor (not a tokenizer), TRL switches into vision mode: it auto-selects DataCollatorForVisionLanguageModeling, which tokenizes text and converts images to pixel values on the fly for each batch (pre-processing every image to disk would be enormous).</li><li>packing=False. Packing concatenates samples to fill the context window; that is incompatible with per-sample images, so TRL raises a ValueError if you leave it on. (For the same reason, padding_free and assistant_only_loss are not supported for VLMs in the current TRL.)</li></ul><h3>8. Train</h3><pre>trainer.train()</pre><p>Start small. The full ChartQA train split is ~28k examples, about 1,750 optimizer steps at the above batch size, which is roughly 4+ hours on one GPU. For your first pass, slice the dataset and cap the steps so you can confirm the whole pipeline (and your loss curve) in a few minutes before committing to a long run:</p><pre>raw = load_dataset(&quot;HuggingFaceM4/ChartQA&quot;, split=&quot;train[:512]&quot;)  # in Section 5<br># ...and add to SFTConfig:<br>#   max_steps=40,</pre><p>The accompanying train.py exposes these as --subset-size and --max-steps (see the README smoke test), So you don&#39;t have to edit the code.</p><p>Watch the training loss (should fall into the 0.2–0.7 range for ChartQA and flatten) and GPU memory via nvidia-smi. If you hit out-of-memory, the cheapest fixes in order are: lower max_pixels (this is usually the biggest lever for VLMs), then per_device_train_batch_size, then raise gradient_accumulation_steps to keep the effective batch size constant.</p><p>Loss curve. Launch tensorboard --logdir qwen2.5-vl-3b-chartqa-lora to watch it live. Below is the curve from a smoke run (60 steps on a 512-example subset, 4-bit QLoRA): training loss falls from ~1.0 to ~0.13 and token accuracy climbs to ~0.95 squarely in the healthy 0.2–0.7 band and flattening.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JCd0pklV-fr_EI6e.png" /></figure><h3>9. Save the adapter</h3><pre>trainer.save_model(&quot;qwen2.5-vl-3b-chartqa-lora/final&quot;)<br>processor.save_pretrained(&quot;qwen2.5-vl-3b-chartqa-lora/final&quot;)</pre><p>Save the processor next to the adapter, not just the tokenizer — you need the image-processing config (including your min/max_pixels) at inference time. The adapter itself is small (tens of MB); the processor adds the tokenizer files.</p><h3>10. Inference</h3><p>Load the base VLM and attach the adapter, then ask a question about an image.</p><pre>from io import BytesIO<br>import requests<br>from PIL import Image<br>from peft import PeftModel<br><br>base = AutoModelForImageTextToText.from_pretrained(<br>    MODEL_ID, quantization_config=bnb_config, dtype=torch.bfloat16, device_map=&quot;auto&quot;,<br>)<br>model = PeftModel.from_pretrained(base, &quot;qwen2.5-vl-3b-chartqa-lora/final&quot;)<br>model.eval()<br>processor = AutoProcessor.from_pretrained(&quot;qwen2.5-vl-3b-chartqa-lora/final&quot;)<br>image = Image.open(&quot;chart.png&quot;).convert(&quot;RGB&quot;)   # or load from a URL<br>messages = [{<br>    &quot;role&quot;: &quot;user&quot;,<br>    &quot;content&quot;: [<br>        {&quot;type&quot;: &quot;image&quot;},<br>        {&quot;type&quot;: &quot;text&quot;, &quot;text&quot;: &quot;How many food items are shown in the bar graph?&quot;},<br>    ],<br>}]<br>text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)<br>inputs = processor(text=[text], images=[image], return_tensors=&quot;pt&quot;).to(model.device)<br>out = model.generate(**inputs, max_new_tokens=128, do_sample=False)<br>generated = out[0][inputs[&quot;input_ids&quot;].shape[1]:]<br>print(processor.decode(generated, skip_special_tokens=True))</pre><p>Two inference details specific to VLMs:</p><ul><li>At inference, you build the content list explicitly with an {&quot;type&quot;: &quot;image&quot;} block (unlike training, where TRL inserted it for you). The apply_chat_template call turns it into the right placeholder tokens; the processor(...) call then produces pixel_values and image_grid_thw next to input_ids. Always slice the output inputs[&quot;input_ids&quot;].shape[1] to drop the prompt.</li><li>Prefer greedy decoding (do_sample=False) for short factual answers like chart readings; turn sampling on only for open-ended descriptions.</li></ul><p>If a freshly trained adapter outputs a stream of identical tokens (e.g. !!!!), it is almost always undertrained or overfit on a tiny set train for more steps on more data not a bug in the inference code. Verify by running the same code with the base model and no adapter; it should answer normally.</p><h3>11. Merge for deployment</h3><p>For production, fold the adapter into the weights and save a standalone model that vLLM or TGI can serve directly.</p><pre>from peft import PeftModel<br><br># Merge into the 16-bit base, NOT the 4-bit one.<br>base_fp16 = AutoModelForImageTextToText.from_pretrained(<br>    MODEL_ID, dtype=torch.bfloat16, device_map=&quot;auto&quot;,<br>)<br>merged = PeftModel.from_pretrained(base_fp16, &quot;qwen2.5-vl-3b-chartqa-lora/final&quot;)<br>merged = merged.merge_and_unload()<br>merged.save_pretrained(&quot;qwen2.5-vl-3b-chartqa-merged&quot;)<br>AutoProcessor.from_pretrained(&quot;qwen2.5-vl-3b-chartqa-lora/final&quot;).save_pretrained(<br>    &quot;qwen2.5-vl-3b-chartqa-merged&quot;<br>)</pre><p>As with text LoRA, merge into the 16-bit base, never the 4-bit one, to avoid re-quantization error. Remember to save the processor into the merged folder too, or downstream serving won’t know how to preprocess images.</p><h3>12. Choosing hyperparameters</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hy8N43aGVL-zhurI9fxyZQ.png" /></figure><p>The biggest VLM-specific lever is max_pixels: it trades visual detail for memory and speed more directly than any text knob.</p><h3>13. Common pitfalls</h3><ul><li>Flat messages format → wrong loss. Use the {&quot;prompt&quot;, &quot;completion&quot;} format so the loss is computed only on the answer. The flat-messages path trains on image-placeholder tokens and inflates the loss massively (Section 5).</li><li>Leaving packing=True. Not supported for VLMs; TRL raises. Set it to False.</li><li>Quantizing the vision tower. Keep it in 16-bit via llm_int8_skip_modules=[&quot;visual&quot;, ...]4-bit vision features degrade quality.</li><li>Vision Tower was adapted by accident. Some VLMs reuse layer names like gate_proj in the vision blocks, so a bare target_modules list adapts them. Use exclude_modules and confirm with print_trainable_parameters().</li><li>Forgetting to save the processor. Saving only the adapter loses the image config, causing inference to preprocess images incorrectly.</li><li>Huge images → OOM. Set max_pixels. This matters far more than for text.</li><li>Wrong compute_dtype for the GPU. bfloat16 needs Ampere or newer; on a T4, use float16 everywhere.</li></ul><h3>14. Evaluating the result</h3><p>Loss going down is necessary, but not sufficient. For VQA-style tasks, hold out a validation split (eval_dataset + eval_strategy=&quot;steps&quot;) and, more importantly, compute task accuracy on held-out images — exact-match or relaxed-accuracy for ChartQA, for example, since that is the number that actually matters. Always read a handful of generated answers on images the model did not see in training; a low loss with copy-paste-looking answers usually means you over-trained on a small set.</p><h3>References</h3><p>[1] E. J. Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models,” 2021. <a href="https://arxiv.org/abs/2106.09685">https://arxiv.org/abs/2106.09685</a></p><p>[2] T. Dettmers et al., “QLoRA: Efficient Finetuning of Quantized LLMs,” 2023. <a href="https://arxiv.org/abs/2305.14314">https://arxiv.org/abs/2305.14314</a></p><p>[3] Qwen Team, “Qwen2.5-VL,” 2025. <a href="https://github.com/QwenLM/Qwen2.5-VL">https://github.com/QwenLM/Qwen2.5-VL</a></p><p>[4] Hugging Face, “TRL — Supervised Fine-tuning Trainer (VLM support).” <a href="https://huggingface.co/docs/trl/sft_trainer">https://huggingface.co/docs/trl/sft_trainer</a></p><p>[5] Hugging Face, “PEFT documentation.” <a href="https://huggingface.co/docs/peft">https://huggingface.co/docs/peft</a></p><p>[6] A. Masry et al., “ChartQA: A Benchmark for Question Answering about Charts,” 2022. <a href="https://arxiv.org/abs/2203.10244">https://arxiv.org/abs/2203.10244</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b22d090f425" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>