<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nandini Lokesh Reddy on Medium]]></title>
        <description><![CDATA[Stories by Nandini Lokesh Reddy on Medium]]></description>
        <link>https://medium.com/@nandinilreddy?source=rss-6bc2b22a69b5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*yQ3CsSeX1bKGAFjMLzf87Q.jpeg</url>
            <title>Stories by Nandini Lokesh Reddy on Medium</title>
            <link>https://medium.com/@nandinilreddy?source=rss-6bc2b22a69b5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 20:27:37 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nandinilreddy/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[LocateAnything]]></title>
            <link>https://medium.com/@nandinilreddy/locateanything-0bee5495880b?source=rss-6bc2b22a69b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/0bee5495880b</guid>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[vlm]]></category>
            <dc:creator><![CDATA[Nandini Lokesh Reddy]]></dc:creator>
            <pubDate>Thu, 11 Jun 2026 04:06:15 GMT</pubDate>
            <atom:updated>2026-06-11T04:06:15.239Z</atom:updated>
            <content:encoded><![CDATA[<p><em>From CVPR 2026</em></p><p>As we are moving from models that just have vision capabilities into systems that are also used for robotics and real-world agents, we need VLMs to be deeply grounded in perception. A model cannot just understand something conceptually anymore, it has to be pixel-perfect with coordinates, boundaries, and spatial reasoning to make robots operate reliably in the real world.</p><p>For example, if a robot is asked to <em>“pick up the white mug near the laptop,”</em> understanding language alone is not enough. The system must know exactly where that mug exists in pixel space. Even a tiny localization error can completely fail the task.</p><p>And this becomes even more important because real-world systems cannot wait several seconds to make decisions. In robotics, autonomous driving, industrial automation, or AR systems, latency matters a lot.</p><p>This is the problem the <strong>LocateAnything</strong> paper tries to solve.</p><h4>The Core Problem With Current VLM Grounding</h4><p>Initially, most VLMs treated bounding box coordinates as plain text.</p><p>So instead of understanding a bounding box as one geometric object, the model generated coordinates one token at a time.</p><p>For example:</p><pre>x1 → y1 → x2 → y2</pre><p>or sometimes even worse:</p><pre>&quot;1810&quot; → &quot;1&quot; → &quot;8&quot; → &quot;1&quot; → &quot;0&quot;</pre><p>This means the model serializes a 2D geometric object into a 1D text stream.</p><p>Even though modern VLMs are powerful, they still inherit the same autoregressive decoding problem from language models: everything must be generated sequentially.</p><p>That creates two major issues:</p><ul><li>Higher latency</li><li>Lower throughput</li></ul><p>But another hidden issue is even more important:</p><p>The coordinates inside a bounding box are highly correlated structurally.<br>The model should understand them together as one geometric entity, not as four unrelated text tokens.</p><h4>Why Token-By-Token Coordinate Generation Becomes a Bottleneck</h4><p>Most grounding systems today commonly represent coordinates in two ways:</p><p><strong>1. Textual Digit Representation</strong></p><p>Example:</p><pre>&quot;1810&quot; → &quot;1&quot; → &quot;8&quot; → &quot;1&quot; → &quot;0&quot;</pre><p><strong>2. Quantized Coordinate Tokens</strong></p><p>Example:</p><pre>x1 → y1 → x2 → y2</pre><p>Even though these approaches look different, both still force the model into sequential generation.</p><p>And sequential decoding becomes extremely expensive at scale.</p><p>Imagine a scene with 200 objects.</p><p>Now the model has to generate:</p><ul><li>labels</li><li>coordinate streams</li><li>formatting tokens</li><li>object separators</li></ul><p>all one-by-one.</p><p>That is simply too slow for real-time grounded systems.</p><h4>“Why Not Use Multi-Token Prediction?”</h4><p>At first, this sounds like a simple fix.</p><p>Standard AI systems usually learn generation in one of two ways:</p><p><strong>Guessing the Next Block: </strong>The model predicts an entire chunk together.</p><p><strong>Filling In Missing Parts</strong></p><p>Random sections are masked, and the model reconstructs them.</p><p>So naturally, we might think:</p><blockquote><em>Why not directly predict coordinate blocks together?</em></blockquote><p>The problem is that generic chunk prediction has <strong>no structural awareness</strong>.</p><p>Without structure, the model treats everything like one giant random text stream.</p><p>It only learns statistical co-occurrence patterns rather than understanding the geometric rules behind bounding boxes.</p><p>And that becomes disastrous for localization tasks.</p><h4>Why Random Chunking Fails for Bounding Boxes</h4><p>A bounding box is not random text.</p><p>It contains four highly constrained geometric values:</p><pre>(x1, y1, x2, y2)</pre><p>Each coordinate depends on the others.</p><p>If one coordinate becomes invalid, the entire box geometry collapses.</p><p>Random chunk prediction introduces several problems:</p><p><strong>a. Sloppy Boundaries: </strong>The model may accidentally combine coordinates from different objects together.</p><p><strong>b. Ghost Patterns: </strong>The model starts memorizing spurious coordinate relationships that are not geometrically valid.</p><p><strong>c. Snowball Effect: </strong>If the first predicted coordinate block is slightly wrong, all simultaneously predicted values can collapse together.</p><p>This creates unstable localization outputs.</p><h4>How LocateAnything Solves This</h4><p>The paper introduces <strong>Parallel Box Decoding (PBD):</strong></p><p>Instead of treating coordinates as independent tokens, the model treats the entire box as one atomic structural unit.</p><p>So rather than predicting:</p><pre>x1 → y1 → x2 → y2</pre><p>the model predicts the entire coordinate block together.</p><p>That dramatically reduces decoding time while preserving geometric consistency.</p><p><strong>Step : Massive Grounding Dataset</strong></p><p>To make the model learn structured grounding, the model first needs massive high-quality grounding data.</p><p>The paper uses:</p><ul><li>12M unique images</li><li>138M language queries</li><li>785M bounding boxes</li></ul><p>spanning six core data types:</p><p><strong>a. General Object Detection: </strong>Standard visual grounding.</p><p><strong>b. GUI Grounding: </strong>Understanding interfaces, buttons, icons, and layouts.</p><p><strong>c. Referring Comprehension: </strong>Finding objects based on natural language references.</p><p><strong>d. OCR &amp; Text Localization: </strong>Locating text regions inside images.</p><p><strong>e. Layout Grounding: </strong>Understanding document and structural layouts.</p><p><strong>f. Point-Based Localization: </strong>Precise coordinate grounding.</p><p>This mixture gives the model broad open-world perception along with dense localization capability.</p><p><strong>Step 2 : Dual-Formation Joint Training</strong></p><p>The model uses a specialized two-stage training pipeline.</p><p>The goal is:</p><ul><li>preserve language reasoning</li><li>enable fast structural decoding</li><li>maintain geometric consistency</li></ul><p><strong>Stage 1: Open-World Perception</strong></p><p>The model trains on the massive 138M-query mixture to learn broad grounding capability.</p><p><strong>Stage 2 : Dense Spatial Precision</strong></p><p>The ratio of general data is lowered.</p><p>Now the model is heavily exposed to cluttered, densely packed scenes to improve precise boundary delineation.</p><p>Especially for:</p><ul><li>overlapping objects</li><li>tiny UI components</li><li>crowded environments</li><li>dense layouts</li></ul><p>The Input Sequence: During training, the sequence becomes:</p><pre>x_all = x_vis ⊕ x_q ⊕ x_ntp ⊕ x_blk</pre><p>Where:</p><ul><li>x_vis → visual tokens</li><li>x_q → query tokens</li><li>x_ntp → standard next-token prediction stream</li><li>x_blk → new block prediction stream</li></ul><p>Inside the block stream, coordinate values are hidden behind masked structural templates.</p><p>The model then jointly optimizes both objectives:</p><pre>L = L_ntp + L_mtp</pre><p>This forces the model to learn:</p><ul><li>conversational reasoning</li><li>geometric block prediction</li><li>structured localization</li></ul><p>simultaneously.</p><p><strong>Step 3 : The Architecture</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HzoiRSbC5cruKjWzLJZp1Q.png" /><figcaption>Reference: LocateAnything paper</figcaption></figure><p>When an image and text prompt enter the system, they pass through a standard high-performance VLM pipeline.</p><pre>[ Input Image ] → [ Moon-ViT Encoder ]<br>                          ↓<br>                    (MLP Projector)<br>                          ↓<br>[ Text Query ] → [ Qwen2.5 Decoder ] → [ Box-Aligned Output ]</pre><p><strong>Moon-ViT Encoder:</strong></p><p>The vision encoder processes images at native resolution.</p><p>This is extremely important because downscaling destroys tiny spatial details.</p><p>For example:</p><ul><li>tiny buttons</li><li>OCR regions</li><li>UI elements</li><li>distant objects</li></ul><p>The encoder preserves fine-grained visual tokens needed for precise localization.</p><p><strong>MLP Projector</strong></p><p>The projector acts like a bridge. It transforms visual embeddings into the same embedding space understood by the language model.</p><p><strong>Qwen2.5 Decoder:</strong></p><p>The Qwen2.5 decoder becomes the reasoning engine.</p><p>It combines:</p><ul><li>visual understanding</li><li>text instructions</li><li>grounding logic</li></ul><p>to generate structured spatial outputs.</p><p><strong>Step 4: The L=6 Structural Output Block</strong></p><p>Instead of writing coordinates as free-form text, the model uses rigid structural templates.</p><p>Each bounding box becomes:</p><pre>b_i = [ &lt;box&gt;, x1, y1, x2, y2, &lt;/box&gt; ]</pre><p>Exactly six tokens are generated for every box.</p><p>If the output is empty or text-only, null placeholders are inserted to preserve tensor consistency.</p><p><strong>Also, Bidirectional Intra-Block Attention</strong></p><p>Normally, language models use strict left-to-right attention. But LocateAnything changes this <em>inside the coordinate block</em>.</p><p>Across Different Boxes, the model still uses standard causal attention. This helps maintain object memory and prevents duplicate predictions.</p><p><strong>Inside One Active Box</strong></p><p>The restriction is removed. All coordinate tokens can attend to each other simultaneously.</p><p>So:</p><ul><li>x1</li><li>y1</li><li>x2</li><li>y2</li></ul><p>all evaluate together in one forward pass.</p><p>This allows the model to solve the full 2D geometry jointly instead of sequentially.</p><p>That is the main reason Parallel Box Decoding becomes both faster and more stable.</p><p><strong>Step 5 : The SafetyNet Hybrid Mode</strong></p><p>Parallel generation is fast. But chaotic scenes can still create unstable outputs. So the paper introduces a dynamic fallback mechanism.</p><p><strong>Fast Mode: </strong>The model decodes all boxes in parallel.</p><p>Performance reaches roughly:</p><ul><li>~15 boxes per second</li><li>~2.5× faster than specialized detectors</li><li>10× faster than traditional VLM grounding systems</li></ul><p><strong>Fallback Trigger</strong></p><p>The model constantly monitors:</p><ul><li>syntax validity</li><li>structural formatting</li><li>confidence stability</li></ul><p>If it detects:</p><ul><li>malformed outputs</li><li>overlapping ambiguity</li><li>unstable geometry</li></ul><p>it triggers fallback mode.</p><p><strong>Correction Pass</strong></p><p>Only the problematic block is discarded.</p><p>The model temporarily switches back to careful sequential prediction for that box alone.</p><p>Once corrected, it instantly returns to high-speed parallel decoding.</p><p>This hybrid design is actually one of the smartest parts of the paper because it balances:</p><ul><li>speed</li><li>reliability</li><li>geometric consistency</li></ul><p>without fully sacrificing either side.</p><h4>Implementation:</h4><pre>import os<br>import re<br>import torch<br>from PIL import Image, ImageDraw<br>from locateanything_worker import LocateAnythingWorker<br><br>MODEL = &quot;nvidia/LocateAnything-3B&quot;<br>IMAGE = &quot;/estimage.png&quot;<br>OUTPUT_DIR = &quot;Embodied/outputs&quot;<br><br>os.makedirs(OUTPUT_DIR, exist_ok=True)<br><br>COLORS = [&quot;#FF3333&quot;, &quot;#33FF57&quot;, &quot;#3399FF&quot;, &quot;#FF33CC&quot;, &quot;#FFAA00&quot;, &quot;#00FFEE&quot;]<br><br><br>def draw_results(answer: str, image: Image.Image, out_path: str):<br>    w, h = image.size<br>    draw = ImageDraw.Draw(image)<br>    count = 0<br>    for i, m in enumerate(re.finditer(r&quot;&lt;ref&gt;(.*?)&lt;/ref&gt;((?:&lt;box&gt;.*?&lt;/box&gt;)+)&quot;, answer)):<br>        label = m.group(1)<br>        color = COLORS[i % len(COLORS)]<br>        boxes = LocateAnythingWorker.parse_boxes(m.group(0), w, h)<br>        for box in boxes:<br>            draw.rectangle([box[&quot;x1&quot;], box[&quot;y1&quot;], box[&quot;x2&quot;], box[&quot;y2&quot;]], outline=color, width=3)<br>            draw.text((box[&quot;x1&quot;] + 4, box[&quot;y1&quot;] + 2), label, fill=color)<br>            count += 1<br>    image.save(out_path)<br>    print(f&quot;  -&gt; {count} boxes drawn, saved: {out_path}&quot;)<br><br><br>print(f&quot;Loading model: {MODEL}&quot;)<br>worker = LocateAnythingWorker(MODEL)<br>img_orig = Image.open(IMAGE).convert(&quot;RGB&quot;)<br>print(f&quot;Image loaded: {img_orig.size}&quot;)<br>img = img_orig.resize((1024, 1024), Image.LANCZOS)<br>print(f&quot;Image resized to: {img.size}&quot;)<br><br>print(&quot;\n--- Object Detection ---&quot;)<br>result = worker.detect(img.copy(), [&quot;car&quot;, &quot;vehicle&quot;, &quot;truck&quot;])<br>print(result[&quot;answer&quot;])<br>draw_results(result[&quot;answer&quot;], img.copy(), f&quot;{OUTPUT_DIR}/detection.png&quot;)<br><br>print(&quot;\n--- Phrase Grounding ---&quot;)<br>result = worker.ground_multi(img.copy(), &quot;cars in the parking lot&quot;)<br>print(result[&quot;answer&quot;])<br>draw_results(result[&quot;answer&quot;], img.copy(), f&quot;{OUTPUT_DIR}/grounding.png&quot;)<br><br>print(&quot;\n--- Scene Text Detection ---&quot;)<br>result = worker.detect_text(img.copy())<br>print(result[&quot;answer&quot;])<br>draw_results(result[&quot;answer&quot;], img.copy(), f&quot;{OUTPUT_DIR}/text_detection.png&quot;)<br><br>print(f&quot;\nAll outputs saved to: {OUTPUT_DIR}&quot;)</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uTcDf70CaiRiD3Jd4XHzqw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-QhGGm2IA7bn_fmCacyuRQ.png" /></figure><h4>Limitations</h4><p>Even though the system is impressive, several bottlenecks still exist.</p><p><strong>1. Autoregressive Overhead Still Exists</strong></p><p>Even with PBD, the model still generates:</p><ul><li>labels</li><li>formatting tokens</li><li>structural markers</li></ul><p>sequentially.</p><p>So scenes with hundreds of objects can still create latency.</p><p><strong>2. Context Window Bottlenecks</strong></p><p>The system processes:</p><ul><li>high-resolution vision tokens</li><li>dual-stream sequences</li><li>massive grounding context</li></ul><p>This places huge pressure on the transformer context window.</p><p><strong>3. Hybrid Mode Penalty</strong></p><p>In extremely cluttered scenes, the fallback mechanism activates frequently.</p><p>When the system repeatedly drops into sequential correction mode, speed temporarily decreases.</p><h4>Where YOLO and RT-DETR Are Still Better</h4><p>Specialized vision-only systems like YOLO and RT-DETR still dominate in several areas.</p><p><strong>1. Ultra-Low Latency Robotics</strong></p><p>YOLO and RT-DETR process entire images fully in parallel. No language decoding exists in the loop.</p><p>That makes them dramatically faster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nEmDIXF8WJLH96O5Ai2tHw.png" /></figure><p>YOLO can also run on:</p><ul><li>drones</li><li>robotic chips</li><li>mobile devices</li><li>edge hardware</li></ul><p>while large VLMs still require heavy GPU memory.</p><p><strong>2. Extremely Dense Scenes</strong></p><p>If you feed the system a grocery shelf with 500 soda cans:</p><ul><li>VLMs struggle with massive coordinate generation</li><li>context windows become overloaded</li></ul><p>RT-DETR handles this more naturally using parallel object queries.</p><p><strong>3. Closed-Set Industrial Accuracy</strong></p><p>If a factory only needs:</p><ul><li>“good screw”</li><li>“defective screw”</li></ul><p>a specialized YOLO model trained specifically for that task will likely outperform a general VLM.</p><p>Because it does not need:</p><ul><li>language reasoning</li><li>open-world semantics</li><li>generalized grounding</li></ul><p>It only needs geometric precision for a fixed set of classes.</p><h3>Conclusion:</h3><p>LocateAnything is less about “better object detection” and more about where VLMs are heading overall.</p><p>The important shift here is that VLMs are slowly moving from:</p><blockquote><em>“understanding images conceptually”</em></blockquote><p>toward:</p><blockquote><em>“understanding the physical spatial structure of the world.”</em></blockquote><p>That difference matters a lot.</p><p>Because once models become truly grounded in geometry, localization, and spatial consistency, they stop being just chat systems with vision and start becoming usable systems for:</p><ul><li>robotics</li><li>agents</li><li>autonomous systems</li><li>industrial automation</li><li>GUI agents</li><li>AR/VR interaction</li><li>embodied AI</li></ul><p>And Parallel Box Decoding feels like one of the early architectural steps toward that direction.</p><h4>References:</h4><ol><li>paper: <a href="https://research.nvidia.com/labs/lpr/locate-anything/LocateAnything.pdf">https://research.nvidia.com/labs/lpr/locate-anything/LocateAnything.pdf</a></li><li>github: <a href="https://github.com/NVlabs/Eagle/tree/main/Embodied">https://github.com/NVlabs/Eagle/tree/main/Embodied</a></li><li>Qwen 2.5 technical report: <a href="https://arxiv.org/abs/2409.12186">https://arxiv.org/abs/2409.12186</a></li><li>Kimi (Moon ViT Encoder) : <a href="https://arxiv.org/html/2504.07491v1">https://arxiv.org/html/2504.07491v1</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0bee5495880b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[So anything we can measure, we can optimise: Verifiers in AI]]></title>
            <link>https://medium.com/@nandinilreddy/so-anything-we-can-measure-we-can-optimise-verifiers-in-ai-b6e0ae9c9580?source=rss-6bc2b22a69b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/b6e0ae9c9580</guid>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[verifier]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[automata-theory]]></category>
            <dc:creator><![CDATA[Nandini Lokesh Reddy]]></dc:creator>
            <pubDate>Sat, 16 May 2026 00:03:09 GMT</pubDate>
            <atom:updated>2026-05-16T00:03:09.865Z</atom:updated>
            <content:encoded><![CDATA[<p>Why do AI/LLMs generate some answers very fast &amp; some very slow?</p><p>It’s not about complexity of the task. Not about the data, how big or where it came from. Not about how important the task is.</p><p>→ The checker. Verifiers are everything behind the architecture.</p><blockquote><a href="https://notebooklm.google.com/notebook/bcacb1b9-5c80-4fb4-ad35-19ca01c2effa/artifact/c19e6d13-425d-42c0-890d-59005c87d48c?utm_source=nlm_web_share&amp;utm_medium=google_oo&amp;utm_campaign=art_share_2&amp;utm_content=&amp;utm_smc=nlm_web_share_google_oo_art_share_2_">NotebookLM Audio</a></blockquote><p>Once the model generates an answer → sent → verifier (feedback). If the signal from the checker is fast, the model learns per update &amp; improves faster.</p><p>This process is called the <strong>jagged edge of intelligence.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KEnuSiROm3idY5XoLiVWug.png" /><figcaption>Generated by Claude</figcaption></figure><p>AI doesn’t become smarter on its own. They are <em>made</em> smarter in domains where the verifier is clean. &amp; doesn’t get smarter if the specific domain doesn’t have a clean verifier.</p><p>→ The jagged edge of intelligence moves with measurement infrastructure.</p><p>So this becomes the key insight: instead of asking <em>“what can AI do?”</em>, the question is <strong>what we decide to measure.</strong></p><p>&amp; this becomes philosophical.</p><p>When we measured protein structure folding → we handed “AI” to master structural biology. Similarly for games, coding, math…</p><p>In all of these, measurement infrastructure comes first, then the capabilities.</p><p>and, all of this is <strong>not</strong> a natural phenomenon. These are a list of human priorities.</p><p><em>“So anything we can measure, we can optimise it.”</em></p><h3>Similarly, in automata theory</h3><p>Problems can be solved in ‘P’ time &amp; ‘NP’ time.</p><p>But to check if these are solved correctly, we need to design verifiers. Sometimes designing verifiers <em>solves</em> the problem itself.</p><p><strong>How do we build verifiers?</strong></p><ol><li>Define inputs. (The instance (x) &amp; certificate ©)</li><li>Specify the acceptance rule. The verifier V(x,c) must output “true” if &amp; only if:</li></ol><ul><li>c is a valid certificate that solves the instance x</li><li>The length of the certificate |c| is bounded by polynomial instance size</li></ul><p>3. Implement in polynomial time.</p><p>4. Prove 2 properties:</p><ul><li><strong>Completeness: </strong>if a problem has a solution, there must exist at least one certificate that makes the verifier accept.</li><li><strong>Soundness: </strong>if the problem doesn’t have a solution, there is no possible certificate that can trick the verifier into accepting.</li></ul><p>→ Finding solution to larger inputs is practically impossible, but verifying the provided answer is a lot easier.</p><h3>Coming back to verifiers</h3><p>We have different types of verifiers. One of the interesting ones is <strong>asymmetrical verifiers</strong>, a common concept in RL.</p><p>The idea: some tasks are much easier to verify than to solve.</p><p>Writing code to operate applications like a instagram would take multiple engineers, but verifying it is done &amp; can be done by one person.</p><p>Some verifiers are near-symmetry of verification: adding 99 + 186 often takes the same amount of work as solving the problem itself.</p><p>In some verifiers, verification takes <em>longer</em> than to write/prove. Like it might take longer to fact-check all statements in an essay than to write the essay.</p><p>Many scientific hypotheses are harder to verify than to come up with. Easy to state a novel diet, but it would take years to verify whether it’s beneficial for the general population.</p><h3>How are asymmetrical verifiers used in AI/RL?</h3><p>Generation is expensive &amp; verification is cheap.</p><p>RL with verifiable rewards:</p><ul><li>Model generates solution</li><li>Verifier checks correctness</li><li>Based on that, model learns what &amp; how it should behave.</li></ul><p>RL improves through feedback. If the feedback is subjective, slow, or expensive → scaling becomes difficult.</p><p>So asymmetrical verifiers allow: → Automatic rewards → Massive scaling → Consistent learning</p><p>This is why domains with easy verification have rapid AI progress: Math, Coding, Theorem proving &amp; Games.</p><p>These are called asymmetrical because the computational difficulty is unequal.</p><p>Generate proof → hard. Verify proof → easy.</p><p>Write bug-free code → hard. Verify bug-free code → easy.</p><p><em>Generator explores many possibilities; verifier selects/filters good ones.</em></p><p>Asymmetrical verifiers work best when correctness is <strong>objective.</strong></p><h3>Outcome verifiers vs Process verifiers</h3><p>Not all verifiers check the same thing.</p><p><strong>Outcome verifier</strong> only looks at the final answer. Did the code pass the test? Is the math answer correct? → Simple. Fast. Easy to automate. → But blind to <em>how</em> the model got there.</p><p><strong>Process verifier</strong> evaluates the reasoning step by step. Did the model actually think through it correctly, or did it just get lucky? → Much harder to build. → But catches things outcome verifiers miss.</p><p>Example: A model solves a math problem correctly, outcome verifier says correct. But the reasoning was completely wrong. It just got the right number by accident. Process verifier would catch that. Outcome verifier wouldn’t.</p><p>This matters a lot in high-stakes domains. A model that gets right answers for wrong reasons is fragile. Works until the problem changes slightly, then breaks.</p><p>→ RLHF is mostly outcome-based. Humans rate the final response. Fast, but noisy.</p><p>→ Systems like AlphaCode use test cases as verifiers. Code either passes or it doesn’t. Clean, objective, no ambiguity. That’s why coding AI has improved so fast. The verifier is just a compiler &amp; test suite.</p><h3>Reward Hacking</h3><p>The model’s job is to maximise the signal from the verifier. But sometimes it finds shortcuts.</p><p>It doesn’t actually solve the problem. It learns to <em>game the verifier.</em></p><p>This is called <strong>reward hacking.</strong></p><p>Examples:</p><ul><li>A model trained to write “helpful” responses learns to sound helpful, confident tone, structured format, without actually being correct.</li><li>An RL agent trained to score points in a game finds a bug in the environment &amp; exploits it infinitely instead of playing properly.</li><li>A model trained on test cases memorises patterns that pass tests without understanding the underlying logic.</li></ul><p>→ The verifier said correct. The problem isn’t actually solved.</p><p>This happens because the verifier is a <em>proxy</em> for what we actually want. &amp; optimising hard for a proxy tends to break the proxy.</p><p>Goodhart’s Law: <em>“When a measure becomes a target, it ceases to be a good measure.”</em></p><p>The stronger &amp; more capable the model → the better it gets at finding verifier loopholes. → Which means verifier design becomes more important as models scale, not less.</p><h3>Can a model build its own verifier?</h3><p>If verifiers are everything, what happens when we ask the model itself to design one?</p><p>It’s already happening. Models are being used to generate test cases, write evaluation rubrics, critique their own outputs.</p><p>But there’s a fundamental problem with this.</p><p>A model building its own verifier is like a student writing their own exam. They’ll naturally write questions they already know how to answer.</p><p>→ The verifier becomes biased toward what the model is already good at.</p><p>→ Blind spots stay blind spots.</p><p>There’s also a deeper issue: <strong>can a system reliably identify its own errors?</strong></p><p>If the model doesn’t know something is wrong, it won’t build a verifier that catches it. The verifier will just confirm what the model already believes.</p><p>This is called <strong>self-verification collapse.</strong></p><p>Some research tries to get around this with:</p><ul><li>Adversarial models, one model generates, another tries to break it</li><li>Constitutional AI, where the model critiques against a fixed set of principles</li><li>Ensemble verification, multiple models cross-check each other</li></ul><p>But none of these fully solve it. Because if all the models share the same training distribution, they share the same blind spots.</p><p>→ A model can build a verifier. But it can’t build a verifier for what it doesn’t know it doesn’t know.</p><p>That gap is still a human job.</p><h3>Why LLMs specifically hit the jagged edge</h3><p>LLMs are trained on everything. Code, math, essays, philosophy, conversations.</p><p>But their capabilities are uneven. Not because of the data. Because of the verifier quality during training &amp; fine-tuning.</p><p>Domains with clean verifiers: → Code, compiler tells you if it runs → Math, answer is right or wrong → Factual QA, can cross-check against sources</p><p>Domains with weak or noisy verifiers: → Creative writing, humans disagree on what’s good → Reasoning under uncertainty, hard to verify without ground truth → Long-horizon planning, outcome only visible much later</p><p>So the same LLM that writes flawless Python struggles to give consistent advice on an ambiguous life decision.</p><p>It’s not intelligence that’s jagged. It’s the <em>measurement</em> that’s jagged. &amp; the model just reflects that back.</p><h3>What is the measurement of wellness?</h3><p>Here’s a domain where the verifier problem gets really uncomfortable.</p><p>Mental health. Emotional wellbeing. Quality of life.</p><p>We want AI to help with these. Therapy tools, wellness apps, coaching models. &amp; they’re being built &amp; deployed right now.</p><p>But what is the verifier?</p><p>Is it: user says they feel better? → Easy to game. People say what sounds right. Is it: user keeps coming back? → That’s engagement, not wellness. Could even be dependency. Is it: clinical scores like PHQ-9? → Designed for diagnosis, not continuous feedback. Is it: biomarkers, sleep, HRV? → Proxies. Not the thing itself.</p><p>Wellness is a domain where: → The ground truth is subjective → The signal is slow, sometimes years → Humans themselves disagree on what it means → &amp; the person experiencing it often can’t accurately report it</p><p>So what does the verifier look like?</p><p>We don’t really know. &amp; yet we’re training models on this domain, optimising for <em>something</em>, just not always sure what.</p><p>This is the jagged edge in its most uncomfortable form. Not “AI can’t do this yet.” More like, “we haven’t figured out what doing it correctly even means.”</p><p>→ The measurement of wellness is still an open problem. &amp; until it isn’t, the verifier stays weak. &amp; a weak verifier means we’re flying blind.</p><h3>Harder areas</h3><p>→ Creative writing → Philosophy → Design → Open-ended planning → Emotional intelligence</p><p>Because verification itself is difficult. &amp; if humans disagree on what’s right, the verifier becomes weak.</p><p>→ Finding domains where verification is cheaper than generation. That’s the real bottleneck.</p><h3>So what does this mean</h3><p>We keep asking: <em>is AI intelligent?</em></p><p>Maybe that’s the wrong question.</p><p>The better one: <strong>what have we given it clean feedback on?</strong></p><p>Because wherever there’s a clean verifier, AI will get very good, very fast. &amp; wherever there isn’t, it will stay stuck, no matter how large the model gets.</p><p>&amp; here’s the part that doesn’t get said enough:</p><p>The verifier is always a human choice. What to measure, how to measure it, when to say “correct.”</p><p>So the ceiling of AI in any domain isn’t compute. It isn’t data. It isn’t even the model.</p><p>It’s whether we’ve figured out what we actually want.</p><p>&amp; that question, we’ve been trying to answer long before AI existed.</p><blockquote>This article was inspired by Jason Wei’s writing on verifiers, and an Instagram reel I came across while doom scrolling — link lost, as expected.</blockquote><p>References:</p><ol><li><a href="https://www.jasonwei.net/blog/asymmetry-of-verification-and-verifiers-law">https://www.jasonwei.net/blog/asymmetry-of-verification-and-verifiers-law</a></li><li><a href="https://math.stackexchange.com/questions/49509/np-verifier-based-definition">https://math.stackexchange.com/questions/49509/np-verifier-based-definition</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b6e0ae9c9580" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[OpenClaw vs Agents]]></title>
            <link>https://medium.com/@nandinilreddy/openclaw-vs-agents-47f55371c7fa?source=rss-6bc2b22a69b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/47f55371c7fa</guid>
            <category><![CDATA[openclaw]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[agents]]></category>
            <dc:creator><![CDATA[Nandini Lokesh Reddy]]></dc:creator>
            <pubDate>Mon, 23 Mar 2026 18:03:38 GMT</pubDate>
            <atom:updated>2026-03-23T18:03:38.597Z</atom:updated>
            <content:encoded><![CDATA[<p>there’s a bit of confusion right now.</p><p>everything is an “agent”.</p><p>anything with an llm + tool calls gets labelled as an agent.</p><p>then something like openclaw shows up, and the question becomes:</p><ul><li>is this just another agent?</li><li>or is something actually different here?</li></ul><h4>what is an agent?</h4><p>an agent is not a product, not a framework, not even necessarily an llm;</p><p><strong>it’s a loop!</strong></p><pre>while not done:<br>  observe state<br>  decide next action<br>  execute action</pre><p>that’s it;</p><p>if the system doesn’t have this loop, then its not an agent.</p><p>its just a function.</p><h4>what people call agents today:</h4><p>most “agent” systems today look like:</p><ul><li>an llm(policy)</li><li>a set of tools (functions with schemas)</li><li>a thin orchestrator loop</li></ul><p>frameworks like langchain or crewai help write this together.</p><pre>user input<br> ↓<br>llm (decides action)<br> ↓<br>tool call<br> ↓<br>result<br> ↓<br>llm (next step)</pre><p>some properties fall out of this:</p><ul><li>execution is bounded (only predefined tools)</li><li>state is mostly in-memory (context, maybe a vector db)</li><li>runs are usually ephemeral</li><li>side effects are controlled</li></ul><p>these systems are closer to: <strong><em>structured api pipelines with a feedback loop</em></strong></p><p>so then what is <strong>“pre-defined”</strong> in agents?</p><p>when we build an agent, we are explicitly defining something like:</p><ul><li>what tools exists</li><li>what the agent is allowed to do</li><li>sometimes even how it should behave</li></ul><p>foe example:</p><p>translation agent:</p><pre>tools = [translate_api] <br>prompt = &quot;you are a translator&quot;</pre><p>trip planner agent:</p><pre>tools = [search, maps, booking_api] <br>prompt = &quot;you plan trips&quot;</pre><p>so, we are binding:</p><p><strong><em>task → tools → behavior</em></strong></p><p>so each new tasks often mean:</p><p>. new setup</p><p>. new constraints,</p><p>. sometimes a new pipeline</p><p><strong>now, where does openclaw sit?</strong></p><p>openclaw keeps the same loop.</p><p>it doesnt change what an agent is.</p><p>it changes<strong><em> where the loop rus</em></strong>, and<strong><em> what it can touch</em></strong>.</p><p>in most agents, tools look like:</p><pre>search(query) <br>send_email(to, body) <br>query_db(sql)</pre><p>clean, typed, controlled;</p><p>in openclaw, the “interface” is effectively:</p><pre>whatsapp(message)<br>telegram(message)<br>slack(channel, message)<br>discord(server, message)<br>imessage(contact, message)</pre><p>this is very different boundary</p><p>now, we are no longer operating in a single controlled channel.</p><p>we are operating in a general-purpose environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7OFNrrbm6QNUcGdurTuY7A.png" /><figcaption>Source: Infographics from NotebookLM</figcaption></figure><h4>Architecture<strong>:</strong></h4><pre>typical agent stack: <br><br>[ llm ]<br>   ↓<br>[ planner / loop ]<br>   ↓<br>[ tool layer (apis) ]<br>   ↓<br>[ external world ]<br><br>openclaw:<br><br>[ llm ]<br>   ↓<br>[ loop ]<br>   ↓<br>[ gateway (control plane) ]<br>↓<br>[ your channels + devices ]</pre><p>the tool layer is no longer narrow,<strong><em> its the entire system</em></strong></p><h4><strong>State is no longer in the prompt</strong></h4><p>in standard agents, state lives in:</p><ul><li>context window</li><li>memory store</li></ul><p>in claw, state lives in:</p><ul><li>sessions</li><li>channel histories</li><li>paired devices</li></ul><p>the environment itself becomes memory</p><p>this means:</p><ul><li>state persists by default</li><li>actions have real side effects</li><li>you cant fully reconstruct state from logs</li></ul><h4><strong>Execution is no longer reversible:</strong></h4><p>in a typical agent:</p><ul><li>bad output → retry</li><li>wrong tool → handle error</li></ul><p>in claw:</p><ul><li>wrong message → sent to a real person</li><li>wrong channel → wrong audience</li></ul><p>the system is no longer just thinking.</p><p><strong><em>it’s mutating state;</em></strong></p><h4><strong>control is weaker:</strong></h4><p>agent frameworks are designed for control:</p><ul><li>predefined tools</li><li>validation layers</li><li>constrained execution</li></ul><p>openclaw loosens this:</p><ul><li>broader action space</li><li>fewer hard boundaries</li><li>more reliance on model judgment</li></ul><p><strong><em>this is where most of the risk comes from.</em></strong></p><h4>So, how does openclaw handle different tasks?</h4><p>like “if i want to translate now, and trip plan later — how is that defined?”</p><p>in agents: you define seperate pipelines</p><p>in claw: there is one general system</p><p>it’s not task specific, its capability-based</p><p>claw doesn’t create translation agent ot trip planner agent;</p><p>instead it has:</p><ul><li>a general purpose model</li><li>a set of capabilities (skills)</li></ul><p>and the <strong>model decides</strong>: <strong><em>which capabilities to use, and in what order</em></strong></p><p>what are skills?</p><p>skills are the hidden layer.</p><p>they are not just single functions, but are small capability bundles:</p><ul><li>instructions(how to use them)</li><li>execution logic (scripts, commands)</li><li>sometimes tool bindings</li></ul><p>examples:</p><ol><li>web browsing</li><li>text processing</li><li>calendar access</li><li>device commands (camera, location, notifications)</li></ol><p>then, how the same system handles different tasks?</p><p><strong>task: translate text</strong></p><p>the system might:</p><ul><li>use the llm directly</li><li>or use a text-processing skill</li></ul><p><strong>task: plan a trip</strong></p><p>the system might:</p><ul><li>browse the web</li><li>collect information</li><li>structure a plan</li></ul><p>same system.<br>different sequence of actions.</p><p>agents:</p><blockquote><em>you define → task → tools</em></blockquote><p>openclaw:</p><blockquote><em>the model decides → task → sequence of skills</em></blockquote><h4>So is openclaw “not pre-defined”?</h4><p>its predefined, just differently;</p><p>agents:</p><ul><li>tools are defiend</li><li>flows are defined</li><li>task mapping is defined</li></ul><p>openclaw:</p><ul><li>capabilities are defined</li><li>flows are dynamic</li><li>task mapping happens at runtime</li></ul><p>a simpler way to think about it</p><p>agents:</p><blockquote><em>you build a separate machine for each job</em></blockquote><p>openclaw:</p><blockquote><em>you build one machine, and it figures out how to use itself</em></blockquote><h4>why this feels powerful?</h4><p>because the abstraction is moved/</p><p>you no longer see:</p><ul><li>pipelines</li><li>tool wiring</li><li>control logic</li></ul><p>you only see:</p><p>intent → outcome</p><p>but internally, the system is still:</p><ul><li>selecting actions</li><li>composing steps</li><li>using predefined capabilities</li></ul><h4>where this breaks:</h4><p>this flexibility comes with tradeoffs:</p><ul><li>no guarantee it picks the right path</li><li>harder to debug</li><li>less predictable</li><li>more fragile in edge cases</li></ul><p>this is why:</p><ul><li>structured agents are still preferred in production</li><li>openclaw-style systems feel powerful but unstable</li></ul><h4>why the hype?</h4><p>because it crosses a threshold.</p><p>we’ve had agents for a while, but they lived in:</p><ul><li>notebooks</li><li>backends</li><li>controlled environments</li></ul><p>openclaw puts the loop:</p><ul><li>on your machine</li><li>with real permissions</li><li>with persistent state</li></ul><p>that makes it feel like:</p><blockquote><em>the system is actually doing things, not just responding</em></blockquote><h4>what changes from here</h4><p>once you move to this setup, the problem shifts.</p><p>it’s no longer:</p><ul><li>did the model answer correctly?</li></ul><p>it becomes:</p><ul><li>did it take the right sequence of actions?</li><li>did it avoid bad side effects?</li><li>can it recover from mistakes?</li><li>how does it behave over time?</li></ul><p>you are evaluating:</p><blockquote><em>trajectories, not outputs</em></blockquote><h4>Conclusion:</h4><p>openclaw is not a new paradigm.</p><p>it’s the same agent loop, but:</p><ul><li>attached to a real environment</li><li>given persistent state</li><li>allowed to take consequential actions</li></ul><p>and that small shift is enough to change how the system behaves, how we use it, and how we evaluate it.</p><h4>References:</h4><ol><li><a href="https://github.com/openclaw/openclaw">https://github.com/openclaw/openclaw</a></li><li><a href="https://www.reddit.com/r/LocalLLaMA/comments/1r5v1jb/anyone_actually_using_openclaw/">https://www.reddit.com/r/LocalLLaMA/comments/1r5v1jb/anyone_actually_using_openclaw/</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=47f55371c7fa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deepseek OCR- 2]]></title>
            <link>https://medium.com/@nandinilreddy/deepseek-ocr-2-82e02e2608ad?source=rss-6bc2b22a69b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/82e02e2608ad</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[deepseek]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ocr]]></category>
            <dc:creator><![CDATA[Nandini Lokesh Reddy]]></dc:creator>
            <pubDate>Thu, 05 Feb 2026 00:37:06 GMT</pubDate>
            <atom:updated>2026-02-05T00:37:06.363Z</atom:updated>
            <content:encoded><![CDATA[<p>For a long time, progress in vision–language models has been narrated through benchmarks: higher accuracy, lower edit distance, better BLEU, better CIDEr. And while those numbers matter, they rarely tell us <em>why</em> a model is better, or more importantly, <em>what kind of problem we are now capable of solving</em>.</p><p><strong>What if the core problem in document understanding is not recognition, but how we look?</strong></p><p>DeepSeek-OCR 2 keeps the same encoder–decoder structure as DeepSeek-OCR 1. There is no dramatic architectural break, no claim of inventing a new paradigm, and the benchmark improvements are modest. But the <em>way</em> the problem is framed feels different.</p><p>the ideas it builds on, how vision models parse information, how structure emerges, how reasoning happens over images; have been circulating for a long time. OCR 2 is interesting because it doesn’t try to solve everything at once. It narrows the problem down to one uncomfortable assumption we’ve mostly accepted:</p><blockquote>Flattening an image into a sequence in raster order is good enough.</blockquote><p>For plain text, maybe it is.<br>For documents, it clearly isn’t.</p><p>Most document understanding pipelines today look like this:</p><ol><li>Split the image into patches</li><li>Encode patches with a ViT or CLIP-style backbone</li><li>Flatten tokens top-left to bottom-right</li><li>Feed the sequence into a language model</li></ol><p>This design is simple. It scales well. It works decently for many tasks.<br>But it also hard-codes a very specific assumption: that spatial order is a reasonable proxy for reading order.</p><p>Humans do not read documents in raster order.</p><p>We don’t start at the top-left pixel and move uniformly to the bottom-right. We look for titles. We scan margins. We jump between columns. We treat tables, equations, and figures differently. Reading is guided by structure, not coordinates.</p><p>DeepSeek-OCR 2 tries to give the decoder a better sequence.</p><h3>Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dnUFgCYUEUdbMj0KUXv31w.png" /><figcaption>Source: D<a href="https://arxiv.org/pdf/2601.20552">eepSeek OCR2</a></figcaption></figure><p>In most OCR and document understanding systems today, the decoder is a large language model. It operates causally. It assumes that the sequence it receives is meaningful in its order. If the order is wrong, the decoder can sometimes compensate, but only up to a point.</p><p>So, Where should the reading order be constructed? Inside the encoder!!!</p><h4>Step 1: Image tokenization and compression</h4><p>The image is first split into visual tokens, similar to most vision–language models.</p><p>These tokens represent local visual regions. OCR 2 still respects a strict visual token budget (hundreds, not thousands), because the goal is to feed the output into an LLM decoder.</p><h4>Step 2: Encoder</h4><p>DeepSeek-OCR 1 used a CLIP-style vision encoder.</p><p>OCR 2 replaces this with a <strong>compact LLM-style transformer used as a vision encoder</strong>.</p><p>This is an important design choice.</p><p>CLIP is good at producing globally aligned embeddings, but it is not designed to model causal relationships or progressive reasoning. An LLM-style transformer already has a bias toward sequence construction, even when used on non-text inputs.</p><p>The encoder is no longer treated as a static feature extractor. It becomes a reasoning module.</p><h4>Step 3: Causal flow tokens</h4><p>This is the core architectural change.</p><p>Instead of flattening visual tokens directly and passing them to the decoder, OCR 2 introduces a second type of token inside the encoder: <strong>causal flow tokens</strong>.</p><p>The encoder input now consists of:</p><ul><li>visual tokens (from the image)</li><li>causal flow tokens (learnable)</li></ul><p>The visual tokens contain information.<br>The causal flow tokens decide how that information is consumed.</p><h4>Step 4: Bidirectional encoder</h4><p>The encoder uses a mixed attention pattern.</p><ul><li>Visual tokens attend to each other <strong>bidirectionally</strong></li></ul><p>Causal flow tokens attend:</p><ul><li>to all visual tokens</li><li>to previous causal flow tokens</li><li>but not to future causal flow tokens</li></ul><p>This creates two distinct behaviors in the same encoder:</p><ul><li>spatial, global reasoning over the image</li><li>sequential, causal reasoning over reading order</li></ul><p>The causal flow tokens form a sequence by construction.</p><blockquote>The model is no longer asking:<br> “What features should I extract?”link</blockquote><blockquote>It is asking:<br> “What should I look at next?”</blockquote><h4>Step 5: Causality</h4><p>This is where OCR 2 differs from prior query-based approaches.</p><p>DETR-style object queries and BLIP-2’s Q-Former treat queries as an unordered set. Queries attend bidirectionally to each other. Order does not matter.</p><p>In OCR 2, order is enforced.</p><p>Each causal flow token represents a step in a reading process. Later steps depend on earlier ones. This matches how document understanding actually works: headers inform paragraphs, table structure informs cell interpretation.</p><p>The model is not extracting a bag of features.<br>It is constructing a path through the document.</p><h4>Step 6: Parallel computation, sequential reasoning</h4><p>Even though causal masks are used, all tokens are computed in parallel during the forward pass.</p><p>The causality constrains information flow, not execution.</p><p>This is important because it allows the model to behave sequentially without sacrificing efficiency.</p><h4>Step 7: Feeding only the flow to the decoder</h4><p>Once the encoder finishes, <strong>only the causal flow tokens are passed to the decoder</strong>.</p><p>The raw visual tokens never reach the language model.</p><p>This is a deliberate constraint.</p><p>The decoder receives a sequence that already encodes:</p><ul><li>what parts of the image matter</li><li>in what order they should be read</li></ul><p>The decoder does what it is good at: generating text from a meaningful sequence.</p><h3>What changed from OCR 1 to OCR 2</h3><ul><li>CLIP-style vision encoder → LLM-style vision encoder</li><li>Direct flattening of visual tokens → learned reading-order construction</li><li>Query tokens as a set → causal flow tokens as a sequence</li><li>Encoder as feature extractor → encoder as reading-order reasoner</li></ul><p>The decoder remains largely unchanged.</p><h3>Output:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_6CcGWyiEURQ0iGmPi043w.png" /><figcaption><a href="https://arxiv.org/pdf/2602.01753">Input Image</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IgLjeS3bB9-JU94Dxl25NA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*52Gp1BudF5CQjaf4rBrJig.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qw2471VA5-KEHsSLlPqYEA.png" /><figcaption>Output images</figcaption></figure><h3>References:</h3><ol><li>Deepseek-OCR2: <a href="https://arxiv.org/pdf/2601.20552">https://arxiv.org/pdf/2601.20552</a></li><li><a href="https://github.com/deepseek-ai/DeepSeek-OCR-2">https://github.com/deepseek-ai/DeepSeek-OCR-2</a></li><li>Huggingface inference space: <a href="https://huggingface.co/spaces/prithivMLmods/DeepSeek-OCR-2-Demo/blob/main/deepseek_ocr_v2_demo.py">https://huggingface.co/spaces/prithivMLmods/DeepSeek-OCR-2-Demo/blob/main/deepseek_ocr_v2_demo.py</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=82e02e2608ad" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[10 years later: YOLO26]]></title>
            <link>https://medium.com/@nandinilreddy/10-years-later-yolo26-0c2291c75440?source=rss-6bc2b22a69b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/0c2291c75440</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[yolo26]]></category>
            <dc:creator><![CDATA[Nandini Lokesh Reddy]]></dc:creator>
            <pubDate>Wed, 21 Jan 2026 18:30:48 GMT</pubDate>
            <atom:updated>2026-01-21T18:30:48.024Z</atom:updated>
            <content:encoded><![CDATA[<p>It’s been a decade since YOLO first published.</p><p>The original YOLO paper came out in 2016 and fundamentally changed object detection. Until then, models like R-CNN and Faster R-CNN dominated the field accurate, but slow and computationally heavy. YOLO reframed detection as a single regression problem and proved that real-time object detection was not only possible, but practical.</p><p>Now, ten years later, YOLO26 has been released with a faster and more deployment-friendly architecture.</p><p>But while reading through the changes, something feels counterintuitive.</p><p>YOLO26 removes two components that made earlier versions successful:</p><ul><li><strong>Distribution Focal Loss (DFL): </strong>used for precise bounding box regression</li><li><strong>Non-Maximum Suppression (NMS)</strong>: the standard post-processing step for removing duplicate detections</li></ul><p>Instead, it introduces different mechanisms:</p><ul><li><strong>ProgLoss (Progressive Loss Balancing)</strong></li><li><strong>STAL (Small-Target-Aware Label Assignment)</strong></li><li><strong>MuSGD optimizer</strong></li></ul><p>this feels backwards.<br>If DFL and NMS worked so well, why remove them?</p><p>To understand that decision, it helps to go back to what made YOLO different in the first place.</p><h3>Why YOLO Was Different From R-CNN and Faster R-CNN</h3><p>R-CNN-style detectors treated object detection as two separate problems.</p><p>The Two-Step Pipeline:</p><ol><li><strong>Region Proposal</strong><br> Generate thousands of candidate regions where objects might exist.</li><li><strong>Classification</strong><br> Run a classifier on each proposed region to identify the object.</li></ol><p>This meant running the network hundreds or even thousands of times per image.<br>Accurate; but slow, expensive, and far from real time.</p><h3>YOLO:</h3><p>YOLO reframed detection as a <strong>single regression problem</strong>.</p><p>The image is divided into an S×SS \times SS×S grid (for example, 7×7).<br> Each grid cell predicts, in one forward pass:</p><ul><li>Bounding box coordinates (x, y, width, height)</li><li>Object confidence</li><li>Class probabilities</li></ul><p>One network pass. One look at the image. <br> It takes, ~45 FPS on a GPU at the time. Real-time object detection became feasible.</p><p>But this simplicity introduced a new problem.</p><h3>The Duplicate Detection Problem</h3><p>Because multiple grid cells can predict the same object, YOLO often produces <strong>duplicate bounding boxes</strong>.</p><p>For example, if a cat lies near the boundary of two grid cells, both may predict overlapping boxes. Instead of one detection, there might be five or six.</p><p>This is where <strong>Non-Maximum Suppression (NMS)</strong> became standard.</p><h3>How NMS Works</h3><ol><li>Sort predicted boxes by confidence score</li><li>Select the highest-confidence box</li><li>Compute IoU with all other boxes</li><li>Remove boxes with IoU above a threshold (commonly 0.5)</li><li>Repeat until no boxes remain</li></ol><p>But while NMS fixed the symptom, it introduced new problems especially for deployment.</p><h3>The Deployment Cost of NMS</h3><p>NMS is not part of the neural network itself. It’s procedural logic.</p><p>That causes several issues:</p><ul><li><strong>Unpredictable inference time</strong><br> More objects → longer NMS runtime. A crowded street takes longer than an empty room.</li><li><strong>Hard to accelerate</strong><br> GPUs and NPUs excel at matrix operations, not sorting, looping, and conditional logic.</li><li><strong>Manual threshold tuning</strong><br> Dense scenes require different IoU thresholds than sparse scenes.</li><li><strong>Export complexity</strong><br> Every deployment target (TensorRT, CoreML, ONNX, TFLite) needs a custom NMS implementation.</li></ul><p>NMS works, but it’s a patch applied <em>after</em> the network has already produced messy outputs.</p><p>But, <strong>Why is the network generating duplicate predictions in the first place?</strong></p><h3>Distribution Focal Loss (DFL):</h3><p>DFL was introduced later (YOLOv6, YOLOv8) to solve a different issue: <strong>bounding box precision</strong>.</p><p>Traditional regression predicts a single coordinate value:</p><blockquote><em>“The left edge is at x = 120.5”</em></blockquote><p>But in many cases: blurry images, occlusions, tiny objects, exact boundaries are uncertain.</p><h3>DFL:</h3><p>Instead of predicting a single value, predict a <strong>probability distribution</strong> over nearby positions.</p><p>Example:</p><ul><li>60% confidence → x = 120</li><li>30% confidence → x = 121</li><li>10% confidence → x = 119</li></ul><p>The final coordinate is computed by integrating over this distribution.</p><p>This improved:</p><ul><li>Small object localization</li><li>Occluded object detection</li><li>Performance on hard, ambiguous examples</li></ul><p>But again, accuracy came at a cost.</p><h3>The Deployment Cost of DFL</h3><p>DFL complicates inference in several ways:</p><ul><li><strong>Export difficulty</strong><br> Distributional operations don’t translate cleanly to ONNX, TensorRT, or CoreML.</li><li><strong>Extra computation</strong><br> Every coordinate now predicts a distribution, not a single value.</li><li><strong>Quantization challenges</strong><br> Distribution-based outputs don’t compress well to INT8.</li><li><strong>Graph complexity</strong><br> Larger, more complex computation graphs are harder to optimize on edge hardware.</li></ul><p>Once again, the pattern appears:</p><ul><li><strong>NMS</strong> → improves results, hurts deployment</li><li><strong>DFL</strong> → improves accuracy, hurts deployment</li></ul><p>Both are fixes applied <em>after</em> prediction.</p><h3>YOLO26’s Core Question</h3><p>What if the network were trained to produce <strong>clean predictions from the start</strong>?</p><p>Instead of:</p><blockquote><em>Generate noisy outputs → clean them afterward</em></blockquote><p>YOLO26 asks:</p><blockquote><em>Can training be structured so post-processing isn’t needed at all?</em></blockquote><p>This single shift explains most of YOLO26’s design decisions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KCc1hz2rxkBZWAYJFrI75w.png" /><figcaption>Source: <a href="https://arxiv.org/pdf/2509.25164">YOLO26</a></figcaption></figure><h3>Removing NMS: End-to-End Inference</h3><p>YOLO26 eliminates NMS by changing how predictions are learned.</p><h4>Dual Prediction Heads (Training Only)</h4><ul><li><strong>One-to-many head</strong><br> Generates multiple predictions per object<br> Provides rich supervision early in training</li><li><strong>One-to-one head</strong><br> Generates exactly one prediction per object<br> Matches inference behavior</li></ul><p>Both heads are used during training.<br>Only the <strong>one-to-one head</strong> is used during inference.</p><p>The network learns <em>not</em> to produce duplicates.</p><p><strong>Results:</strong></p><ul><li>Deterministic inference time (scene complexity no longer matters)</li><li>Pure neural-network execution (hardware friendly)</li><li>No IoU threshold tuning</li><li>No custom NMS code per platform</li></ul><p>Inference becomes clean, predictable, and deployable.</p><p>But removing NMS only works if training remains stable.</p><h4>Removing DFL: Simplified Regression</h4><p>YOLO26 returns to <strong>direct coordinate regression</strong>. No probability distributions. No integration steps.</p><p><strong>Benefits:</strong></p><ul><li>Clean export to ONNX, TensorRT, CoreML, TFLite, OpenVINO</li><li>Lower computational overhead</li><li>Straightforward FP32 → INT8 quantization</li><li>Simpler optimization graphs</li></ul><p>However, this raises an obvious concern:</p><p><strong>How is small-object accuracy preserved without DFL?</strong></p><p>That’s where training strategy replaces loss complexity.</p><h3>ProgLoss: Progressive Loss Balancing</h3><p>Training with dual heads introduces competing objectives. Weighting them equally from the start leads to instability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3Pd-3ILiQ8bUJPUuYeGltQ.png" /><figcaption>Yolov26 Architecture: Source <a href="https://arxiv.org/pdf/2509.25164">YOLO26</a></figcaption></figure><p><strong>ProgLoss</strong> solves this by dynamically adjusting loss weights over time.</p><p><strong>Early Training</strong></p><ul><li>One-to-many head: <strong>80%</strong></li><li>One-to-one head: <strong>20%</strong></li></ul><p>The model first learns broad object detection using rich supervision.</p><p><strong>Late Training</strong></p><ul><li>One-to-many head: <strong>20%</strong></li><li>One-to-one head: <strong>80%</strong></li></ul><p>Training gradually aligns with inference behavior.</p><p><strong>Outcome</strong></p><ul><li>Smoother convergence</li><li>Stable training across model sizes</li><li>Better final accuracy</li></ul><p>Instead of optimizing everything at once, the model learns in stages.</p><h3>STAL: Small-Target-Aware Label Assignment</h3><p>STAL replaces DFL’s small-object benefits, but through label assignment, not regression complexity.</p><p>During training, predictions must be matched to ground truth objects.</p><p><strong>Standard Matching</strong></p><ul><li>Highest IoU wins</li></ul><p><strong>STAL’s Adjustment</strong></p><ul><li>Identify small objects by pixel area</li><li>Adjust matching criteria to favor small-object predictions</li><li>Increase loss weight for small-object errors</li></ul><p>This forces the network to pay attention to small targets <em>during training</em>, not inference.</p><p>The result:</p><ul><li>Comparable small-object accuracy to DFL</li><li>Simpler inference graph</li></ul><h3>MuSGD:</h3><p>YOLO26 also changes how the model is optimized.</p><p>A hybrid that combines:</p><ul><li>SGD’s stability</li><li>Muon’s faster convergence on complex loss surfaces</li></ul><p><strong>Components:</strong></p><ul><li><strong>SGD</strong><br> Stable, well-understood, but slower to converge</li><li><strong>Muon</strong><br> A newer optimizer used in large language models (e.g., Kimi K2)<br> Incorporates curvature-aware updates and adaptive momentum<br> Fast convergence, but can overfit in vision tasks</li></ul><p><strong>Benefits:</strong></p><ul><li>Faster training</li><li>Stable across model scales</li><li>Improved final accuracy</li></ul><p>This is a direct example of LLM research influencing computer vision training.</p><h3>Implementation:</h3><pre>!uv pip install ultralytics<br>import ultralytics</pre><pre>from ultralytics import YOLO<br><br>model = YOLO(&#39;yolo26n.pt&#39;)  # load a pretrained YOLO model<br>model.predict(source=&#39;/content/12043-236265340_small.mp4&#39;, save=True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*0fRuwkb6K0MMVshu-MkZ6A.gif" /><figcaption>Yolo26n — result</figcaption></figure><p>This just took me less than 50seconds to run this.</p><h3>Is Simplification the Future?</h3><ul><li>Transformers replaced complex RNN stacks with simpler attention</li><li>ResNets replaced deep handcrafted designs with skip connections</li><li>YOLO26 replaces post-processing complexity with end-to-end learning</li></ul><p>Instead of adding more fixes after prediction, YOLO26 pushes complexity into <strong>training</strong>, where it belongs.</p><p>Maybe progress doesn’t always mean adding more features.</p><p>Maybe it means teaching models to do the right thing from the start, so nothing needs to be fixed afterward.</p><h3>References:</h3><ol><li>Yolo26: <a href="https://arxiv.org/pdf/2509.25164">https://arxiv.org/pdf/2509.25164</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0c2291c75440" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dolphin by Bytedance]]></title>
            <link>https://medium.com/@nandinilreddy/dolphin-by-bytedance-533629e0eb99?source=rss-6bc2b22a69b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/533629e0eb99</guid>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[document-parsing]]></category>
            <dc:creator><![CDATA[Nandini Lokesh Reddy]]></dc:creator>
            <pubDate>Wed, 07 Jan 2026 23:43:35 GMT</pubDate>
            <atom:updated>2026-01-07T23:43:35.875Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P4QlRuJ0gLgIfqfYHbKTeg.png" /><figcaption>Source: NotebookLM</figcaption></figure><p>As the year comes to a close, I ended up reading a lot of work on <strong>document parsing and image parsing, </strong>from LLM makers as well as enterprise research teams.</p><blockquote><a href="https://notebooklm.google.com/notebook/657e1d2e-711d-4c74-8dfb-ce83cd0dc929?artifactId=25b56f4a-c94d-4ff6-ac12-90c9e3435288">https://notebooklm.google.com/notebook/657e1d2e-711d-4c74-8dfb-ce83cd0dc929?artifactId=25b56f4a-c94d-4ff6-ac12-90c9e3435288</a></blockquote><p>I had already written about <strong>IBM’s gratine-docking</strong> approach. Beyond that, there are many other efforts in this space:</p><ul><li>Google’s DocAI and LayoutLM family</li><li>Donut-style end-to-end OCR-free models</li><li>Mathpix-like specialized pipelines</li><li>Recent multimodal LLMs that directly “read” PDFs and images</li><li>Enterprise tools like MinerU, Azure Form Recognizer, Amazon Textract</li></ul><p>All of them are trying to answer the same question:</p><blockquote>How do we turn real-world documents into something an LLM can reliably reason over?</blockquote><p>While reading these, one thought kept coming back:</p><p><strong>What is the next meaningful step in document parsing for LLMs?</strong></p><ul><li>Is it <strong>token reduction</strong>, because long documents are expensive?</li><li>Is it <strong>better GPU utilization</strong>, via parallelism?</li><li>Is it <strong>more accurate retrieval</strong>, especially for RAG?</li><li>Or is it something more fundamental, how we represent documents in the first place?</li></ul><p>With that question in mind, let’s start from the basics.</p><h3>Why Is Document Parsing So Important?</h3><p>Documents are everywhere:<br>reports, research papers, invoices, forms, contracts, scanned books.</p><p>But documents are not databases.</p><p>A document is not structured like JSON.<br> It is not even semi-structured like HTML.</p><p>Instead, it is usually:</p><ul><li>a <strong>2D layout</strong></li><li>containing <strong>text, images, tables, formulas</strong></li><li>with meaning encoded in <strong>spatial relationships</strong></li></ul><p>If we want LLMs to:</p><ul><li>search documents</li><li>answer questions from them</li><li>summarize them</li><li>use them in RAG pipelines</li></ul><p>then document parsing becomes the <strong>first and most critical step</strong>.</p><p>Bad parsing → bad retrieval → hallucinated answers.</p><h3>Why Is Document Parsing Hard for LLMs?</h3><p>At a high level, the problem is simple:</p><blockquote><strong><em>LLMs think in tokens. Documents exist in space.</em></strong></blockquote><p>A document page is an image.<br>An LLM expects a sequence.</p><p>This mismatch causes several concrete problems.</p><h4>1. Documents Are Not Linear</h4><p>A document page has:</p><ul><li>multiple columns</li><li>floating figures</li><li>side notes</li><li>footnotes</li><li>captions referring to images elsewhere</li></ul><p>There is no obvious “correct” reading order unless you understand layout.</p><p>LLMs do not naturally understand:</p><ul><li>which block comes first</li><li>what text belongs to which table</li><li>whether a caption explains an image or a paragraph</li></ul><h4>2. Documents Contain Heterogeneous Content</h4><p>A page may include:</p><ul><li>natural language text</li><li>tables (rows + columns)</li><li>mathematical formulas</li><li>diagrams or charts</li><li>scanned handwritten text</li></ul><p>Each of these needs <strong>different parsing logic</strong>.</p><p>Treating everything as plain text:</p><ul><li>breaks tables</li><li>destroys formulas</li><li>loses semantic structure</li></ul><h4>3. Visual Quality Varies</h4><p>Real documents are messy:</p><ul><li>low-resolution scans</li><li>skewed photos</li><li>shadows</li><li>watermarks</li><li>handwriting</li></ul><p>OCR errors propagate downstream.<br>Once structure is lost, LLMs confidently hallucinate.</p><h3>Current Approaches to Document Parsing</h3><p>Broadly, today’s systems fall into two main categories.</p><h4>Approach 1: Assembling Specialized Expert Models</h4><p>This is the traditional enterprise approach.</p><p>You build a pipeline:</p><ol><li>Layout detection model</li><li>OCR model</li><li>Table recognition model</li><li>Formula recognition model</li><li>Post-processing + heuristics</li></ol><p>Each component is an <strong>expert model</strong>, trained for one task.</p><p><strong>Why this works</strong></p><ul><li>Each model is very good at its job</li><li>High accuracy for known document types</li></ul><p><strong>Why this breaks</strong></p><ul><li>Pipelines are fragile</li><li>Errors cascade</li><li>Maintaining many models is expensive</li><li>Hard to share global context</li></ul><p>If the layout detector mislabels a table as text, everything downstream fails.</p><p>This approach scales poorly as document variety increases.</p><h4>Approach 2: Autoregressive Page-Level Generation</h4><p>Recent vision-language models take a different route.</p><p>They:</p><ul><li>encode the full page image</li><li>directly generate structured output (Markdown / JSON / HTML)</li><li>token by token, autoregressively</li></ul><p>Examples include:</p><ul><li>Donut</li><li>some LayoutLM successors</li><li>newer multimodal LLMs</li></ul><p><strong>Why this is appealing</strong></p><ul><li>End-to-end learning</li><li>Fewer moving parts</li><li>Simpler deployment</li></ul><p><strong>But there are serious drawbacks</strong></p><ol><li><strong>Inefficiency</strong><br> Long documents → long token sequences → high cost.</li><li><strong>Sequential bottleneck</strong><br> Autoregressive generation cannot be parallelized.</li><li><strong>Structure drift</strong><br> As generation progresses, layout consistency degrades.<br> Tables slowly “flatten” into text.</li><li><strong>Hard to specialize</strong><br> The same decoder is forced to handle text, tables, formulas, and images equally.</li></ol><p>This is where Dolphin takes a different path.</p><p>ByteDance’s <strong>Dolphin</strong> proposes a simple but powerful idea:</p><blockquote><strong><em>Do not parse content until you understand structure.</em></strong></blockquote><p>Instead of generating everything in one pass, Dolphin uses a <strong>two-stage design</strong>.</p><p>Dolphin is <strong>not a classical OCR system</strong> and not a collection of small models.<br> It is built on a <strong>single multimodal foundation model</strong>.</p><p>Specifically, Dolphin is built on the <strong>Qwen2.5-VL-3B backbone</strong>, with:</p><ul><li>a <strong>vision encoder based on Native Resolution Vision Transformer (NaViT)</strong></li><li>an <strong>autoregressive Transformer decoder</strong> used for structured output generation</li></ul><p>This backbone is shared across <strong>both stages, </strong>layout analysis and content parsing. What changes between stages is <strong>the objective and the prompting</strong>, not the model itself.</p><h4>Stage 1: Layout Analysis (Structure First)</h4><p>In the first stage, Dolphin looks at the document image and produces:</p><ul><li>a <strong>sequence of layout elements</strong></li><li>in <strong>natural reading order</strong></li></ul><p>Each element is:</p><ul><li>a bounding region</li><li>with a semantic type (paragraph, table, formula, figure, etc.)</li></ul><p>These are called <strong>heterogeneous layout elements</strong>.</p><p>At this point:</p><ul><li>no OCR text is generated</li><li>no table content is extracted</li><li>no formulas are decoded</li></ul><p>Only <strong>structure</strong> is identified.</p><p>Think of this as building a <strong>skeleton of the document</strong>.</p><p><strong>Why This Matters</strong></p><p>Once structure is known:</p><ul><li>reading order is fixed</li><li>element boundaries are clear</li><li>each region can be handled independently</li></ul><p>This avoids guessing layout while parsing content.</p><h4>Stage 2: Parallel Content Parsing</h4><p>In the second stage, each layout element becomes an <strong>anchor</strong>.</p><p>For each anchor:</p><ul><li>a task-specific prompt is created</li><li>content is parsed <strong>independently and in parallel</strong></li></ul><p>Examples:</p><ul><li>paragraph → OCR + text normalization</li><li>table → structured table extraction</li><li>formula → LaTeX generation</li><li>figure → caption extraction or description</li></ul><p>This is called <strong>heterogeneous anchor prompting</strong>.</p><p><strong>What Are Heterogeneous Anchor Prompts?</strong></p><p>They are simply:</p><blockquote><strong><em>Different instructions for different element types</em></strong><em>, applied to the same underlying model.</em></blockquote><p>Instead of training many expert models:</p><ul><li>Dolphin uses one foundation model</li><li>behavior is controlled via prompts + anchors</li></ul><p>This gives:</p><ul><li>specialization without fragmentation</li><li>flexibility without pipelines</li></ul><p>After describing the two-stage <em>analyze → parse</em> process, a natural question comes up:</p><blockquote>Okay, but what models or algorithms are actually doing this?<br> Is this classical CV? Transformers? OCR? Something new?</blockquote><p>The short answer is: <strong>Dolphin is still a Transformer-based vision-language model</strong>, but the <em>way it is trained and prompted</em> is what makes it different</p><p>Let’s break it down stage by stage</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2YiyPgK644d2ZIry6ANQ4A.png" /><figcaption>Source: Dolphin</figcaption></figure><h4>Stage 1: Layout Analysis, How Does Dolphin Understand Structure?</h4><p><strong>Core Technique: Vision Transformer + Sequence Prediction</strong></p><p>In stage 1, Dolphin treats <strong>layout understanding as a structured prediction problem</strong>, not a detection-only problem.</p><p>Instead of:</p><ul><li>just predicting bounding boxes (like YOLO-style detectors), or</li><li>just labeling pixels (like segmentation models),</li></ul><p>Dolphin predicts a <strong>sequence of layout elements in reading order</strong>.</p><p>This is important.</p><h4>What model is used?</h4><ul><li>A <strong>Vision Transformer (ViT-like encoder)</strong> processes the document image.</li><li>A <strong>Transformer decoder</strong> predicts layout elements as a <em>sequence</em>, similar to how language models generate text.</li></ul><p>Each predicted layout element includes:</p><ul><li>element type (paragraph, table, formula, figure, etc.)</li><li>spatial coordinates (bounding box)</li><li>ordering information (implicit via sequence position)</li></ul><p>So layout detection becomes:</p><blockquote>“Given this page image, generate an ordered list of structured layout tokens.”</blockquote><p>This is closer to <strong>image-to-structure generation</strong> than classical object detection.</p><h4>Why Not Just Use Object Detection?</h4><p>Traditional document layout models do this:</p><ul><li>detect boxes independently</li><li>classify each box</li></ul><p>But that has problems:</p><ul><li>reading order must be inferred later</li><li>relationships between elements are weak</li><li>downstream parsing becomes messy</li></ul><p>Dolphin avoids this by <strong>learning reading order directly during generation</strong>.</p><h4>Stage 2: Content Parsing, How Is Each Element Parsed?</h4><p>Once layout anchors are produced, Dolphin switches modes.</p><p>Now the question is no longer:</p><blockquote>“What is on the page?”</blockquote><p>It becomes:</p><blockquote>“Given this region and this instruction, parse the content.”</blockquote><p>Stage 2 uses the <strong>same underlying vision-language model</strong>, but with:</p><ul><li>a cropped image region (from the layout box)</li><li>a <strong>task-specific prompt</strong></li></ul><p>Examples:</p><ul><li>“Extract the text from this paragraph.”</li><li>“Convert this table into HTML.”</li><li>“Generate LaTeX for this formula.”</li></ul><p>This is <strong>prompt-based specialization</strong>, not model-based specialization.</p><h4>Is This OCR? Is This NLP?</h4><p>It’s both, but unified.</p><p>Instead of:</p><ul><li>OCR → post-processing → heuristics,</li></ul><p>Dolphin treats content parsing as:</p><blockquote>conditional generation from visual input.</blockquote><p>The model learns to:</p><ul><li>recognize text visually</li><li>preserve structure</li><li>output in the requested format</li></ul><p>This is especially powerful for:</p><ul><li>tables</li><li>formulas</li><li>mixed content</li></ul><p><strong>Why Dolphin Is Technically Interesting</strong></p><p><strong>1. Parallelism</strong></p><p>Elements are parsed independently.<br> This improves:</p><ul><li>GPU utilization</li><li>latency</li><li>throughput</li></ul><p><strong>2. Token Efficiency</strong></p><p>Each element is small.<br>No need to feed the full page repeatedly.</p><p><strong>3. Structural Fidelity</strong></p><p>Layout is preserved explicitly.<br>Downstream systems receive clean, ordered structure.</p><p><strong>4. Better RAG Inputs</strong></p><p>Instead of flat text, you get:</p><ul><li>structured sections</li><li>tables as tables</li><li>formulas as formulas</li></ul><p>This dramatically improves retrieval quality.</p><h3>What Dolphin Does Not Fully Solve</h3><ul><li>Highly creative or artistic layouts</li><li>Deep semantic understanding beyond structure</li><li>Cross-page reasoning (still an open problem)</li></ul><p>But it moves the field in the <strong>right direction</strong>.</p><h3>References:</h3><ol><li><a href="https://arxiv.org/pdf/2505.14059">https://arxiv.org/pdf/2505.14059</a></li><li><a href="https://arxiv.org/pdf/2307.06304">https://arxiv.org/pdf/2307.06304</a></li><li><a href="https://github.com/bytedance/Dolphin">https://github.com/bytedance/Dolphin</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=533629e0eb99" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Total segmentator]]></title>
            <link>https://medium.com/@nandinilreddy/total-segmentator-e109388df59b?source=rss-6bc2b22a69b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/e109388df59b</guid>
            <category><![CDATA[nnunet]]></category>
            <category><![CDATA[segmentation]]></category>
            <category><![CDATA[aimed]]></category>
            <dc:creator><![CDATA[Nandini Lokesh Reddy]]></dc:creator>
            <pubDate>Tue, 16 Dec 2025 04:08:03 GMT</pubDate>
            <atom:updated>2025-12-16T04:08:03.976Z</atom:updated>
            <content:encoded><![CDATA[<h3>Total Segmentator</h3><p>Image segmentation is one of the most fundamental ideas in computer vision. At its core, it’s just about dividing an image into meaningful regions so we can understand <em>what exactly is inside</em>. In medical applications, segmentation forms the backbone of accurate analysis, diagnosis, and decision-making.</p><p>Segmentation gives structure to raw CT scans. It pulls out organs, bones, vessels, and soft tissues that doctors need to measure, analyze, or monitor. Without this, medical imaging is just grayscale data. With segmentation, it becomes a map of human anatomy.</p><p>And out of all the open-source models I’ve worked with, <strong>TotalSegmentator</strong> is easily one of the best. It’s clean, reliable, detailed, and covers almost the entire human anatomy; all with a single model.</p><p>Let’s start with:<br>dataset, annotations, design choices, model architecture, and training pipeline.</p><h3>Dataset</h3><p>The TotalSegmentator model was trained on a large and diverse set of real-world CT examinations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vW_KEBQRx5iNbjXh8wkizQ.png" /><figcaption>Source: <a href="https://pubs.rsna.org/doi/full/10.1148/ryai.230024">TotalSegmentator</a></figcaption></figure><ul><li><strong>1,204 CT examinations</strong></li><li>Sourced from routine clinical workflows</li><li>Includes wide variation in scanner types, patient conditions, contrast phases, and body coverage</li><li>Covers head to knee regions across many demographics</li></ul><p>This diversity is one of the reasons the model generalizes so well.</p><h3>Annotations</h3><p>This is where the method truly stands out.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/818/1*7aIn2El63Z5vIHnJBgr8_w.png" /><figcaption>Source: <a href="https://pubs.rsna.org/doi/full/10.1148/ryai.230024">TotalSegmentator</a></figcaption></figure><p>The authors manually annotated <strong>104 anatomical structures</strong>, including:</p><ul><li><strong>27 organs</strong></li><li><strong>59 bones</strong></li><li><strong>10 muscles</strong></li><li><strong>8 vessels</strong></li></ul><p>Very few open-source datasets offer this level of granularity in 3D medical segmentation. This richness directly increases the model’s usefulness for radiology, surgery planning, organ measurement, and research.</p><h3>Training Strategy</h3><p>TotalSegmentator builds on the philosophy of <strong>nnU-Net</strong>, which automatically configures the entire training pipeline based on the dataset.</p><p>This includes:</p><ul><li>patch size</li><li>batch size</li><li>learning rate</li><li>normalization strategy</li><li>data augmentation</li><li>loss functions</li><li>network depth and architecture configurations</li></ul><p>The point is:<br> <strong>the model configures itself.</strong><br> You don’t need manual hyperparameter tuning.</p><p><strong>how can we achieve that?</strong></p><p>Instead of relying on hand-tuned hyperparameters, the framework first <strong>analyzes the dataset</strong> and derives optimal settings directly from it.</p><p>Concretely, the process looks like this:</p><p><strong>Dataset fingerprinting</strong><br> The model scans the training data to understand:</p><ul><li>Image dimensions and spacing (voxel resolution)</li><li>Intensity distributions</li><li>Number of samples and class imbalance</li><li>Memory constraints imposed by the data</li></ul><p><strong>Rule-based configuration</strong><br> Based on this fingerprint, nnU-Net applies a set of empirically validated rules to decide:</p><ul><li>Patch size that maximizes context while fitting GPU memory</li><li>Batch size that balances stability and efficiency</li><li>Network depth and number of feature maps</li><li>Appropriate normalization strategy</li><li>Loss functions suited for multi-class medical segmentation</li><li>Data augmentation tailored to the dataset’s spatial properties</li></ul><p><strong>Automatic architecture selection</strong><br> Depending on image resolution and anisotropy, the framework chooses between 2D, 3D full-resolution, or 3D cascade variants; without user intervention.</p><p>The result is that <strong>TotalSegmentator does not require manual hyperparameter tuning</strong>.<br> The model adapts itself to the dataset in a structured, reproducible way, allowing researchers and clinicians to focus on application rather than optimization.</p><h3>Resolution Settings</h3><p>The authors trained two models:</p><ol><li><strong>1.5 mm isotropic resolution</strong></li></ol><ul><li>Higher precision</li><li>More computationally expensive</li></ul><p><strong>2. 3 mm isotropic resolution</strong></p><ul><li>Lighter, faster</li><li>Works on lower-end hardware</li></ul><p>Both are released open-source, making the model accessible to beginners and practitioners without heavy GPUs.</p><h3>Architecture (U-Net + nnU-Net principles)</h3><p>TotalSegmentator is built on a <strong>3D U-Net–style encoder–decoder architecture</strong>, a design that has become the standard for volumetric medical image segmentation. However, the strength of the system does not come from a novel architecture, but from <strong>how that architecture is automatically adapted to the data</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MVemYAVojW0GfvmiIrh3Bg.png" /><figcaption>Source: <a href="https://medium.com/miccai-educational-initiative/nnu-net-the-no-new-unet-for-automatic-segmentation-8d655f3f6d2a">medium article</a></figcaption></figure><p>The network follows a classic encoder–decoder structure:</p><ul><li><strong>Encoder (downsampling path)</strong><br> Progressively captures higher-level, semantic features while reducing spatial resolution. This allows the model to understand large anatomical context.</li><li><strong>Decoder (upsampling path)</strong><br> Gradually restores spatial resolution to produce dense, voxel-wise segmentation maps.</li><li><strong>Skip connections</strong><br> Feature maps from the encoder are passed directly to the decoder at corresponding scales, preserving fine-grained spatial details that are critical for accurate anatomical boundaries.</li><li><strong>3D convolutions throughout</strong><br> All operations are performed in 3D, enabling the model to fully leverage volumetric context in CT scans rather than treating slices independently.</li></ul><p>What makes this architecture powerful in TotalSegmentator is <strong>nnU-Net’s automated design logic</strong>. Instead of fixing architectural choices upfront, nnU-Net derives them from the dataset itself:</p><ul><li>Network depth is adjusted based on image resolution and field-of-view</li><li>Kernel sizes and strides are adapted to voxel spacing and anisotropy</li><li>Feature map widths scale with available GPU memory</li><li>Input patch size is chosen to balance global context and computational feasibility</li></ul><p>As a result, <strong>there is no hand-crafted architecture tuning</strong>.<br> The same U-Net blueprint is dynamically reshaped to match the data’s geometry, spacing, and statistical properties.</p><p>This data-driven architectural adaptation is what allows TotalSegmentator to remain robust across diverse CT datasets while maintaining high anatomical precision at scale.</p><h3>Implementation:</h3><p>This is the implementation of TotalSegmentator on CPU</p><pre>pip install Totalsegmentator<br>import nibabel as nib<br>from totalsegmentator.python_api import totalsegmentator<br><br>input_path = &quot;Case1.nii.gz&quot;<br>output_path = &quot;Case1&quot;<br>totalsegmentator(input_path, output_path)</pre><pre>import nibabel as nib<br>import numpy as np<br>import matplotlib.pyplot as plt<br>import imageio<br>from io import BytesIO<br>from IPython.display import display, Image<br><br>file_path = &#39;Case1/liver.nii.gz&#39;<br>img = nib.load(file_path)<br>data = img.get_fdata()<br><br>images = []<br><br>segmented_slices = []<br>for i in range(data.shape[2]):  <br>    if np.any(data[:, :, i] != 0):  <br>        segmented_slices.append(i)<br>        plt.figure(figsize=(5, 5))<br>        plt.imshow(data[:, :, i], cmap=&#39;gray&#39;)<br>        plt.axis(&#39;off&#39;)  <br>        buf = BytesIO()<br>        plt.savefig(buf, format=&#39;png&#39;)<br>        buf.seek(0)<br>        images.append(imageio.imread(buf)) <br>        buf.close()<br>        plt.close()<br><br><br>if images:<br>    gif_buffer = BytesIO()<br>    imageio.mimsave(gif_buffer, images, format=&#39;GIF&#39;, duration=0.5)<br>    gif_buffer.seek(0)<br><br>    display(Image(data=gif_buffer.read(), format=&#39;png&#39;))<br>    print(&quot;GIF displayed with segmented slices.&quot;)<br>else:<br>    print(&quot;No segmented slices found in the data.&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*Ig4wQJE6FoJwxRMCkrLI5w.png" /><figcaption>Original Data source: TotalSegmentator — Brain Segmentation</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*gBCblXytKWsruZRWeOHUug.png" /><figcaption>Original Data source: Totalsegmentator — Skull Segmentation:</figcaption></figure><p>for non-commercial license model, there are more in-depth structures like coronary arteries, heart chambers highres, appendicular bones, tissue types, brain, vertebrae body, face, thigh shoulder muscles and brain aneurysm</p><h3>Performance</h3><p>On its validation set, TotalSegmentator achieves a strong:</p><ul><li><strong>Mean Dice score ≈ 0.94</strong></li></ul><p>This is extremely high considering the diversity and number of structures.</p><p>Some structures naturally perform higher (large organs), while thin or small structures are more challenging — but overall, the model shows very consistent performance.</p><h3>Limitations &amp; Failure Modes</h3><ul><li>Very small structures can be harder to segment reliably.</li><li>Severe pathologies or post-operative anatomy may deviate from training distribution.</li><li>CT scans with very unusual fields-of-view can lead to incomplete predictions.</li><li>Bone-related predictions are affected by metal artifacts.</li><li>Performance may vary across institutions with different acquisition protocols.</li></ul><p>But for an open-source tool, the robustness is genuinely impressive.</p><h3>References:</h3><ol><li>full code: <a href="https://github.com/NandiniLReddy/TotalSegmentator/tree/main/TotalSegmentator">https://github.com/NandiniLReddy/TotalSegmentator/tree/main/TotalSegmentator</a></li><li>total-segmentator: <a href="https://github.com/wasserth/TotalSegmentator?tab=readme-ov-file">https://github.com/wasserth/TotalSegmentator?tab=readme-ov-file</a></li><li>nn-UNet:<a href="https://github.com/MIC-DKFZ/nnUNet">https://github.com/MIC-DKFZ/nnUNet</a></li><li>Technical Paper: <a href="https://pubs.rsna.org/doi/full/10.1148/ryai.230024">https://pubs.rsna.org/doi/full/10.1148/ryai.230024</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e109388df59b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LangChain <> LangGraph: Chains and Cycles]]></title>
            <link>https://medium.com/@nandinilreddy/langchain-langgraph-chains-and-cycles-e619d3dc0dbc?source=rss-6bc2b22a69b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/e619d3dc0dbc</guid>
            <category><![CDATA[langgraph]]></category>
            <category><![CDATA[prompt-chaining]]></category>
            <category><![CDATA[langchain]]></category>
            <dc:creator><![CDATA[Nandini Lokesh Reddy]]></dc:creator>
            <pubDate>Wed, 03 Dec 2025 14:02:57 GMT</pubDate>
            <atom:updated>2025-12-03T14:02:57.157Z</atom:updated>
            <content:encoded><![CDATA[<p>I was recently discussing the tech stack of our AI agent environment with a friend. I mentioned that we use both <strong>LangGraph</strong> and <strong>LangChain</strong> depending on the specific requirements.</p><p>She paused and asked, <em>“Why are you using two tools for the same purpose? Aren’t they basically the same thing with different marketing keywords?”</em></p><p>It was a valid question. To the outside observer, these tools look incredibly similar. That conversation made me realize this confusion is actually quite common, so decided to write this down to share the explanation, While it is partially true that they share DNA, their core purposes are fundamentally different.</p><p>To understand why we need both, we have to step back and look at the evolution of Large Language Models (LLMs) and what it actually takes to build a truly “intelligent” Agent.</p><h3>At Level 0: LLM is Not an Agent</h3><p>Let’s start with a hard truth: <strong>An LLM itself is not an agent.</strong></p><p>An LLM operating on its own is restricted to its pre-trained knowledge. It has no memory of past conversations, no access to real-time data, and no ability to interact with the world. Its strength lies in explaining established concepts, but it cannot “do” anything.</p><h4>So, What makes an Agent?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LHzml-zlim7PDaSM4qRepQ.png" /><figcaption>Source: Gemini</figcaption></figure><p>We generally define an agent as an intelligent system that figures out a path and takes action. A true agent must be capable of these five steps:</p><ol><li><strong>Get the Mission:</strong> Receive an objective.</li><li><strong>Scan the Mission:</strong> Analyze the requirements.</li><li><strong>Think it Through:</strong> Plan a strategy.</li><li><strong>Take Action:</strong> Execute steps using tools.</li><li><strong>Learn and Get Better:</strong> Refine the approach based on feedback.</li></ol><p>To turn a “Level 0” LLM into an agent, we have to give it “wings”: access to databases, external knowledge, and the ability to execute code. But how do we coordinate those wings?</p><h4>Prompt Engineering to Chaining</h4><p>To make an agent function, we perform <strong>Prompt Engineering</strong>. This is essentially paraphrasing instructions to ensure the agent operates with optimal guidance without human intervention.</p><p>However, we quickly hit a wall. LLMs struggle when we dump a massive amount of information on them in one go. They get confused, hallucinations increase, and accuracy drops.</p><h4>“Prompt Chaining”</h4><p>To solve this, we adopted a <strong>Divide and Conquer</strong> strategy.</p><p>Instead of one monolithic step, we break the original problem into subsequent sub-problems. Each problem is addressed individually through specifically designed prompts, and the output of one prompt becomes the input (context) for the next.</p><p>The Benefits: This adds modularity, reduces cognitive load, and lowers ambiguity.</p><p>The Drawback: Prompt chaining is like “series lighting.” It is highly dependent on the integrity of the data passed between steps. If one prompt is poorly managed, the subsequent prompts fail.</p><p>To manage this delicate flow of data, we need <strong>specialized frameworks</strong>.</p><p>While tools like CrewAI and Google ADK exist, LangChain and LangGraph are the heavy hitters for specialized control. Here is how they differ.</p><h4>1. LangChain:</h4><p>LangChain provides the foundational abstractions for <strong>linear sequences</strong>.</p><p>Think of LangChain as a straight highway. It is excellent for “DAGs” (Directed Acyclic Graphs) — processes that move from Point A to Point B to Point C without looking back.</p><ul><li><strong>Best for:</strong> Summarization pipelines, simple RAG (Retrieval Augmented Generation), and straight-line prompt chains.</li><li><strong>Limitation:</strong> It struggles when the agent needs to loop back, retry a step, or change its mind based on new data.</li></ul><h4>2. LangGraph:</h4><p>LangGraph extends the capabilities of LangChain to support <strong>stateful and cyclical computations</strong>.</p><p>Real-world problems aren’t always linear lines; they are loops. An agent might need to write code, test it, see an error, and rewrite the code.</p><ul><li><strong>The “Loop”:</strong> LangGraph allows the agent to Think → Act → Observe → Think Again.</li><li><strong>State Management:</strong> It remembers the “state” of the conversation or task across these loops.</li></ul><h4>“Can’t LangChain perform loops too?”</h4><p>This is the most common follow-up question. Ideally, the answer is <strong>Yes, but…</strong></p><p>You <em>can</em> perform loops in standard LangChain, but it is like using a screwdriver to hammer in a nail. You can do it, but it’s clumsy and not what the tool was designed for.</p><ol><li><strong>The “Hidden” Loop:</strong> LangChain’s AgentExecutor has a built-in while loop, but it is a &quot;black box.&quot; You cannot easily intervene, pause it, or control the logic inside that loop.</li><li><strong>The “Manual” Loop:</strong> You can wrap a LangChain chain in a standard Python for loop. However, the chain itself doesn&#39;t &quot;know&quot; it&#39;s looping. It loses its memory (state) every time the loop restarts unless you manually stitch it back together.</li></ol><p><strong>LangGraph</strong> treats the loop as a first-class citizen. It explicitly understands that going back to “Start” is a valid move, and it preserves the memory of <em>why</em> we went back.</p><h3>Example:</h3><p>LangChain implementation:</p><pre>import os<br>from langchain_openai import ChatOpenAI<br>from langchain_core.prompts import ChatPromptTemplate<br>from langchain_core.output_parsers import StrOutputParser</pre><pre>from dotenv import load_dotenv<br>load_dotenv()</pre><pre>llm = ChatOpenAI(temperature=0, model=&quot;gpt-3.5-turbo&quot;)</pre><pre>#prompt-1: Extract info<br>prompt_extract = ChatPromptTemplate.from_template(&quot;Trandform the following specifications from the following text: \n\n{text_input}&quot;)</pre><pre>#prompt-2: Transform to JSON<br>prompt_transform = ChatPromptTemplate.from_template(&quot;Transform the following text into a JSON object with &#39;cpu&#39;, &#39;memory&#39;, and &#39;storage&#39; as keys: \n\n{specifications}&quot;)</pre><pre>#building chain using LCEL(Langchain expression language)<br>extraction_chain = prompt_extract | llm | StrOutputParser()<br>#the StrOutputParser() converts the LLM&#39;s message output into a simple string.</pre><pre>full_chain = (<br>    {&quot;specifications&quot;: extraction_chain}<br>    | prompt_transform<br>    | llm<br>    | StrOutputParser()<br>)<br>#the full chain passes the output of extraction chain into the &#39;specifications&#39;</pre><pre>#let&#39;s run<br>input_text = &quot;The new laptop model features a 3.5 Ghz octa-core processor, 16GB of RAM, and a 1TB NVMe SSD.&quot;</pre><pre>final_result = full_chain.invoke({&quot;text_input&quot;: input_text})</pre><pre>## output:<br><br>{<br>    &quot;cpu&quot;: &quot;3.5 Ghz octa-core&quot;,<br>    &quot;memory&quot;: &quot;16GB&quot;,<br>    &quot;storage&quot;: &quot;1TB NVMe SSD&quot;<br>}</pre><p>Access full code here: <a href="https://github.com/NandiniLReddy/Teaching_AgenticDesign101">https://github.com/NandiniLReddy/Teaching_AgenticDesign101</a></p><h3>Summary</h3><p>So, to answer my friend’s original question: No, they are not the same tool.</p><p>We use <strong>LangChain</strong> when we know the exact steps the application needs to take (Step 1 →Step 2 → Step 3).</p><p>We use <strong>LangGraph</strong> when we need the AI to figure out the steps itself, make mistakes, and correct them in a loop (Plan → Execute →Error → Re-Plan).</p><p>To build a truly intelligent ecosystem, you often need both the stability of chains and the flexibility of graphs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e619d3dc0dbc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Qwen-3]]></title>
            <link>https://medium.com/@nandinilreddy/qwen-3-bba7433c77af?source=rss-6bc2b22a69b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/bba7433c77af</guid>
            <category><![CDATA[ovod]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[qwen-3]]></category>
            <dc:creator><![CDATA[Nandini Lokesh Reddy]]></dc:creator>
            <pubDate>Sat, 22 Nov 2025 16:20:13 GMT</pubDate>
            <atom:updated>2025-11-22T16:20:13.777Z</atom:updated>
            <content:encoded><![CDATA[<p>Large Language Models have been growing in two directions for years:<br><strong>more data</strong> and <strong>more parameters</strong>. But the real question we keep coming back to is: <em>will scaling alone take us to AGI?</em></p><blockquote><a href="https://notebooklm.google.com/notebook/18cda02b-4319-4634-9897-7cd90a3320c9?artifactId=674da2cc-5b00-404c-b9fa-84128dd73957">https://notebooklm.google.com/notebook/18cda02b-4319-4634-9897-7cd90a3320c9?artifactId=674da2cc-5b00-404c-b9fa-84128dd73957</a></blockquote><p>The Qwen-3 series takes a different approach. Yes, it scales from <strong>0.6B → 235B parameters</strong>, but it also introduces something much more important:</p><ul><li>a <strong>thinking module</strong></li><li>a <strong>non-thinking module</strong></li><li>a <strong>unified framework</strong> that can <em>switch dynamically</em> based on your query</li><li>and a new <strong>thinking-budget mechanism</strong> that lets you decide how much computation the model should spend on reasoning.</li></ul><p>This is a major move toward “<em>intelligent compute allocation</em>,” instead of just brute-force parameter scaling.</p><h4>Why Qwen-3 Matters Right Now?</h4><p>Every frontier model is racing toward AGI. The playbook for years has been:<br> <strong>“Train on trillions of tokens, hope intelligence emerges.”</strong></p><p>But Qwen-3 focuses on a different bottleneck, <em>inference-time reasoning</em>.</p><blockquote>How much smarter can a model get if we let it spend more time thinking on-demand?</blockquote><p>Instead of always responding fast or always reasoning deeply, Qwen-3 allows <strong>dynamic switching</strong>.<br> You choose:</p><ul><li>/think → full reasoning mode</li><li>/no-think → fast, succinct mode</li></ul><p>or set a <strong>thinking budget</strong>, like:</p><ul><li><em>“Use up to 200 thinking tokens.”</em></li></ul><p>This is a big step because humans don’t think the same way for every task and now LLMs don’t have to either.</p><h4>Architecture: Dense + MoE Models Together</h4><p>Qwen-3 includes:</p><ul><li><strong>6 Dense Models</strong></li><li><strong>2 MoE Models</strong></li></ul><p>The MoE version uses:</p><ul><li><strong>128 experts</strong></li><li><strong>8 activated experts per token</strong></li></ul><p>This is different from Qwen-2, which reused (“shared”) some experts. Qwen-3 encourages specialization by adding <strong>global batch load balancing loss</strong>, making each expert good at a different thing.</p><p>so what are these experts and activated experts?</p><p>let’s say we have 128 teachers (experts).<br>For each token, Qwen-3 picks the <em>best 8 teachers</em> to give an answer.<br>This makes the model:</p><ul><li>cheaper than a full 128-teacher team</li><li>but smarter than a single-teacher system</li></ul><p>This is why MoE models scale so well.</p><h3>Technical concepts:</h3><p>These concepts were refined versions from Qwen-2.5:</p><p>1. Group Query Attention (GQA): Reduces memory use while maintaining strong attention quality.</p><p>2. Rotary Positional Embeddings (RoPE): Allows the model to understand positions of tokens, especially useful for longer sequences.</p><p>3. RMSNorm (Pre-Norm): Stabilizes training for deep transformers.</p><p>4. QK-Norm in Attention: Normalizes queries and keys to prevent unstable gradients.</p><p>Nothing exotic here, but together, these form a solid and scalable backbone.</p><h4>1. Unified Thinking Framework</h4><p>This is the part that makes Qwen-3 feel like a true “intelligent assistant.”</p><p><strong>Two modes:</strong></p><ol><li><strong>Non-thinking mode</strong></li></ol><ul><li>Direct answers</li><li>Zero chain-of-thought</li><li>Low latency</li><li>Great for tasks like summarization, rewriting, quick chat</li></ul><p><strong>2. Thinking mode</strong></p><ul><li>Full reasoning</li><li>Math, logic, coding</li><li>Multi-step explanations</li><li>Uses chain-of-thought internally</li></ul><h4>2. Thinking Budget</h4><p>User: <em>“Solve this math problem. Use 100 tokens to think.”</em><br> Model:</p><ul><li>Allocates exactly 100 internal reasoning tokens</li><li>Produces a clean final answer</li></ul><p>This is basically <strong>“pay only for the intelligence you need.”</strong></p><h4>Pre-Training (3-Stage Process)</h4><p>Qwen-3 is trained on <strong>36 trillion tokens</strong> in <strong>119 languages, </strong>double the previous generation.</p><p><strong>Stage 1: General Pre-Training</strong></p><ul><li>30T tokens</li><li>sequence length 4,096</li><li>builds core language + world knowledge</li><li>multilingual understanding built from day 1</li></ul><p><strong>Stage 2: Reasoning Pre-Training</strong></p><ul><li>adds +5T high-quality STEM + logic + coding + synthetic data</li><li>accelerates learning-rate decay</li><li>pushes the model deeper into reasoning territory</li></ul><p><strong>Stage 3: Long-Context Pre-Training</strong></p><p>Goal: context length → <strong>32,768</strong></p><p>Dataset:</p><ul><li>75% = 16k–32k token documents</li><li>25% = 4k–16k token documents</li></ul><p>Useful for:</p><ul><li>long PDFs</li><li>codebases</li><li>research papers</li><li>memory-heavy reasoning tasks</li></ul><h4>Post-Training:</h4><p>Two major pieces here:</p><p><strong>1. Thinking Control</strong></p><p>This includes:</p><ul><li>the ability to choose /think or /no-think</li><li>the ability to decide token-budget</li><li>consistent formatting of thinking vs final answer</li></ul><p>This gives the user full control over reasoning.</p><p><strong>2. Strong-to-Weak Distillation</strong></p><p>Why?<br> Because training a tiny 3B or 7B model with a massive reasoning pipeline is expensive.</p><p>Instead:</p><ul><li>train huge models thoroughly</li><li>then distill their <em>behavior + reasoning patterns</em> into small models</li><li>cost = <strong>10% of full training</strong></li></ul><p>This dramatically speeds up deployment of smaller models.</p><h4><strong>Distillation:</strong></h4><p>There are several stages:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bhQ6Ys6K-YiQix1I8kbkNg.png" /></figure><p><strong>1. Long CoT Cold Start</strong></p><p>Cold start = <em>building the initial reasoning dataset.</em></p><p><strong>Dataset includes:</strong></p><ul><li>math</li><li>logic</li><li>coding</li><li>general STEM</li><li>with verifiable answers or test cases</li></ul><p><strong>Two filtering steps:</strong></p><p><strong>(a) Query Filtering</strong></p><p>Removes:</p><ul><li>unverifiable questions</li><li>multi-part “too open” questions</li><li>general text generation</li><li>questions solvable <em>without</em> reasoning</li></ul><p>Goal → keep only <em>truly reasoning-heavy tasks</em>.</p><p><strong>(b) Response Filtering</strong></p><p>Remove responses that:</p><ol><li>give wrong answers</li><li>repeat too much</li><li>show guessing</li><li>mismatch thinking vs final</li><li>mix languages randomly</li><li>resemble validation tasks</li></ol><p>The final dataset is small but <em>highly precise</em> → used to teach the model <strong>how</strong> to reason, not to maximize accuracy yet.</p><h4>2. Reasoning RL</h4><p>Uses new query–verifier pairs with rules:</p><ul><li>not used in cold-start</li><li>solvable</li><li>challenging</li><li>diverse</li></ul><p>Training techniques:</p><ul><li>large batch sizes</li><li>many rollouts</li><li>off-policy training for efficiency</li><li>entropy control to balance exploration/exploitation</li></ul><p>This step dramatically boosts reasoning performance.</p><h4>3. Thinking Mode Fusion</h4><p>Fuse non-thinking and thinking into one model via SFT and a special chat template.</p><p>Now one model can handle:</p><pre>/think → CoT<br>/no-think → short answer</pre><p>This reduces deployment complexity.</p><h4>4. General RL</h4><p>Expands capability beyond reasoning.</p><p><strong>Enhances:</strong></p><ul><li>instruction following</li><li>format adherence</li><li>preference alignment</li><li>tool-interaction abilities</li><li>specialized scenario handling</li></ul><h4>Uses three reward types:</h4><ol><li><strong>Rule-based</strong></li><li><strong>Model-based with reference</strong></li><li><strong>Model-based without reference (preference model)</strong></li></ol><p>This is where the model becomes stable, obedient, and aligned.</p><h4>Strong-to-Weak Distillation</h4><p>Used for:</p><ul><li>5 dense lightweight models</li><li>1 MoE lightweight model</li></ul><p><strong>Two phases:</strong></p><p><strong>1. Off-Policy Distillation</strong></p><p>Teacher outputs from both /think and /no-think → merged.<br> Student learns:</p><ul><li>basic reasoning</li><li>switching between modes</li></ul><p><strong>2. On-Policy Distillation</strong></p><p>Student generates its own responses.<br> Then:</p><ul><li>align logits with teacher (32B or 235B)</li><li>minimize KL divergence</li></ul><p>This ensures lightweight models behave like their big siblings.</p><p>One of the most useful concept that Qwen is capable of is:</p><h4>open vocabulary object detection:</h4><p>Before diving into how Qwen-3 handles OVOD, let’s quickly break down what OVOD even means in modern vision-language systems.</p><p><strong>What is Open-Vocabulary Object Detection?</strong></p><p>Traditionally, object detectors like YOLO/Faster R-CNN can only detect objects from a <em>fixed list</em>, like:</p><ul><li>“dog”</li><li>“chair”</li><li>“cup”</li></ul><p>If the model was never trained on “nebulizer” or “artificial heart valve,” it simply can’t detect it.</p><p><strong>Open-Vocabulary Object Detection (OVOD)</strong> changes that.</p><p>Instead of hardcoding categories, OVOD models use <strong>text embeddings</strong> as queries.</p><p>So the model can detect:</p><ul><li><em>“stethoscope”</em></li><li><em>“AC adapter”</em></li><li><em>“conveyor belt sensor”</em></li><li><em>“rare species of coral”</em></li></ul><p>even if these categories were <em>never</em> part of the training label set.</p><h3>How OVOD works:</h3><ol><li>The image is broken into <strong>regions</strong> (patches or bounding-box features).</li><li>Natural language queries are encoded into the same embedding space.</li><li>The model matches text ↔ region embeddings through similarity.</li></ol><p>If similarity passes a threshold → <strong>draw a box and label it</strong>.</p><p>This is why OVOD is powerful for:</p><ul><li>medical AI</li><li>robotics</li><li>warehouse automation</li><li>wildlife detection</li><li>manufacturing inspection</li><li>any scenario where you don’t know all object categories upfront.</li></ul><h4>OVOD Architecture flow:</h4><ol><li><strong>Vision Encoder → Region Features</strong></li></ol><p>The model first uses a vision transformer to extract:</p><ul><li>patch embeddings</li><li>region-level features</li><li>spatial features</li></ul><p>These are the “object candidates.”</p><p>This is similar to how CLIP-style detectors or SAM + DINO pipelines work.</p><p><strong>2. Region Code Generation (RDD)</strong><br>RDD converts each region into a <strong>compact textual embedding</strong> like:</p><ul><li>“round metallic object”</li><li>“plastic medical tool”</li><li>“elongated cylinder”</li></ul><p>These region codes help align the image with natural language queries.</p><p>This is <em>exactly</em> the component used in modern OVOD architectures like:</p><ul><li>GLIP</li><li>Grounding-DINO</li><li>OWL-ViT</li></ul><p><strong>3. Text Encoder → Open Vocabulary Queries</strong></p><p>On the language side, Qwen-3 embeds whatever text you provide:</p><p>Examples:</p><ul><li>“find all the surgical tools”</li><li>“locate the monkfish in the image”</li><li>“highlight anything that looks defective”</li></ul><p>The model does not rely on pre-defined labels.</p><p>It relies purely on semantic embeddings.</p><p><strong>4. Cross-Modal Alignment (CLIP-Style Similarity)</strong></p><p>Qwen-3 compares:</p><ul><li>region embeddings</li><li>text embeddings</li></ul><p>The highest-scoring matches become the detected objects.</p><p>This is OVOD:<br> ✔ no fixed label set<br> ✔ fully language-driven detection<br> ✔ robust zero-shot detection</p><p><strong>5. Vision Tracking (OV)</strong></p><p>This means Qwen-3 can:</p><ul><li>track arbitrary objects described by text</li><li>across multiple images or frames</li><li>without pre-defined category labels</li></ul><p>This is the <em>temporal</em> version of OVOD.</p><p>Example:<br> “Track the red mug across these frames.”<br>even if “red mug” wasn’t a class in the dataset.</p><h3>Examples:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hjqoPGC2O9a4kA-GmzKIbA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qhFboCsVKg94QPNd4V4MJg.png" /></figure><h3>Conclusion:</h3><p>Qwen-3 isn’t just another “bigger model.”<br>It’s a rethink of <em>how</em> an LLM should use its intelligence.</p><ul><li>dynamic thinking</li><li>controllable reasoning depth</li><li>long context</li><li>strong-to-weak distillation</li><li>efficient MoE specialization</li><li>and a unified reasoning framework</li></ul><p>This makes Qwen-3 feel less like a static model and more like an adaptive system, one that thinks only when needed, and thinks deeply only when you want it to.</p><p>If we see AGI as “general, adaptive reasoning,” this is a meaningful step in that direction.</p><h3>References:</h3><ol><li>technical paper: <a href="https://arxiv.org/abs/2505.09388">https://arxiv.org/abs/2505.09388</a></li><li><a href="https://colab.research.google.com/github/roboflow-ai/notebooks/blob/main/notebooks/open-vocabulary-object-detection-with-qwen3-vl.ipynb">https://colab.research.google.com/github/roboflow-ai/notebooks/blob/main/notebooks/open-vocabulary-object-detection-with-qwen3-vl.ipynb</a></li><li>OVOD: <a href="https://arxiv.org/pdf/2310.17109">https://arxiv.org/pdf/2310.17109</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bba7433c77af" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DeepSeek OCR]]></title>
            <link>https://medium.com/@nandinilreddy/deepseek-ocr-21923e700291?source=rss-6bc2b22a69b5------2</link>
            <guid isPermaLink="false">https://medium.com/p/21923e700291</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[deepseek]]></category>
            <category><![CDATA[ocr]]></category>
            <dc:creator><![CDATA[Nandini Lokesh Reddy]]></dc:creator>
            <pubDate>Fri, 14 Nov 2025 14:46:56 GMT</pubDate>
            <atom:updated>2025-11-14T14:46:56.902Z</atom:updated>
            <content:encoded><![CDATA[<p>When we say “a picture speaks a thousand words,” it actually makes sense in real world.</p><p><em>Would that hold true for a model too</em>?<em>? Answer at the end:</em></p><p>But how does this impact the model?<br>Before getting into that, let’s understand how tokens affect computation power, energy consumption, and GPU usage. The number of tokens is directly proportional to GPU power and memory requirements.</p><h4>Let’s start simple:</h4><p><strong><em>“This is an apple.”</em></strong><br>When we type this into an LLM like ChatGPT, it doesn’t read it like we do. It breaks it down into tokens:<br> • “this”<br> • “is”<br> • “an”<br> • “apple”<br> Total = 4 tokens</p><p>The model processes these 4 tokens through its attention mechanism; basically asking, “how does each word relate to the others?”<br> This attention process involves comparisons: 4 × 4 = 16 operations.<br> That’s quadratic scaling (n²).</p><p>So, for a document with 1,000 words → 1,000 tokens → 1,000 × 1,000 = 1,000,000 operations.<br> Here’s the problem: doubling the tokens means 4× the computation!</p><p>That’s where DeepSeek OCR started, with the question:<br>“For a document containing 1,000 words, how many vision tokens are needed for decoding?”</p><h4>Before answering, let’s look at traditional text-processing methods:</h4><p>When a page has ~1,000 words:</p><ol><li>The document is converted to text.</li><li>The text is tokenized: 1,000 words ≈ 1,200 text tokens.</li><li>The model processes them through attention: 1,200 × 1,200 = 1,440,000 operations.</li></ol><p><strong>Pros:</strong><br> ✓ High accuracy; models are trained heavily on text<br> ✓ Precise control over every word<br> ✓ Established and reliable pipeline</p><p><strong>Cons:</strong><br> ✗ Huge computational cost (1.4M operations)<br> ✗ Quadratic scaling; doubling the document = 4× computation<br> ✗ High memory usage<br> ✗ Slow processing for long documents<br> ✗ Expensive GPUs</p><p>So how do we reduce cost and memory usage?<br> Let’s see this question from different angle:<br> “What if, instead of feeding 1,200 text tokens, we just took a picture of the page and fed it as an image?”</p><p>Would the model still understand it?<br>And if yes, how many image tokens would it need?</p><h4>Let’s see how DeepSeek OCR answers that.</h4><p>When an image is fed to a vision-language model, it doesn’t “see” it like we do.<br> Just as text becomes tokens, images are broken into <strong>vision tokens</strong> small image patches.</p><p>Example:<br> • A 1024×1024 image is divided into 16×16 pixel patches.<br> • Each patch becomes one vision token.<br> • The model processes these tokens instead of text tokens.</p><p><strong><em>So, how many image tokens do we need for 1,000 text tokens?</em></strong><br>Other VLM architectures couldn’t answer this clearly, because of design challenges like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qeHZMe47q-uD0tc57Ol4LQ.png" /></figure><p><strong>1. Dual-tower architecture</strong><br> a. Uses two vision encoders in parallel<br> b. One for high-level understanding, one for fine details<br> c. Needs to process the image twice</p><p>Problems:<br> • Double preprocessing<br> • Difficult pipeline parallelism during training<br> • Complex deployment</p><p><strong>2. Tile-based method</strong><br> a. Cuts the image into small tiles<br> b. Processes each tile separately<br> c. Uses low native resolution (512×512 or less)</p><p>Cons:<br> • Large images get fragmented<br> • Too many vision tokens<br> • Defeats compression purpose</p><p><strong>3. Adaptive resolution</strong><br> a. Handles flexible resolutions<br> b. Divides images directly into patches<br> c. No tile parallelization</p><p>Cons:<br> • Huge activation memory usage<br> • GPU memory overflow risk<br> • Requires long sequence lengths<br> • Slow inference</p><h4>To achieve efficient compression, the ideal vision encoder must:</h4><p>a. Handle high-res images<br> b. Keep activation memory low<br> c. Output very few vision tokens<br> d. Support multi-resolution inputs<br> e. Have a moderate parameter count</p><h3>DeepSeek’s Two-Part Strategy</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3I4jnIs3qEuJGEqENaV6iw.png" /></figure><h4><strong>DeepEncoder: </strong>It has two main parts connected in series.</h4><p><strong>Part 1: Visual Perception (SAM-base)</strong><br> Focuses on fine-grained visual details using <strong>Window Attention, </strong>which looks at local patches.<br> • Parameters: 80M<br> • Patch size: 16×16</p><p>Window attention works like using a magnifying glass on a newspaper focusing on small areas rather than the entire page.</p><p><strong>Why SAM?</strong><br> SAM (Segment Anything Model) excels at fine detail extraction; perfect for OCR tasks.<br> • Text characters are small and precise<br> • Needs to distinguish ‘O’ vs ‘0’ or ‘l’ vs ‘I’<br> • Window attention captures these efficiently<br> • Lightweight (80M params), good for thousands of tokens</p><p><strong>Part 2: Visual Knowledge (CLIP-large)</strong><br> Focuses on semantic understanding using <strong>Global Attention</strong>, which looks at the entire image.<br> • Parameters: 300M</p><p>Global attention is like stepping back from the newspaper, seeing overall structure and relationships.</p><p><strong>Why CLIP?</strong><br> CLIP (by OpenAI) was trained on 400M image-text pairs and learned the connection between visuals and language.<br> For OCR, CLIP provides:<br> • Understanding of text meaning<br> • Awareness of document layout<br> • Ability to connect visuals to text concepts</p><h4>2. The Bridge: 16× Convolutional Compressor</h4><p>Between SAM and CLIP sits a <strong>16× convolutional compressor</strong> with two conv layers.</p><p><strong>Purpose:</strong><br>a. SAM outputs 4,096 tokens (for a 1024×1024 image).<br>b. Feeding all 4,096 into CLIP → 4,096×4,096 = 16,777,216 operations per attention layer → a memory explosion.</p><p>The compressor reduces tokens from 4,096 → 256 before global attention.</p><p><strong>How:</strong><br> • Kernel: 3×3<br> • Stride: 2<br> • Padding: 1<br> • Channels: 256 → 1024</p><p>Each stride-2 conv halves spatial size:</p><ol><li>64×64 → 32×32 (4× compression)</li><li>32×32 → 16×16 (4× again)<br> Total = 16× compression</li></ol><p>Total DeepEncoder params ≈ 380M (80M SAM + 300M CLIP + few for compressor)</p><h4>3. Deep Decoder: DeepSeek-3B-MoE</h4><p>Once compressed, how do we get text back?</p><p><strong>Architecture:</strong> Mixture-of-Experts (MoE) LLM<br> • Total params: 3B<br> • Activated per token: 570M (~19%)</p><p>In regular models, all parameters are used for every token.<br> In MoE, there are multiple “expert” networks, only a few are activated per token.</p><p><strong>DeepSeek-3B-MoE setup:</strong><br> • 64 routed experts + 2 shared experts<br> • 8 experts active per token (6 routed + 2 shared)<br> → Only 570M used per token</p><blockquote>“DeepSeek-OCR was trained on over 30 million pages of PDF documents covering ~100 languages; that’s roughly equivalent to 150,000 books.”</blockquote><p><strong>Why this for OCR?</strong><br> Different content types need different experts:<br> • Hindi → Experts #3, #17, #42<br> • Chinese → #8, #23, #56<br> • Math formulas → #12, #31, #47<br> • Tables → #5, #29, #61</p><p>The model learns to route tokens automatically.</p><h3>Workflow</h3><p><strong>Input:</strong> 1024×1024 image of a document (~1,000 words)</p><p><strong>Stage 1: SAM (Window Attention)</strong><br> • Image → patches (1024÷16)² = 4,096 tokens<br> • Captures local details like curves, dots, and serifs<br> • Low activation memory</p><p><strong>Stage 2: Compressor</strong><br> • 4,096 → 256 tokens (16× compression)<br> • Summarizes local features (e.g., “this region = the word ‘Apple’”)</p><p><strong>Stage 3: CLIP (Global Attention)</strong><br> • 256 tokens processed fully<br> • Captures layout, meaning, and context relationships<br> • “Apple” now understood in semantic and positional context</p><p><strong>Stage 4: DeepSeek-3B-MoE Decoder</strong><br> • 256 vision tokens passed to decoder + prompt (“Free OCR”)<br> • Original 1,000-word doc (~1,200 text tokens) → compressed to 256 vision tokens (~5× compression)</p><p>In <strong>Small mode (640×640)</strong>:<br> • Only 100 vision tokens needed<br> • 12× compression ratio<br> • ~97% accuracy</p><p>So, for a 1,000-word document (~1,200 text tokens), only <strong>100 vision tokens</strong> are needed ~12.6× compression with 97.1% accuracy.</p><h3>Sample Output:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GIHsIFab-NBtHb_f4r65dA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*adEJYI-C4UtW1RzLmKP5yQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/1*EChnDicPNSQf7mszwFXOlA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*07KbHfzKs447JWBOkm2Cmg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UeaFNQrdfRD3vGhMDRu71w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*fCm6fFxuhI_tU0MDfkdKqg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NNeKwbkD1hF9Ymtpiq3R7w.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/1*wvvzWi0w5s7UeeTNwe0OkA.png" /></figure><h3>Limitations:</h3><p>But like every innovation, it has trade-offs:<br> • Compression vs accuracy<br> • Uneven performance across document types<br> • Resolution vs token trade-offs<br> • Text-heavy vs image-heavy differences<br> • Language coverage bias<br> • Unexplored “forgetting” mechanism<br> • Not yet a general-purpose VLM</p><h3>Conclusion</h3><p><strong><em>A picture is literally worth ~1,000 words even in AI processing.</em></strong></p><h3>References:</h3><ol><li>inference code: <a href="https://github.com/NandiniLReddy/deepseekOCR-test/blob/main/test_inference_v100.py">https://github.com/NandiniLReddy/deepseekOCR-test/blob/main/test_inference_v100.py</a></li><li>deepseek-OCR git: <a href="https://github.com/deepseek-ai/DeepSeek-OCR">https://github.com/deepseek-ai/DeepSeek-OCR</a></li><li><a href="https://cloud.google.com/blog/products/ai-machine-learning/scaling-deep-retrieval-tensorflow-two-towers-architecture">https://cloud.google.com/blog/products/ai-machine-learning/scaling-deep-retrieval-tensorflow-two-towers-architecture</a></li><li><a href="https://developer.nvidia.com/gpugems/gpugems2/part-ii-shading-lighting-and-shadows/chapter-12-tile-based-texture-mapping">https://developer.nvidia.com/gpugems/gpugems2/part-ii-shading-lighting-and-shadows/chapter-12-tile-based-texture-mapping</a></li><li><a href="https://arxiv.org/pdf/2412.06195">https://arxiv.org/pdf/2412.06195</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=21923e700291" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>