<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Den Pluhatar on Medium]]></title>
        <description><![CDATA[Stories by Den Pluhatar on Medium]]></description>
        <link>https://medium.com/@mahabharahta?source=rss-c91a000e0ec5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*syMVNvFXjgsyEG49ZHIucw.png</url>
            <title>Stories by Den Pluhatar on Medium</title>
            <link>https://medium.com/@mahabharahta?source=rss-c91a000e0ec5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 22:08:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mahabharahta/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Vision Language Action Models: Part 2]]></title>
            <link>https://medium.com/@mahabharahta/vision-language-action-models-part-2-c25b9d1fab0f?source=rss-c91a000e0ec5------2</link>
            <guid isPermaLink="false">https://medium.com/p/c25b9d1fab0f</guid>
            <dc:creator><![CDATA[Den Pluhatar]]></dc:creator>
            <pubDate>Tue, 04 Aug 2026 07:47:47 GMT</pubDate>
            <atom:updated>2026-08-11T08:43:36.825Z</atom:updated>
            <content:encoded><![CDATA[<p><em>LoRA vs full fine-tune, loss functions, training loop, and how to debug when nothing is working</em></p><p><strong>TL;DR:</strong> In <a href="https://medium.com/@mahabharahta/vision-language-action-models-part-1-1969ef71d095">Part 1</a> we built the architecture and understood the data format. Now we train it. This article covers every decision between “I have a dataset” and “I have a working policy” — fine-tuning strategy, loss functions, the training loop, and a debugging guide for the most common failure modes.</p><p>Quick recap of Part 1 so this article stands alone:</p><ul><li><strong>Model:</strong> VLAModel(vision_encoder, llm, projector, action_head)</li><li><strong>Data:</strong> triplets of (image, instruction, action) where action is a 7-dim delta vector, tokenized into bin indices [0, 255] × 7</li><li><strong>Backbone options (2026):</strong> Qwen3-VL-7B or Gemma 4 E4B for most cases</li></ul><p>The question now is: which parts of the model do we actually update during training?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MpDD9LPCXcoJ1k0RlP0WLg.png" /></figure><p>Not everything needs gradients. A VLA has four trainable components and the choice of what to update has a bigger impact on result quality than almost any hyperparameter:</p><p><strong>Vision encoder</strong> — almost always frozen. SigLIP 2 was pretrained on billions of image-text pairs; your robotics dataset is too small to improve it and will likely hurt its representations. The only exception is very long fine-tuning runs (100k+ steps) on diverse robotics data.</p><p><strong>Projector (linear layer)</strong> — always trained from scratch. It’s small (768 → LLM hidden dim, typically 2M parameters) and initialised randomly. It must learn to bridge the vision and language modalities for your specific domain.</p><p><strong>LLM backbone</strong> — the main decision point. Full fine-tune vs LoRA (see Section 1.2).</p><p><strong>Action head</strong> — always trained from scratch, always.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NkrUGQZsklTp2kupss05WQ.png" /></figure><p><strong>LoRA (Low-Rank Adaptation)</strong> injects small trainable matrices into the attention layers of the LLM while keeping original weights frozen. The key hyperparameters:</p><p><strong>When to use LoRA:</strong></p><ul><li>Dataset &lt; 5K demos (roughly &lt; 1M triplets — see the demo-to-triplet mapping in Section 7)</li><li>Limited GPU memory — LoRA fits even on a single 24 GB card</li><li>You want to preserve the LLM’s language understanding (important if the model also needs to answer questions)</li><li>Prototyping / first experiments — always start here</li></ul><p><strong>When to use full fine-tune:</strong></p><ul><li>Dataset &gt; 5K demos (&gt; 1M triplets) from diverse tasks</li><li>Multi-GPU setup (4+ × 80GB)</li><li>You’ve already validated the architecture works with LoRA</li><li>Targeting SOTA performance on a benchmark</li></ul><p><strong>The Qwen3-VL special case:</strong> when using Qwen3-VL as a fused backbone, apply LoRA to the language model part only. The visual encoder inside Qwen3-VL follows the same rule — freeze by default.</p><p>If you’re using the discretized approach (N=256 bins per dimension), training is a multi-class classification problem. The loss is the cross-entropy averaged over all 7 action dimensions:</p><p><strong>Why not sum?</strong> Averaging makes the loss scale-invariant to the number of action dimensions. When you later experiment with 14-dim bimanual actions, your learning rate won’t need to change.</p><p>If you’re predicting continuous values directly (regression head):</p><p><strong>Delta=0.1</strong> is a good starting point for normalised actions. Outlier actions (the robot teleoperator made a sudden large movement) will otherwise dominate the loss and destabilise training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*JgvM2_MYau5EJ7phIGLuJg.png" /><figcaption>flow-matching loss</figcaption></figure><p>Flow-matching loss optimises the vector field between noise and target action:</p><p>The gripper dimension (dim 6) is binary — open or closed. It’s often the most critical dimension for task success (a 1mm positioning error matters less than failing to close the gripper). Consider weighting it:</p><p>This static per-dimension weighting is orthogonal to the per-bin class weighting in Section 5.2 — the former says “the gripper dimension matters more”, the latter fixes open/closed class imbalance <em>within</em> that dimension. You can apply both.</p><p>If your GPU can’t fit batch_size=32, use gradient accumulation:</p><p><strong>Note on fp16:</strong> GradScaler is only required for float16, where the narrow dynamic range causes gradient underflow. If your GPU doesn&#39;t support bfloat16 (pre-Ampere), run autocast with dtype=torch.float16 and wrap the backward/step in torch.amp.GradScaler - but on any modern GPU, prefer bf16 and skip the scaler entirely.</p><p>Good logging saves days of debugging. Log these metrics every N steps:</p><p><strong>The gripper accuracy metric</strong> is the single most useful early signal — but use <em>balanced</em> accuracy (mean of per-class recall), not raw accuracy. Demonstrations are heavily imbalanced towards “closed” (see Section 5.2), so a model that always predicts “closed” already scores 80–90% raw accuracy while learning nothing. A model whose balanced accuracy climbs above 90% within the first 1–2 epochs is learning the task structure; if it’s stuck near 50%, the model is predicting the majority class and ignoring the scene.</p><p><strong>Symptoms:</strong> Training loss stays flat or oscillates from the first epoch.</p><p><strong>Checklist:</strong></p><ul><li>Action tokens not normalised before tokenization → actions all map to bins 0 or 255</li><li>LR too high → reduce by 10×, restart</li><li>Projector not connected to gradients → check requires_grad=True</li><li>Instruction tokenizer padding issue → all text embeddings are [PAD]</li></ul><p><strong>Symptoms:</strong> Robot always predicts gripper fully open or fully closed regardless of scene. (Convention used below: gripper dim normalised to [0, 1], where &lt; 0.5 = open, ≥ 0.5 = closed. If your pipeline normalises to [-1, 1], shift the threshold to 0.)</p><p><strong>Root cause:</strong> Class imbalance. Demonstrations typically have 80–90% of timesteps with gripper closed. The model learns to always predict “closed” and gets decent loss.</p><p><strong>Fix:</strong> the gripper is effectively binary, but its tokens live in a 256-bin space. Build the class-weight vector over all 256 bins — weight every “open” bin by the imbalance ratio, leave “closed” bins at 1.0:</p><p><strong>Symptoms:</strong> All rotational predictions biased in one direction; robot circles or drifts.</p><p><strong>Root cause:</strong> Dataset collection bias. If the demonstrator tends to approach objects from the left, yaw actions will be consistently negative.</p><p><strong>Fix:</strong> Balance the dataset per dimension, or use data augmentation — flip images horizontally and negate the mirrored action dimensions:</p><p><strong>Two caveats specific to VLA.</strong> First, unlike plain imitation learning, your inputs include language: flipping the image without editing the instruction silently corrupts every sample containing spatial words (“pick up the object on the left”). Either skip those samples (as above) or mirror the instruction text too. Second, which translation axis maps to left-right in the image depends on your camera mounting and coordinate convention — verify on a few samples before trusting the augmentation.</p><p><strong>Symptoms:</strong> Validation loss looks fine (&lt; 0.5), but the robot fails basic tasks.</p><p><strong>Most common causes:</strong></p><ul><li>Normalisation mismatch: training used one min/max, inference uses another → always save and load the same action_stats.json</li><li>Covariate shift: robot camera angle differs from training data</li><li>Action frequency mismatch: trained at 10 Hz, running at 5 Hz → each predicted delta covers only half the motion the control step needs, so the robot moves at half the intended speed (run faster than trained and it overshoots instead)</li></ul><p><strong>Symptoms:</strong> The model predicts reasonable-looking actions for ANY instruction regardless of content — swap “pick up the cup” for “open the drawer” and the behaviour doesn’t change.</p><p><strong>Root cause:</strong> Shortcut learning. On small datasets with one dominant task per scene, the image alone is enough to predict the action, so the model never learns to condition on the text. (Note this is <em>not</em> classic catastrophic forgetting — LoRA actually reduces forgetting risk compared to full fine-tune, which is one of the reasons Section 1.2 recommends it.)</p><p><strong>Fix:</strong> Increase instruction diversity — collect multiple tasks per scene layout, so the text becomes the only disambiguator. Mixing in a small fraction (5–10%) of pure language instruction-following samples also helps keep the LLM’s text pathway active.</p><p>Don’t wait until the end to evaluate. Run a lightweight simulation check every N epochs:</p><p>A model worth deploying typically crosses 60% success rate on LIBERO-Object or LIBERO-Spatial before you invest in real-robot evaluation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K4jtiAkliegYrsihKK1wxw.png" /></figure><p><em>Sizes here are in demonstrations (episodes). One demo typically yields 100–300 </em><em>(image, instruction, action) triplets, so 500 demos ≈ 50-150K triplets and 5K demos ≈ 0.5-1.5M - that&#39;s how this table maps onto the thresholds in Section 1.2 (LoRA below ~5K demos, full fine-tune worth considering above).</em></p><p>This article is Part 2 of a three-part series.</p><p><strong>Part 1: Theory &amp; Architecture</strong> — <a href="https://medium.com/@mahabharahta/vision-language-action-models-part-1-1969ef71d095">read here</a></p><p><strong>Part 2: Training Your VLA</strong> ← <em>you are here</em></p><p><strong>Part 3: Inference &amp; Evaluation</strong> — deployment pipeline, latency on real hardware, success-rate benchmarks <em>(coming soon)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c25b9d1fab0f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Vision Language Action Models: Part 1]]></title>
            <link>https://medium.com/@mahabharahta/vision-language-action-models-part-1-1969ef71d095?source=rss-c91a000e0ec5------2</link>
            <guid isPermaLink="false">https://medium.com/p/1969ef71d095</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[visual-language-model]]></category>
            <dc:creator><![CDATA[Den Pluhatar]]></dc:creator>
            <pubDate>Mon, 25 May 2026 08:15:59 GMT</pubDate>
            <atom:updated>2026-05-25T08:15:59.050Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Theory &amp; Architecture</strong></p><p><em>From vision encoders to action heads — everything you need to understand before writing a single line of training code</em></p><p><strong>TL;DR:</strong> Here I’m going to dive into real example of implementing <strong>Vision Language Action Model. </strong>There will be 3 articles covering theory, training and inference.</p><h3>1. What Is a Vision Language Action Model?</h3><p>The past few years have seen a remarkable convergence: large language models learned to reason, vision-language models learned to see, and now a new class of model is learning to act. Vision Language Action models — VLAs — are the architecture that ties these threads together.</p><p>A VLA takes two inputs — an image of the current scene and a natural language instruction — and outputs an action that an agent (robot, simulated arm, drone) can execute immediately. This closes the loop between perception, language understanding, and physical control.</p><p>As of 2026, VLAs have crossed from research artifact to production infrastructure: VLA adoption is present in over 40% of all new robot deployments, and quantized models run at 10–25 Hz on consumer GPUs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oLeSAGJTLB_GUR1HiWktGA.png" /><figcaption>VLM vs VLA. A Vision Language Model outputs text; a VLA outputs actions, completing a feedback loop with the environment.</figcaption></figure><p>The key difference from a standard VLM is the <strong>feedback loop</strong>. A VLM generates text and stops. A VLA generates an action, that action changes the world, and the new state of the world becomes the next observation — a closed-loop, sequential decision-making system.</p><h3>2. Why Not Just Use a VLM?</h3><p>A natural question: could you just prompt <strong>Gemma 4 </strong>or <strong>Qwen3-VL</strong> with an image and ask “what should the robot do?”, then map the text to motor or servo commands? In practice this breaks down:</p><ul><li><strong>Latency. </strong>LLM text generation is too slow for real-time control (10–100 Hz required)</li><li><strong>Precision . </strong>“Move left a little” is not a joint angle</li><li><strong>Continuity. </strong>Action sequences need smooth interpolation; text is discrete</li><li><strong>Closed-loop awareness. </strong>VLM has no built-in mechanism to re-observe after acting</li></ul><h3>3. Core Architecture</h3><p>Every VLA has three functional blocks. Understanding each is prerequisite to building one.</p><h4>3.1 Vision Encoder</h4><p>The vision encoder converts a raw image into a sequence of dense vector embeddings — one per image patch. These become the visual tokens that the LLM attends to alongside text.</p><p>The dominant encoder in 2025–2026 is <strong>SigLIP 2</strong>, the backbone of <strong>SmolVLA</strong> and <strong>π0</strong>. It outperforms the original <strong>SigLIP</strong> and <strong>CLIP</strong> on robotics vision benchmarks.</p><pre>vision_encoder = SiglipVisionModel.from_pretrained(&quot;google/siglip2-base-patch16-224&quot;)<br>inputs = processor(images=image, return_tensors=&quot;pt&quot;)<br>visual_features = vision_encoder(**inputs).last_hidden_state<br># shape: (1, 196, 768) — 14×14 patch grid, 768-dim each</pre><h4>3.2 Language Model Backbone</h4><p>The LLM receives a joint sequence of visual tokens (projected into the LLM embedding space) and text tokens, and produces a hidden state that captures both scene and instruction.</p><ul><li><strong>Gemma 4 E2B / E4B — </strong>best for on-device and low-latency (5 GB at 4-bit, 128K context, Apache 2.0)</li><li><strong>Gemma 4 26B MoE / 31B Dense — </strong>higher quality, 256K context, #3 open model on LMArena</li><li><strong>Qwen3-VL 7B — </strong>native multimodal backbone, encoder already fused, 256K context, strong for VLA fine-tuning</li><li><strong>Qwen3.6–27B — </strong>latest open dense model (April 2026), excellent for scratch-built stacks</li></ul><p>In 2026: default to Qwen3-VL-7B or Gemma 4 E4B. Both fit on a 24 GB GPU at bfloat16. For on-device (&lt; 50ms latency), use Gemma 4 E2B at 4-bit.</p><h4>3.3 Action Head</h4><p>The action head maps the LLM’s final hidden state to a robot action. Two dominant approaches in 2026:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K4UY0Fwe4Pyp3N4PeOie2A.png" /><figcaption><em>Two action head strategies: discretized classification (simple, fast, matches autoregressive LLMs) and flow-matching/diffusion (smoother trajectories, preferred for dexterous manipulation).</em></figcaption></figure><ul><li><strong>Discretized (classification) — </strong>bin each action dimension into N=256 categories, predict with a linear classifier. Simple and fast. Default for first implementations.</li><li><strong>Flow-matching / Diffusion — </strong>predict continuous action distributions via learned denoising. Used in π0 and <strong>SmolVLA</strong>. Better for dexterous tasks.</li></ul><h3>3.4 Full Model</h3><pre>class VLAModel(nn.Module):<br>    def __init__(self, vision_encoder, llm, projector, action_head):<br>        super().__init__()<br>        self.vision_encoder = vision_encoder<br>        self.llm            = llm<br>        self.projector      = projector  # linear: 768 → llm.hidden_size<br>        self.action_head    = action_head<br><br><br>    def forward(self, images, input_ids, attention_mask):<br>        vis_feats  = self.vision_encoder(images).last_hidden_state<br>        vis_tokens = self.projector(vis_feats)<br>        txt_embeds = self.llm.get_input_embeddings()(input_ids)<br>        combined   = torch.cat([vis_tokens, txt_embeds], dim=1)<br>        outputs    = self.llm(inputs_embeds=combined, output_hidden_states=True)<br>        last_h     = outputs.hidden_states[-1][:, -1, :]<br>        return self.action_head(last_h)</pre><p>Note: if you use Qwen3-VL as backbone, the encoder and LLM are <strong>already fused</strong>. You pass images directly to the model’s forward — only the action_head sits on top.</p><h3>4. Data: Training Triplets in Detail</h3><p>A VLA is trained on demonstrations — recordings of a robot (or human teleoperation) completing tasks. Each timestep in a demonstration becomes one training example, called a triplet.</p><h4>4.1 What Is a Triplet?</h4><p>Every triplet contains exactly three things: what the robot sees, what it was told to do, and what it did.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lxOSP85SG2letp2T-TiY7g.png" /></figure><p>A single 30-second demonstration at 10 Hz yields 300 triplets. Every triplet is an independent training sample — the model sees image + instruction and must predict the action. There is no temporal dependency baked in by default (though some architectures add a short history window of 2–4 frames).</p><p><em>Why repeat the instruction for every timestep? Because the model sees each triplet independently. It has no memory of previous steps. The instruction is the only source of task context at inference time.</em></p><h3>4.2 The 7-Dimensional Action Vector — In Depth</h3><p>The action vector encodes what the robot’s end-effector (gripper) does at a single timestep. The standard choice for a 6-DoF robot arm with a 1-DoF gripper is a 7-dimensional delta-action representation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1N7gnbCcpyX-woCkg65PSg.png" /></figure><h4>Normalisation</h4><p>Before tokenization, actions are normalised to [-1, 1] per dimension. The normalisation statistics (min/max or mean/std) are computed on the training dataset and saved alongside the model checkpoint — you need them at inference time to convert predictions back to real joint values.</p><pre># Compute once over training dataset<br>action_min = dataset_actions.min(axis=0)   # shape: (7,)<br>action_max = dataset_actions.max(axis=0)   # shape: (7,)<br><br><br>def normalize(action):<br>    return 2.0 * (action - action_min) / (action_max - action_min) - 1.0<br><br><br>def denormalize(norm_action):<br>    return (norm_action + 1.0) / 2.0 * (action_max - action_min) + action_min</pre><h4>Tokenization — how the 7D vector becomes LLM input</h4><p>For discretized action heads, each of the 7 dimensions is independently binned into N=256 categories:</p><pre>def tokenize_action(action: np.ndarray,<br>                    bins: int = 256,<br>                    min_val: float = -1.0,<br>                    max_val: float = 1.0) -&gt; np.ndarray:<br>    &quot;&quot;&quot;<br>    Input:  action  — shape (7,), values in [-1, 1] after normalisation<br>    Output: tokens  — shape (7,), integer bin indices in [0, 255]<br>    &quot;&quot;&quot;<br>    clipped    = np.clip(action, min_val, max_val)<br>    normalized = (clipped - min_val) / (max_val - min_val)  # [0, 1]<br>    return (normalized * (bins - 1)).round().astype(int)    # [0, 255]<br><br><br>def detokenize_action(tokens: np.ndarray,<br>                      bins: int = 256,<br>                      min_val: float = -1.0,<br>                      max_val: float = 1.0) -&gt; np.ndarray:<br>    &quot;&quot;&quot;Reconstruct continuous action from bin indices.&quot;&quot;&quot;<br>    return tokens / (bins - 1) * (max_val - min_val) + min_val<br><br><br># Example: full round-trip<br>action   = np.array([0.02, 0.00, -0.03, 0.00, 0.01, 0.00, 1.0])<br>tokens   = tokenize_action(action)<br># tokens  → [130, 128, 124, 128, 129, 128, 255]<br>recovered = detokenize_action(tokens)<br># recovered ≈ [0.020, 0.000, -0.031, 0.000, 0.008, 0.000, 1.000]</pre><p>With 256 bins and range [-1, 1], the resolution is 2/255 ≈ 0.008 per bin. For a 30 cm workspace, that corresponds to sub-millimetre precision — sufficient for most pick-and-place tasks.</p><h4>How many action dimensions do I actually need?</h4><p>It depends on the robot. The 7-dim representation is for a typical 6-DoF arm + gripper. Here is what changes for other platforms:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XOA94n6_007x-_dLeYHetA.png" /></figure><h3>4.3 Building a Triplet Dataset in Code</h3><p>A minimal dataset class for HuggingFace Trainer:</p><pre>class VLADataset(torch.utils.data.Dataset):<br>    &quot;&quot;&quot;<br>    Each item is a triplet:<br>      image       : RGB tensor  (3, 224, 224)<br>      instruction : tokenized text<br>      action      : 7-dim float or 7-dim int (if discretized)<br>    &quot;&quot;&quot;<br>    def __init__(self, episodes_dir, tokenizer, processor,<br>                 action_min, action_max, bins=256):<br>        self.triplets = []<br>        for ep in load_episodes(episodes_dir):          # RLDS / LeRobot format<br>            for t in range(len(ep[&quot;images&quot;])):<br>                self.triplets.append({<br>                    &quot;image&quot;:       ep[&quot;images&quot;][t],<br>                    &quot;instruction&quot;: ep[&quot;instruction&quot;],   # same for all t<br>                    &quot;action&quot;:      ep[&quot;actions&quot;][t],    # shape (7,)<br>                })<br><br><br>    def __getitem__(self, idx):<br>        item = self.triplets[idx]<br>        image   = self.processor(images=item[&quot;image&quot;], return_tensors=&quot;pt&quot;)<br>        text    = self.tokenizer(item[&quot;instruction&quot;], return_tensors=&quot;pt&quot;)<br>        action  = normalize(item[&quot;action&quot;])             # → [-1, 1]<br>        tokens  = tokenize_action(action)              # → [0, 255] × 7<br>        return { &quot;pixel_values&quot;: image.pixel_values[0],<br>                 &quot;input_ids&quot;:    text.input_ids[0],<br>                 &quot;action_tokens&quot;: torch.tensor(tokens, dtype=torch.long) }</pre><h3>4.4 Public Datasets in 2026</h3><ul><li><strong>Open X-Embodiment — </strong>22 robot platforms, 1M+ episodes, RLDS format. Best starting point for pretraining.</li><li><strong>BridgeData v2</strong>–60K kitchen manipulation demos. Ideal for fine-tuning on tabletop tasks.</li><li><strong>LIBERO — </strong>simulation benchmark, 4 task suites. Good for no-hardware evaluation.</li><li><strong>HuggingFace LeRobot dataset — </strong>community-collected, standardised format, directly compatible with SmolVLA and π₀. The fastest path to a working baseline in 2026.</li></ul><p>The average cost of high-quality teleoperation data fell from $340/hour in 2024 to $118/hour in early 2026 — a 60% decrease — putting a $50K–$150K data budget within reach for enterprise pilots.</p><h3>5. The 2026 VLA Model Landscape</h3><p>Before building from scratch, survey the models you can fine-tune or build upon.</p><p><strong>RT-2 (Google DeepMind, 2023)</strong></p><p>The paper that defined the field. PaLM-E backbone, co-fine-tuned on robotics + web data. Closed-source, but the ideas underpin every subsequent model.</p><p><strong>OpenVLA (2024)</strong></p><p>First major open-source VLA: LLaMA 2 7B + SigLIP, trained on Open X-Embodiment. The canonical reference implementation.</p><p><strong>π₀ / π₀-FAST (Physical Intelligence, 2024–2025)</strong></p><p>Flow-matching head on PaLiGemma backbone. Trained on 8 robot platforms. π₀-FAST adds DCT-based action compression. Ported to LeRobot.</p><p><strong>GR00T N1 (NVIDIA, March 2025)</strong></p><p>Open VLA for humanoids. Dual-system: VLM for planning + diffusion policy at 10ms latency. Trained on robot, human video, and synthetic data.</p><p><strong>SmolVLA (HuggingFace, 2025)</strong></p><p>450M parameters, fully open-source, flow-matching head, asynchronous inference. Comparable to Octo and OpenVLA. The practical baseline for 2026.</p><p><strong>Qwen3-VL as VLA backbone (community, 2025–2026)</strong></p><p>Qwen3-VL 2B fine-tuned with full LM head on action tokens. 60–70% task success with ~35 demos. Also used in TwinBrainVLA and QwenGR00T research.</p><h3><strong>Continue the Series</strong></h3><p>This article is Part 1 of a three-part series. The full roadmap:</p><p><strong>Part 1: Theory &amp; Architecture</strong><em> ← you are here</em></p><p><strong>Part 2: Training Your VLA</strong> — LoRA vs full fine-tune, loss functions, training loop, debugging</p><p><strong>Part 3: Inference &amp; Evaluation</strong> — deployment pipeline, latency on real hardware, success-rate benchmarks</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1969ef71d095" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Journey Through 6 RAG Strategies to Find the Appropriate One]]></title>
            <link>https://medium.com/@mahabharahta/the-journey-through-6-rag-strategies-to-find-the-appropriate-one-853a51f1daa1?source=rss-c91a000e0ec5------2</link>
            <guid isPermaLink="false">https://medium.com/p/853a51f1daa1</guid>
            <category><![CDATA[agentic-rag]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Den Pluhatar]]></dc:creator>
            <pubDate>Tue, 05 May 2026 19:15:30 GMT</pubDate>
            <atom:updated>2026-05-05T19:15:30.792Z</atom:updated>
            <content:encoded><![CDATA[<p>Retrieval-augmented generation (RAG) has become a foundation for building high-accuracy, reliable systems on top of large language models (LLMs). There is a range of RAG strategies available today compared to its early days that help improve data retrieval in one way or another. Choosing the right approach is critical because it defines which trade-off you’re ready to accept between precision, latency, and budget.</p><p>The six techniques are outlined below, from the simplest viable approach to full agentic workflows, with the ‘why’ and ‘when’ for each.</p><h3>Naive RAG and Why Everyone Begins Here</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QqFX11h1O2244-0QFQ7BHw.png" /></figure><p>Naive RAG is the simplest-to-implement approach teams use to test the waters.</p><p>In this setup, documents are split into fixed-size chunks regardless of content boundaries, embedded into vectors, and indexed. When a query comes in, the system finds the top-k most semantically similar chunks and passes them to the LLM as context.</p><p>The time-to-first-result is also short, which is exactly why it’s a good choice for an MVP, uniform, small-batch knowledge base, or any stage where you’re still validating whether retrieval solves your problem at all.</p><h3>Where it breaks down</h3><p>Fixed-size chunking splits documents at arbitrary points, so a chunk that lands in the top-k may be missing the sentence and lose context. Semantic similarity doesn’t differentiate between relevant context and noise, decreasing the quality of the final answer. And when the relevant chunk simply isn’t retrieved, the LLM fills the blanks from parametric memory, which is where hallucinations originate.</p><p>If your accuracy requirements are loose and your data is clean and well-structured, naive RAG works. The moment any of those conditions breaks, it’s time to consider more context-aware retrieval strategies.</p><h3>Chunk-Aware RAG: Structure-Preserving Chunking</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4MADR689wNuFrFT7FiSmOA.png" /></figure><p>Chunk-aware RAG is a relatively low-effort optimization with a noticeable impact on retrieval quality.</p><p>Instead of splitting documents at fixed token intervals, it recognizes natural breaks in a document, like sections, subsections, or paragraphs, and determines the size of the chunk by semantic completeness. The same principle applies to tables, code blocks, and graphics, which are processed as atomic units that the system doesn’t cut mid-way. It gives LLMs a cleaner and more complete context.</p><p>You’ll see the most value from the chunk-aware approach when your knowledge base consists of highly structured data like technical manuals or API documentation, where the document’s layout has equally important semantic weight as the text itself.</p><h3>Where it breaks down</h3><p>Even though the quality of chunking is better, you’re still relying on semantic similarity for retrieval. Applied to unstructured or loosely formatted content, the effectiveness decreases, since there are no structural markers to guide chunking. It also doesn’t address retrieval ranking. If the relevant chunk exists but gets lost in the search results, better splitting won’t fix the issue.</p><h3>Hybrid RAG: Combining Semantic and Keyword Retrieval</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bTEh1RV3PeZVHcmrd4JSOA.png" /></figure><p>Hybrid RAG combines vector search with keyword-based search to cover what neither method handles well on its own.</p><p>Vector search is effective for semantic, concept-level queries and can find relevant content to answer the question “how to reduce inference latency,” even if the exact phrase doesn’t appear in the document. Keyword search (BM25 and similar) does the opposite. It looks for exact term matches but has no understanding of synonyms or paraphrases. “Car” and “automobile” are unrelated tokens to BM25.</p><p>When vector models optimize for semantic proximity, they suppress exact but rare identifiers like SKUs, serial numbers, model names, etc., in favor of contextually adjacent content. If your system needs to fetch document X when a user queries a specific product ID, pure vector search may sometimes miss it. Keyword search won’t.</p><p>So, during query execution, both retrieval signals are run in parallel, and the results are then combined and re-ranked using a Reciprocal Rank Fusion (RRF) scoring function before the final top-k reaches the LLM.</p><p>A hybrid strategy is a top choice for companies dealing with heterogeneous data, where you have long-form guides and databases full of specific IDs. It allows the system to manage different types of searches within the same retrieval pipeline. For this reason, it’s considered a practical option for production systems.</p><h3>Where it breaks down</h3><p>You get much more relevant results, but the pipeline is now more complex to manage. Maintaining and syncing vector and keyword indexes increases the storage and compute needs. Combining their outputs also requires additional manual tweaking and testing with the weighting to find the sweet spot.</p><h3>Multi-Stage RAG: Retrieval with Reranking</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qMTzDu3I5sWZQ0D8w6RVoA.png" /></figure><p>A single retrieval pass makes one attempt to find the right answer and moves on, which works fine, until the query is complex or the index is large enough that ranking becomes the bottleneck. Multi-stage RAG is a production-ready option that addresses both issues through additional filtering.</p><p>The search runs in two phases. The first phase uses a fast, lightweight retrieval model to pull a broad candidate set of, let’s say, 20–50 chunks. It’s intended for speed here, not precision. The second phase hands those candidates to a reranker, a model that evaluates each chunk’s relevance against the original query with higher scrutiny. It ignores mathematical distance and calculates a direct relevance score to reorder the list. This way, the system selects only the top 3 to 5 chunks from the reranked set and provides them to the LLM.</p><p>The key indicators telling it fits your case are large indexes where initial retrieval is semantically noisy, and domains where retrieval accuracy directly affects downstream trust, such as legal and financial documentation.</p><h3>Where it breaks down</h3><p>The primary limitation is latency. A two-pass pipeline with a reranker is slower than single-step retrieval, and wait times climb if the candidate set is larger, the reranker is heavier, or more steps are added. This is a real cost in latency-sensitive applications.</p><p>The less evident limitation is that the reranker reads only the text. If you feed them multimodal documents that involve tables, diagrams, or screenshots, they will only see the text and ignore the data hidden inside the visuals. This might be a critical deal-breaker for industrial or scientific use cases.</p><h3>Query-Rewriting RAG: Improving Retrieval via Query Transformation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*90JVOGhj1-xCtn5woOu-oA.png" /></figure><p>Since users tend to phrase queries as if in a natural conversation, which may be incomplete, vague, or lack the details that would significantly improve the model’s output, query-rewriting solves this by reformulating the query into a more explicit form.</p><p>Query-rewriting modifies the original query before retrieval using either a dedicated rewriter or the same LLM used for generation. It analyzes the query together with the conversation history and enhances or reformulates it to clarify the intent. A support ticket that reads “why did it break again?” becomes “root cause service deployment failure [date],” sent to the vector or hybrid search.</p><p>This approach works particularly well in conversational AI bots where users rely on context from previous messages, in customer support applications where queries may be emotional or imprecise, and anywhere users expect the system to read between the lines.</p><h3>Where it breaks down</h3><p>Again, one of the trade-offs is latency. Every time you rewrite a query, you’re making an extra call to an LLM, adding response time regardless of how fast the retrieval is.</p><p>The second pitfall is evaluation complexity. Query rewriting breaks the assumption of standard RAG quality metrics built around a direct query-to-answer relationship. The evaluation chain is now query → rewritten query → answer, and the middle step brings in variability you can’t capture using quantitative metrics. Quality assessment needs people to review the outputs, which is slower and more expensive than automated evals.</p><h3>Agentic RAG: Tool-Oriented Retrieval with Reasoning Loops</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TNqUVz9bCeS-IoJ8jCKTFg.png" /></figure><p>Every strategy covered so far follows a fixed retrieval sequence. Agentic RAG replaces that established order with the so-called reasoning loop.</p><p>There is a common belief that multi-step reasoning is the key differentiator of AI agents. But my practice shows that tool isolation is a true game-changer. Instead of searching through one massive, unorganized heap of data, the agent perceives each data source as an independent tool.</p><p>An agent’s toolbox may consist of a customer database, a product manual, and a live external API as separate instruments. After receiving the user’s question, it decides which of the tools to use and extracts data only from those sources. The agent evaluates the result and re-queries if it’s incomplete. Whether you use the LangChain framework or a similar one or build the workflow yourself, the core idea is to give an agent the right tools and the logic to use them.</p><p>Using agentic RAG makes sense for complex tasks, the kind where the answer needs to be assembled from multiple sources, or research-style tasks when each search depends on the results of the previous one. If data lives in silos and unifying it is impractical, this RAG strategy is a natural fit. My <a href="https://beetroot.co/services/generative-ai-services/">AI engineering team at Beetroot</a> has been working more on autonomous reasoning loops recently, seeing growing use cases around multi-step retrieval.</p><h3>Where it breaks down</h3><p>For this flexibility, be ready to pay in three aspects. The first is token consumption, as multiple LLM calls per query drive up cost fast, particularly at large scale. Then we have latency, with each reasoning step, extending response time. The catch here is that you don’t know how many steps an agent will take and how many times iterate before giving an answer.</p><p>Finally, the debugging part. Due to the agent’s unpredictable decision-making, it’s more difficult to get to the bottom of the failure because the next time you try to replicate the same error, it might not make that mistake. To find out what went wrong, you have to retrace the agent’s full thought process step by step.</p><h3>Final Thoughts</h3><p>Don’t look for the best RAG strategy because it doesn’t exist. The point is that there is a problem you can solve more effectively with a given strategy or a combination of several. Every project is a unique combination of trial, error, and breakthrough moments. The absence of a clear-cut road to production-ready AI applications is what keeps us on our toes and makes AI engineering indeed exciting.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=853a51f1daa1" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>