<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ville Kuosmanen on Medium]]></title>
        <description><![CDATA[Stories by Ville Kuosmanen on Medium]]></description>
        <link>https://medium.com/@villekuosmanen?source=rss-1a700a66e281------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*zwKa-neHEGj0KvLR0zJabw.jpeg</url>
            <title>Stories by Ville Kuosmanen on Medium</title>
            <link>https://medium.com/@villekuosmanen?source=rss-1a700a66e281------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 12:37:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@villekuosmanen/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Research Notes From Reproducing RL Token]]></title>
            <link>https://villekuosmanen.medium.com/research-notes-from-reproducing-rl-token-f375ecfd3c28?source=rss-1a700a66e281------2</link>
            <guid isPermaLink="false">https://medium.com/p/f375ecfd3c28</guid>
            <category><![CDATA[vision-language-action]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Ville Kuosmanen]]></dc:creator>
            <pubDate>Thu, 23 Apr 2026 21:18:43 GMT</pubDate>
            <atom:updated>2026-04-23T21:18:43.684Z</atom:updated>
            <content:encoded><![CDATA[<h4>Open-sourcing our research notebook from post-training VLA models for high reliability</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*uLykrVpmz3Hep37YCPxIYg.png" /></figure><blockquote>The definition of insanity is doing the same thing over and over again and expecting different results.</blockquote><blockquote><em>Anonymous Philosopher</em></blockquote><p>We’ve been playing with our reproduction of <a href="https://www.pi.website/research/rlt">RL Token, a VLA post-training method proposed by Physical Intelligence</a>, for the past few weeks. Post-training via online RL holds great promise for allowing 3rd-party developers to adapt pre-trained robotics foundation models for new tasks and environments, and optimise success rates close to 100% along with improvements in policy execution speed. Simply rolling out a pre-trained VLA model often results in the model making the same mistakes over and over again — RL post-training aims to correct these behaviours by learning to reinforce good behaviours as well as to avoid bad ones.</p><p>Sounds excellent — now how does it work in practice? The following is a technical deep dive into our findings. We advise opening the RL Token paper, as well as an AI Chat Assistant on a separate tab or screen, and consulting these resources when necessary as we expect the reader to maintain a strong working knowledge of the reference paper.</p><h4>Engineering</h4><p>The original RL Token paper does not publicise the source code, so we have to reproduce it based on the paper description. We have a few other resources to use as well:<br>1. The <a href="https://github.com/Physical-Intelligence/openpi">openpi repository from Physical Intelligence</a>, providing pre-trained checkpoints as well as training and inference code for the pi0.5 foundation model.<br>2. The open implementation of <a href="https://huggingface.co/docs/lerobot/hilserl">HIL-SERL in Hugging Face’s LeRobot</a>, which contains many engineering-heavy components needed. This includes the gRPC client-server setup used for Learner and Actor process communication, Replay Buffers used in the learner, Learner Service etc.</p><p>We start by implementing the <a href="https://github.com/pravsels/openpi/pull/6">RL Token auto-encoder</a> and training the RL Token on top of a frozen, fine-tuned VLA checkpoint. We then adapt the Learner/Actor setup from LeRobot’s HIL-SERL implementation to our robot setup. I won’t elaborate too much on the implementation details, as they are complicated and largely not relevant to the findings. Our training task is a simple block stacking task. The frozen VLA model for the task is able to pick up blocks with fairly good accuracy, but is mostly unable to place them due to issues with aligning the blocks on top of each other, and an inherent hesitancy to open the gripper.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FRLpTtKS3_Cs%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DRLpTtKS3_Cs&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FRLpTtKS3_Cs%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/fcd72f882e7c9ddd5e374bac082e4104/href">https://medium.com/media/fcd72f882e7c9ddd5e374bac082e4104/href</a></iframe><p>Before moving to online RL training, we need to understand whether the RL Token we trained contains useful information about the VLA’s internal states? If it doesn’t, our RL actor will be working on top of noise and will fail to learn anything useful. We test this by running a principal component analysis and a t-SNE visualisation over the captured RL Tokens in successful and failed episodes where the location of each block is kept roughly the same. The results end up clustering with far more clarity than expected — the ordinary pick and place frames form a smooth curve across all episodes, with failed grasps, and a failed placements forming two clusters in the middle, indicating the RL Token contains useful information into the inner workings of the frozen VLA policy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XWpED_xdP6nCwea94qvIcw.png" /><figcaption>An annotated visualisation of 2 successful and 2 failed episodes in the t-SNE feature space.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/973/1*ihXxb1xfv_yX8sML6newUA.png" /><figcaption>The full visualisation of 20 episodes. The smooth curve, along with deviations for failed grasps and block tower collapses, are easily visible with only a few outliers. Dot brightness indicates the ordering of the frame within the episode — the tail on the right consists of frames from the beginning of the episode.</figcaption></figure><h4>The Hyperparameter Search</h4><p>Much of the time spent iterating on our online RL setup was spent finding the right hyperparameters. RL algorithms are notoriously sensitive to them, and a particular challenge with real-world RL in robotics is clock time. Any change we make requires running the Actor process for &gt;1h before knowing with confidence what effect your changes had. Sometimes a change in hyperparameters can cause the model to learn slower with a higher ceiling for instance. How would you evaluate this might be happening?</p><p>Here is a non-exhaustive list of the hyperparameters we find meaningfully impact the performance of the Actor:</p><ul><li>bc_beta: this hyperparameter controls how much we penalise the actor for deviating from the frozen VLA’s action predictions, and acts as a regulariser. We use bc_beta=0.05, noting that larger values constrained the policy too much, with smaller values resulting faster, unpredictable motions. The original paper does not state the specific bc_beta value used. The magnitude of bc_beta also depends on whether the policy predicts absolute or relative actions, and whether the RL actor works on normalised or unnormalised action space (we use normalised relative actions). When using absolute actions, your practical action space is going to be more narrow, meaning large changes in actions will result in large movements, thus likely requiring bc_beta to be larger to regularise the policy more. We also add an additional modification here: our policy struggles with the gripper controls in particular, so we decrease the gripper bc_beta to 0.01 to allow more exploration of the gripper positioning. Exploring more with the gripper is safer than with joints that carry more leverage, and this resulted in much faster learning in our setup.</li><li>Discount factor: controls how much future rewards are discounted. Lowering the value (i.e. increasing the discount, lower value means higher discount) encourages the robot to move faster which may come at the cost of precision. We use discount=0.985, for a task with a timeout of 30s. Note that your control frequency affects the discount factor — larger control frequencies mean larger number of actions for the same time horizon, thus increasing control frequency means we need to increase the discount factor (i.e. lower the discount) all else being equal.</li><li>We use two separate replay buffers — a demo buffer initialised with data from previous Actor runs, and an online buffer containing data from the current policy’s rollouts. We warm up the actor with 4k–20k steps from the demo buffer (depending on how much data we had there) or run the frozen VLA for 20 episodes if demo buffer does not exist. We do not implement any methods to balance the sampling of successes and failures from the buffers, and we let the online buffer grow as we find clearing the online buffer periodically has a negative impact of performance. We reset the online buffer between Learner runs, unless resuming training from a previous Learner checkpoint.</li><li>The paper describes two network sizes for RL experts: [256, 256] for a simple task, and [512, 512, 512] for a more complex one. We started with the smaller network but found learning to be very slow, increasing network size and updates-to-data ratio (from 5 to 10) increased learning efficiency by quite a bit. Adding a 4th layer to the larger network increased learning efficiency ever further, to the point that we were able to get successes with offline-only training for a base policy that scored 0%, though this caused stability issues and we ended up sticking to the larger of the networks described in the paper.</li><li>We finally add a few modifications of our own. To balance the low bc_beta value, we add an additional jerk regulariser to encourage the actor to predict smoother, less jerky actions. This works well in our setup. We also try giving the actor a longer context of proposed actions (h=40) than the action chunk it predicts (h=20) with the intention of allowing the actor to “compress” actions into faster motions. This speeds up the policy when bc_beta is low, but has less effect at bc_beta-=0.05. We think it’s a promising idea nonetheless.</li></ul><h4>Handling variance</h4><p>We started with an objecting of picking a block from anywhere in the manipulation area, and placing it on the block tower. The best hyperparameters achieved ~40% success rates after 1 hour of learning, after which learning seemed to plateau. We believe this is because of the inherent variance of the task. RLT is designed to refine actions and not invent totally new ones. An MLP network is also not designed to handle multi-modality the same way diffusion does.</p><p>To limit the impact of variance and focus on RL Token’s ability to optimise precision, we picked the hardest single block tower configuration we could find, which was to place the green block on top of a three-tall tower with the yellow block on top (somehow, the policy really struggles with aligning the green block on the yellow one, while the reverse seems to be easier. I don’t know why). The positions and angles of the block towers are fixed to ~5cm and ~15 degree rotational margin. This task would be easy to solve with many alternative methods (so don’t @ me for the task being too easy!), so we intend it as an illustration of RL Token’s ability to optimise a poorly-performing base model using online learning only, and not as a reflection of RL Token’s ability to solve hard tasks (when we eliminate variance, a simple motion planning system with an occupancy map would score &gt;99%, and even an ACT policy would likely score &gt;90%).</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F1WPvf1Tnycg%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D1WPvf1Tnycg&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F1WPvf1Tnycg%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/96c4d94c80d0a7eec83dc8fbe3e9cd34/href">https://medium.com/media/96c4d94c80d0a7eec83dc8fbe3e9cd34/href</a></iframe><p><strong>The trained RL Expert achieves ~90% success rate in one hour of clock time</strong>, though this success rate is not consistent during online training. After a period of steady improvement, the policy sometimes suffers repeated failures and then settles to a temporarily lower success rate (e.g. 60%). Perhaps our learning rate is too high, so the policy is unable to recover from bad exploration and unable to explore with small enough gradient steps?</p><p>Training the policy this way makes it sensitive to any variance in the environment— changing the start position of the <strong>robot</strong> dropped the success rates on some evals from 90% to 33%, even if the position of the blocks remains in distribution. There’s probably some balance between adding and removing variance, together with hyperparam tuning, that we haven’t quite found yet.</p><h4>Preventing Reward Hacking</h4><p>The biggest challenge we saw during online training was reward hacking. Increasing the network size made the RL expert more capable, but made it far more likely for the actor to find a set of OOD actions that the critic assigns abnormally high Q-values. This quickly causes Q-values to spike and derails training, as well as causing the robot to behave dangerously. Through chance, I happened to capture one of these anomalies in real-time while starting to record an policy evaluation</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FYmtPvrq_sWw%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DYmtPvrq_sWw&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FYmtPvrq_sWw%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/0d94c3570dfb5be920286dab92eec701/href">https://medium.com/media/0d94c3570dfb5be920286dab92eec701/href</a></iframe><p><em>(these behaviours quickly go from funny to terrifying when it’s your robot and $$$ on the line)</em></p><p>Here’s the WandB metrics seen in the above video — notice the spiking Q-values, mean action deviation, critic loss, and the actor loss turning <strong>negative</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gRRDF9fgvM1FcC4-Uc2YQA.png" /><figcaption>not the metrics you want to see during training.</figcaption></figure><p>After this anomaly, we halved our gradient clip hyperparameter (40 -&gt; 20) and doubled the number of critics from 2 to 4. The critic change made the Q-anomaly less likely, while the gradient clipping meant the critic could sometimes recover from the reward hacking, by slowing the process down just enough for the critic weights to adjust.</p><h4>Observations and Open Questions</h4><p>We haven’t figured out everything yet, and still have many open questions to answer on the best ways to train RL experts in the real world using the RL Token methodology.</p><ol><li>Can we reduce bc_beta further? testing at 0.01 and the extended action context made the robot significantly faster at the cost of safety. Is there a way to reduce unsafe exploration while reducing regularisation? Or should we increase the underlying control frequency and using the RL expert to adapt to the resulting shift in controller behaviour, if we want faster policies?</li><li>Increasing network sizes reduced learning time and improved performance but required a larger utd ratio and caused problems with reward hacking. Would performance continue to improve with larger networks? Are there hyperparameter configurations that prevent reward hacking while allowing the actor to explore and learn safely?</li><li>Our frozen π0.5 policy performs poorly, which likely limits the performance of the RL expert. We have since improved the base policy, which means the RL experts will likely learn faster and achieve ever higher success rates.</li><li>When generating actions using the pi0.5 model, we sample random noise for the diffusion process. The RL token is captured <strong>before</strong> this noise sampling process, meaning the relation between the RL Token features and the generated actions are not deterministic, especially in multimodal scenarios. Would using a consistent starting noise vector for action generation improve the RL expert’s learning efficiency or performance, particularly in high-variance scenarios?</li><li>How do we solve the unstable training and periodic regressions that currently prevent reaching success rates above ~90%? Would learning rate scheduling help? Or will these issues go away after we improve our base policy and increase bc regularisation?</li></ol><h4>Conclusion</h4><p>Online RL as described by Physical Intelligence really does work. We went from 0% to 80–95% in one hour of wall clock time, albeit for a very simple task. You have to start from somewhere. But as much as I had hoped, adding RL experts on top of poor frozen policies won’t solve all your problems. The worse your base policy is, the more time you spend babysitting RL policies — time you could spend collecting teleoperation or dAgger-style interventions for improving the frozen policy itself. Though I feel automating some manual parts of the RL loop, such as success detection or environment reset, and scaling from one to many Actors, could help speed up learning and enable ever higher success rates and speeds.</p><p>Variance remains an unsolved problem. The method proposed in RL Token seems to work best for low-variance tasks (e.g. screwing rather than folding clothes), which is a similar finding to HIL-SERL and many other online RL methods. Although I’d be interested in trying whether a better frozen VLA policy would also manage high-variance scenarios well.</p><p>Our RL training runs were also highly sensitive to hyperparameters, and keeping a learner process on-track is a challenge we haven’t solved yet. How much of this is engineering and hyperparam selection and how much is genuine challenges with the methodology? We’re not sure about it.</p><p>Our robot suffered some wear and tear to the gripper fingers when online learning, and we had a few strong hits against the table that would have broken less robust hardware, mainly as a consequence of reward hacking. I’ve heard of humanoid hands being broken during real-word RL experiments. Improving low-level safety systems to prevent cracked actors from damaging the robot and environment is essential.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*SGhGcYxeaZn7WBxvtL3tRA.jpeg" /><figcaption>Wear and tear in robot fingertips, caused by pressing into hard objects and scraping the table carelessly during online RL exploration.</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f375ecfd3c28" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Guiding Action Diffusion Policies in Robots via Motion Tokens]]></title>
            <link>https://villekuosmanen.medium.com/guiding-action-diffusion-policies-in-robots-via-motion-tokens-f8a5fe4ef8b1?source=rss-1a700a66e281------2</link>
            <guid isPermaLink="false">https://medium.com/p/f8a5fe4ef8b1</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[physical-ai]]></category>
            <category><![CDATA[vision-language-action]]></category>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[computer-science]]></category>
            <dc:creator><![CDATA[Ville Kuosmanen]]></dc:creator>
            <pubDate>Mon, 06 Apr 2026 19:10:33 GMT</pubDate>
            <atom:updated>2026-04-06T19:10:33.857Z</atom:updated>
            <content:encoded><![CDATA[<h4>Prompting robotics policies with a language they understand</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*1ZkgWsKFOPH_KhxLNfxyvg.png" /></figure><p>Today’s AI-driven robotics has a hard boundary between two control modes: teleoperation, used for data collection and manual correction, and autonomous policy execution. When a policy fails mid-task, the operator has no option but to halt execution entirely and take over manually — there is no native way to nudge the robot in the right direction without breaking out of the autonomous loop entirely.</p><p>Policies built on Vision-Language model backbones [1][2][3] do allow users to guide the robot towards desired actions via language instructions. This prompt is usually fixed for the lifetime of the task, though some methods [4][5] have explored using language commands as an intervention method, preferring to guide the policy across challenging subtasks rather than stopping the policy to manually teleoperate it. This guidance tends to work best with semantic errors, such as when the robot is approaching the wrong object or is misunderstanding the task entirely.</p><p>While language allows us to interact with robots at high and low levels, a domain gap from words to spatial direction, such as detecting “left” from “right”, remains. If we want to guide a robot to move to the left, or to grasp an object at a different angle via a 90 degree wrist roll, is language really the most optimal medium to give this guidance in? Some works have explored using rough sketches [6] or trajectories [7] for prompting, which offer one way of guiding policies but aren’t quite “native” to the way robots actually model actions either.</p><p>This work proposes guiding diffusion-based robot action policies [8][9][10] using a set of learned motion tokens. These motion tokens enable a native interaction mode for guiding the policy towards, or away from, a particular cardinal direction that requires no specialist datasets to train with, is intuitive for humans to use, and easy and cheap to implement via various devices. In fact, we use a Logitech G F710 gamepad purchased for £37 as an interface for the guidance input.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F8YbcDGx3XAA%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D8YbcDGx3XAA&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F8YbcDGx3XAA%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ff7ca55f507e4597705a0306a5ae3764/href">https://medium.com/media/ff7ca55f507e4597705a0306a5ae3764/href</a></iframe><h4>Theory</h4><p>Various work over the past few years has attempted to explain the representations learned for Action Diffusion Policies [8]. Most significantly,[9] proposes that diffusion policies don’t generalise, but rather memorise a set of observation-action pairs from the training set, and the models predict actions at test-time via a form of fuzzy similarity search rather than true generalisation.</p><p>This lookup policy hypothesis implies that sufficiently trained diffusion policies have two main failure modes:</p><ol><li>Inability to generate an appropriate action because the current scenario is sufficiently different to what the model saw during training</li><li>Inability to select the right action out of many reasonable actions, a common problem in tasks with many acceptable target objects such as bin picking</li></ol><p>This work addresses the second failure mode. Under the lookup table hypothesis, a trained diffusion policy operates by retrieving action sequences similar to what it has seen during training — effectively running a weighted similarity search over the training distribution conditioned on the current observation. For many tasks, this search returns a set of plausible candidates, some of which make task progress and some of which do not. The policy lacks the higher-level reasoning needed to discriminate between them. Motion token guidance aims to bias this search — not to generate novel motions the policy has never seen, but to make it more likely that the retrieved action chunk is one of the good candidates already present in the training distribution.</p><p>Our hypothesis then becomes:</p><blockquote>If diffusion policies fail because the model is incapable of generating actions to make progress in the task, guiding the policy via motion tokens should have no effect. Whereas, if policy failures are caused by a failure to select good actions in a sea of mediocre ones, human guidance via motion tokens should allow the model to achieve significantly higher success rates than unguided baselines.</blockquote><h4>Method</h4><p>We start with a standard implementation of the Robotics Diffusion Policy [8] from the LeRobot library [11]. We then extend the global conditioning of the diffusion policy inputs from proprioceptive state and vision encoder embeddings to also include a set of 7 motion tokens {x, y, z, roll, pitch, yaw, gripper}, corresponding to the 6 cardinal directions plus gripper. This conditioning method is similar to the Advantage token conditioning proposed in [12].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eznGFX-IAM2HNewG2u5Ipg.jpeg" /><figcaption>Proposed extension of Diffusion Policy [8] with motion token based conditioning.</figcaption></figure><p>We then augment our dataset of robot data with motion token information. Because the datasets we use already contain proprioceptive state and actions, we can calculate the motion token to imitate for a frame <strong>f</strong> by looking at the change in position of the robot in cartesian space <strong>Δp</strong>, across a horizon <strong>H</strong> of future actions<strong>.</strong></p><blockquote><strong>Δp = f(t + H) − f(t)</strong></blockquote><p>We calculate <strong>Δp</strong> across the whole dataset, and for each of our 7 degrees of freedom label the lowest N percentile with a (-1) motion token, and the highest N with a (+1) token (N=15%, horizon=20, FPS=20), with the motion token for all other frames set to 0. <strong>Δp</strong> for each degree of freedom is calculated separately, meaning a given frame can be labelled with more than one non-zero motion token.</p><p>At training time we use two methods of dropout to condition our model on motion tokens. We randomly mask all motion tokens 20% of the time, and we have an additional 30% to mask the token of any particular degree of freedom. The former dropout exists to support classifier-free guidance by showing the model examples of unguided samples, while the latter is used to break up frames with multiple motion tokens active (e.g. move right AND roll clockwise) to support guiding the model with different combinations of active motion primitives.</p><p>At inference-time, we implement both CFG [13] and CFG++ [14], and experiment with various values of guidance <strong>γ</strong>. We also experiment with only guiding a subset of denoising steps at inference time. We find (anecdotally) the best compromise between smoothness of generated trajectories and responsiveness to guidance to be the CFG++ algorithm with γ=2.5 and applying guidance on the first 75% of denoising steps only, leaving the latter 25% unguided. Applying a strong γ early on ensures enough guidance, while leaving the last few steps unguided improves the smoothness of the generated trajectories.</p><h4>Results</h4><p>Here are videos of a guided diffusion policy trained with ~300 episodes of block stacking, of which 200 were recorded via dAgger-style human interventions [15]. The frames with a green border were human-guided via motion tokens.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FmppeJ4RVlzA%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DmppeJ4RVlzA&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FmppeJ4RVlzA%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/45d9f494d154795dd042ea90e4de7d0d/href">https://medium.com/media/45d9f494d154795dd042ea90e4de7d0d/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FWO0gn-RuUX0%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DWO0gn-RuUX0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FWO0gn-RuUX0%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/b8e7fa91c2c63fced755fc9b00dc102b/href">https://medium.com/media/b8e7fa91c2c63fced755fc9b00dc102b/href</a></iframe><p>For quantitative results, we compare success rates for the same diffusion policy between unguided and human-guided rollouts. Each rollout has a hard 2-minute timeout, and we report the highest achieved tower in cases where the robot knocks over a tower it managed to stack (to consider a block as stacked, it has to stay on the tower for a few seconds without falling over).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ejyQ_2dW2nrlMqvchplNdQ.png" /><figcaption>Unguided versus human-guided diffusion policy success rates in block stacking, 20 rollouts each.</figcaption></figure><p>The results show a consistent and substantial improvement across all tower heights when human guidance is applied, with the effect becoming more pronounced at higher difficulty levels.</p><p><strong><em>Guided rollouts outperform unguided policy rollouts</em></strong></p><p>This suggests most failures in diffusion policies are due to the lack of high-level intelligence in selecting the actions that make the most progress amongst many possible actions. This injection of high-level intelligence is what large robotics foundation models are aiming to address by adding a pre-trained vision-language [1][2][3] or video model backbone [16] into a robotics action expert.</p><p><strong>Motion token guidance is a complement to, not yet a replacement for, teleoperation</strong></p><p>Not all teleoperation systems are made equal, but a good teleoperation system allows a human to achieve most non-dexterous tasks with &gt;99% success rates, which is far higher than what motion token guidance achieves in our task. One way to improve the reliability of guidance could be to allow the human operator to adjust γ dynamically. Using a high γ degrades performance in highly precise tasks, such as in precisely placing the block on top of a tower (seen in the success rate drop-off in stacking the final block), while a low γ hurts the speed and efficiency of completing tasks as well as increasing the likelihood of the robot getting stuck.</p><p><strong><em>Performance of guiding declines rapidly in out of distribution scenarios</em></strong></p><p>Guiding is heavily reliant on the policy’s training data covering scenarios faced by the policy during inference rollouts. When hitting a “here be dragons” section of the manifold, the policy has few examples to understand what motion-based guidance means. This reinforces the findings of [9] that diffusion policies have weaker than commonly understood generalisation abilities, and seeing examples of motion guidance in one scenario doesn’t automatically help the model use them for an unseen scenario. Applying guidance in OOD scenarios tends to result in one of two outcomes:</p><ol><li>Little to no impact in predicted actions. A robot that drifted too far left in the manipulation area refuses to move right, even when prompted with strong motion guidance.</li><li>Dangerous shakiness. Moving the policy off-manifold can also cause dangerously shaky behaviour due to lack of training data in this environment. The stronger guidance we apply, the more we amplify any random jitters in the predicted actions. Detecting and stopping these motions via downstream safety systems, such as setting maximum joint position change, velocity, or torque limits thus becomes essential.</li></ol><p>While these findings may sound like bad news for the future of motion guidance, there is a positive spin to the method. As robot policies get stronger and cover more scenarios they are expected to see at inference-time, motion and language-based guidance is expected to become more reliable and viable as a control mode. Autonomous cars like Waymo are nowadays guided via question answering, waypoints, and trajectories [17] when under human supervision, rather than directly teleoperating them with a steering wheel and pedals. As robot policies get stronger, direct teleoperation is likely to be gradually phased out in favour of multi-modal prompting and guidance, of which motion token guidance is one candidate.</p><p>Here are a few examples showing failure cases where motion token steering was not able to complete the block stacking task. These cases would require a manual reset or teleoperation-based remote interventions.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FLqfoHoNBDm0%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DLqfoHoNBDm0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FLqfoHoNBDm0%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/470eb6c64b8166005b66dc19fca3587c/href">https://medium.com/media/470eb6c64b8166005b66dc19fca3587c/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FLFWGDITayls%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DLFWGDITayls&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FLFWGDITayls%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/c9b367f3354f6114d2017b87ebb07c0a/href">https://medium.com/media/c9b367f3354f6114d2017b87ebb07c0a/href</a></iframe><h4>Future work</h4><p>This work focuses on human prompting of robot policies via motion tokens. However, the generation of motion token guidance is by no means restricted to humans; many recent robotics foundation models describe a hierarchical system in which a fast low-level action policy (the “action expert”, or “System 1”) is prompted by a slower high-level, “System 2“, reasoning model [18][19][5].&quot; Integrating motion tokens into the input space of the System 1 action policy would enable the high-level system 2 model to guide the robot policy’s motions natively, which could help augment language-based prompting, and improve instruction following and overall performance of the system.</p><p>If you think this work is worth citing in your research, please use this citation to acknowledge the effort taken to run the experiments and let the world know about the results.</p><pre>@misc{kuosmanen2026guiding,<br>    author = {Kuosmanen, Ville},<br>    title = {Guiding Action Diffusion Policies in Robots via Motion Tokens},<br>    howpublished = &quot;\url{}&quot;,<br>    year = {2026}<br>}</pre><h4>References</h4><p>[1] A. Brohan et al., “RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control,” arXiv:2307.15818, 2023.<br>[2] K. Black et al., “π₀: A Vision-Language-Action Flow Model for General Robot Control,” arXiv:2410.24164, 2026.<br>[3] Physical Intelligence et al., “π₀.₅: a Vision-Language-Action Model with Open-World Generalization,” arXiv:2504.16054, 2025.<br>[4] L. X. Shi et al., “Yell At Your Robot: Improving On-the-Fly from Language Corrections,” arXiv:2403.12910, 2024.<br>[5] L. X. Shi et al., “Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models,” arXiv:2502.19417, 2025.<br>[6] P. Sundaresan et al., “RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketches,” arXiv:2403.02709, 2024.<br>[7] J. Gu et al., “RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches,” arXiv:2311.01977, 2023.<br>[8] C. Chi et al., “Diffusion Policy: Visuomotor Policy Learning via Action Diffusion,” <em>The International Journal of Robotics Research</em>, 2024.<br>[9] C. He et al., “Demystifying Diffusion Policies: Action Memorization and Simple Lookup Table Alternatives,” arXiv:2505.05787, 2025.<br>[10] TRI LBM Team et al., “A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulation,” arXiv:2507.05331, 2025.<br>[11] R. Cadene et al., “LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch,” GitHub, 2024. [Online]. Available: <a href="https://github.com/huggingface/lerobot">https://github.com/huggingface/lerobot</a><br>[12] Physical Intelligence et al., “π*₀.₆: a VLA That Learns From Experience,” arXiv:2511.14759, 2025.<br>[13] J. Ho and T. Salimans, “Classifier-Free Diffusion Guidance,” arXiv:2207.12598, 2022.<br>[14] H. Chung et al., “CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models,” arXiv:2406.08070, 2024.<br>[15] M. Kelly et al., “HG-DAgger: Interactive Imitation Learning with Human Experts,” arXiv:1810.02890, 2019.<br>[16] E. Nava et al., “mimic-one: a Scalable Model Recipe for General Purpose Robot Dexterity,” arXiv:2506.11916, 2025.<br>[17] “Fleet Response Providing the Waymo Driver Additional Context,” Waymo, 2025. [Online]. Available: <a href="https://www.youtube.com/watch?v=T0WtBFEfAyo">https://www.youtube.com/watch?v=T0WtBFEfAyo</a><br>[18] J. Zhang et al., “HiRT: Enhancing Robotic Control with Hierarchical Robot Transformers,” arXiv:2410.05273, 2025.<br>[19] “Helix: A Vision-Language-Action Model for Generalist Humanoid Control,” Figure AI, 2025. [Online]. Available: <a href="https://www.figure.ai/news/helix">https://www.figure.ai/news/helix</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f8a5fe4ef8b1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Does learning from experience benefit small AI robotics models?]]></title>
            <link>https://villekuosmanen.medium.com/does-learning-from-experience-benefit-small-ai-robotics-models-3de024f930e0?source=rss-1a700a66e281------2</link>
            <guid isPermaLink="false">https://medium.com/p/3de024f930e0</guid>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[humanoid-robot]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <dc:creator><![CDATA[Ville Kuosmanen]]></dc:creator>
            <pubDate>Mon, 22 Dec 2025 11:11:49 GMT</pubDate>
            <atom:updated>2025-12-22T11:11:49.879Z</atom:updated>
            <content:encoded><![CDATA[<h4>Replicating the RL loop behind Physical Intelligence’s Pi*0.6 foundation model without VLAs or diffusion</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*dLde8xagq05pDOsqVd4WPQ.png" /></figure><p>Behavioural Cloning (BC) is the ruling paradigm in AI robotics, allowing AI models to learn to imitate human-controlled robots based on successful actions. When training a BC policy, we hide errors and mistakes from our dataset to train on successful episodes only — we don’t want to learn to imitate mistakes after all!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/460/0*4ZM928Bmu5JHc2hP.jpg" /><figcaption>The failed grasp — it ought to be in the training dataset but it isn’t…</figcaption></figure><p>The most promising direction in AI robotics today is “learning from experience”. Because BC policies only learn to imitate successful actions, they struggle to recover from errors that inevitably happen in messy real-world scenarios. Learning to distinguish good actions from bad ones would help the policy learn recovery behaviour and optimise the speed and efficiency of motions. Post-training via Reinforcement Learning (RL) promises just that — learning from your mistakes rather than merely imitating human behaviour!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wKlDB74wwBOP2hFX" /><figcaption>Children use a variety of learning strategies. Many play sessions involve imitating adults, such as when “playing house”, while learning to ride a bike involves falling over and over until you learn to balance yourself — learning from experience! Photo by <a href="https://unsplash.com/@jakubpabis?utm_source=medium&amp;utm_medium=referral">Jakub Pabis</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Perhaps the most exciting research paper in robotics this autumn came from Physical Intelligence, <a href="https://www.physicalintelligence.company/blog/pistar06">who publicised their method <strong>Pi*0.6</strong> for post-training robotics foundation models via offline RL</a>. The most striking part about this paper is its simplicity — you first classify actions as “strong” or “weak” based on how advantageous they are, then simply add this classification into the context window of the action policy. At runtime, we simply ask for “strong” actions, allowing us to sample from the set of actions that produced good outcomes during training. No need for PPO or other complex gradient methods, or issues in regularisation or catastrophic forgetting that often follows from them. Just say “I want good actions” and the robot responds with them. Sounds too good to be true?</p><p>While Physical Intelligence’s paper demonstrates a post-training method for large VLA foundation models, I wanted to test if the same advantage-conditioning mechanism works in smaller models too. ACT is transformer-based policy often used as a baseline in robot learning due to its simplicity, ease of training, and comparatively low latency during inference time. As I’ve demonstrated ACT can function as a strong value model, I was wondering if Pi*0.6-style advantage conditioning would work on the architecture as well? Let’s try it out!</p><p>We only need a few changes to train advantage-conditioned ACT policies.</p><p>To calculate ground-truth reward labels for each frame in our dataset, we manually label each episode as either a success or failure. Failures are caused by time running out, or by the robot making a mistake that causes a block to fall out of its reach.</p><p>For successful episodes, our reward is the number of frames until the end (e.g. if there’s 200 frames left in the episode, the reward will be —200), while for failures we add some large negative constant C (for example <em>C = —10,000</em>) on top of the number of frames remaining to ensure a very small reward. We then train a value function to estimate this ground-truth reward, normalising the values between 0 and 1 (or -1 and 0 as in the pi paper, it doesn’t matter in practice). We use <a href="https://github.com/villekuosmanen/rewACT">RewACT</a>, my open-source modification of the ACT policy, as a value function, replacing the original regression-based value prediction head with a binned classifier, trained with cross-entropy loss, to match the pi paper. A useful property of this method is that the lowest bin in effect estimates the chance of failure at a given time step, potentially acting as a useful signal in itself. Based on some experimentation, RewACT is able to learn a fairly robust representation of expected rewards for the tower building task I used as an example for this project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fUo2eC2w1BXoWskjk1I6aw.gif" /><figcaption>A visualisation of predicted rewards in an unseen example (3x speed) shows that our value function learns to predict expected rewards fairly well.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fC8AbT0nv3vlGhDjGhU_4w.gif" /><figcaption>By plotting the lowest bin probability, we can also approximate failure prediction as an interesting side-effect. The training dataset is imbalanced and contains few complete failures and a lot more successes via recoveries — adding more failure data would likely make these predictions more accurate and potentially useful for monitoring robot behaviour during runtime</figcaption></figure><p>The value model’s predictions are used in calculating the <strong>advantage</strong> of each action. This is determined by comparing the difference between the current predicted reward with the reward after a horizon of actions N (here, N=50), and whether the predicted change in reward is more or less than ground truth labels. The intuition behind this calculation is, “strong” actions should increase reward over the time horizon more than the average action (see the Pi paper for a complete mathematical definition). We also use a few heuristics — human interventions are always assigned a positive advantage (as done in Pi’s paper), while <em>N</em> actions leading up to an intervention are set to negative advantage as they required an intervention (a heuristic added by myself after some trial and error). Again, we can plot the calculated advantage values for the above episode from our training dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7oKBIZquof67BwBbm2Z1WQ.gif" /><figcaption>The plotted advantage values closely resemble how I’d label them as a human, I was surprised at how good this part of the system gets</figcaption></figure><p>So far so good. Now, all we need to do is add the Advantage value as an extra token into the inputs of our ACT model (with a dropout probability of 0.3, as in the Pi paper) and iteratively train the model. As we run through the RL loop, we should get better and better results. Right?</p><p>Here’s a rollout from an “advantage-conditioned” ACT policy I trained. The goal is to stack the cubes in a tower, with the only constraint being that the purple block needs to be at the bottom. Judge for yourself.</p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DO7bi1T4Ed0U&amp;type=text%2Fhtml&amp;schema=youtube&amp;display_name=YouTube&amp;src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FO7bi1T4Ed0U%3Ffeature%3Doembed" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/bd44bf4b2024083e54110a1dc87cc3ca/href">https://medium.com/media/bd44bf4b2024083e54110a1dc87cc3ca/href</a></iframe><p>As the mathematically-inclined readers may have anticipated, it doesn’t work. Looks like adding a single token for advantage into the context of a transformer model doesn’t neatly separate the distribution of positive and negative actions the way conditioning works for diffusion-based policies. Learning to mimic the failures is not going to lead to positive results.</p><p>“Perhaps the problem is with your data, or the task is too difficult?”, you say. As an ablation, I also trained a regular ACT model with human demonstrations and corrections only, discarding all policy-controlled frames from the dataset.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FyytKeUo61nU%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DyytKeUo61nU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FyytKeUo61nU%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/70959a385ddcd3827bbfdbe4dd8ac153/href">https://medium.com/media/70959a385ddcd3827bbfdbe4dd8ac153/href</a></iframe><p>Much better results! This policy still makes occasional mistakes, which we could probably mostly clean up after a few more rounds of human corrections. Here’s more example rollouts from a variety of starting positions, including failures and successes with minor human help, for those who like watching robots and are suspicious of cherry-picked demos.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FQi-vjL14EUM%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DQi-vjL14EUM&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FQi-vjL14EUM%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/8f175e85e545f7d2dc88457fd1d57b2c/href">https://medium.com/media/8f175e85e545f7d2dc88457fd1d57b2c/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FEe467OV5nCY%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DEe467OV5nCY&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FEe467OV5nCY%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/202edccbbac462b4bd6abd3105425f31/href">https://medium.com/media/202edccbbac462b4bd6abd3105425f31/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F7AW_YWQ4bPo%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D7AW_YWQ4bPo&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F7AW_YWQ4bPo%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/7e0b3c3f623a6f538d031e70c1f3f0e6/href">https://medium.com/media/7e0b3c3f623a6f538d031e70c1f3f0e6/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FB2VFEgTMIio%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DB2VFEgTMIio&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FB2VFEgTMIio%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/8db669d7c03945c091e0c2bfa15b6597/href">https://medium.com/media/8db669d7c03945c091e0c2bfa15b6597/href</a></iframe><h4>A closing chapter for ACT</h4><p><a href="https://arxiv.org/abs/2304.13705">Published 2.5 years ago, ACT</a> has been one of the most influential robot learning architectures of the last few years. The small size of the neural network, ability to train with consumer GPUs, low latency cost that doesn’t require asynchronous inference, and behaviour that is generally more “intuitive” than diffusion-based policies has made it a common baseline and a good starting point for robotics developers. Foundation models get all the attention, but if you actually talk to robotics developers it’s remarkable how many admit their flashy demo is “just ACT”.</p><p>This year, the ACT name was repurposed by one of its authors, Tony Zhao, to represent <a href="https://www.sunday.ai/journal/no-robot-data">ACT-1, the Sunday Robotics foundation model</a> that despite the name is not based on the original ACT architecture. This rebranding comes at a fitting time. Small single-task models like ACT have been immensely successful, but their limitations are beginning to show. Learning robot control from scratch every time we train a new task no longer feels appropriate.</p><p>I’ve ran most of my experiments in 2025 on top of the ACT architecture. Now is the right time to say goodbye to it. I might rerun this advantage-conditioning experiment on top of the similarly small <a href="https://diffusion-policy.cs.columbia.edu/">Diffusion Policy</a> if we have time to integrate it with the robot learning stack, but a more appropriate direction for 2026 is focusing on scaling. Like <br>VLAs or not, the future of robotics lies in foundation models. Pre-training remains expensive in both data and compute, but the beauty of foundation models is their relative flexibility for fine-tuning and post-training, as well as applications in other downstream tasks. Many opportunities lie here, let’s see what we can find :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3de024f930e0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sharing GPUs in AI Robotics with CUDA Streams]]></title>
            <link>https://villekuosmanen.medium.com/sharing-gpus-in-ai-robotics-with-cuda-streams-2c56b2d5988b?source=rss-1a700a66e281------2</link>
            <guid isPermaLink="false">https://medium.com/p/2c56b2d5988b</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[cuda]]></category>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Ville Kuosmanen]]></dc:creator>
            <pubDate>Mon, 13 Oct 2025 06:46:47 GMT</pubDate>
            <atom:updated>2025-10-13T06:46:47.689Z</atom:updated>
            <content:encoded><![CDATA[<h4>The challenges of maintaining a control frequency in AI robotics when sharing GPUs with other AI models</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mtiJWrSpji-og01-_a1-Gw.png" /></figure><p>End-to-end control policies in AI robotics, such as ACT, Diffusion policy, and foundation models like Pi0.5 and SmolVLA broadcast their actions at a specific <em>control frequency</em>, for example 20Hz. Unless methods such as <a href="https://huggingface.co/blog/async-robot-inference">asynchronous inference</a> are utilised, the control policy needs to execute in a single step of the chosen control frequency to maintain normal operations. With 20Hz, it means maximum inference time of 50ms — 50Hz only leaves 20ms to predict a single action or chunk.</p><p>From my experience, a small neural network like ACT has no problem meeting these timings under normal conditions [1]. Problems start to arise when we want to run multiple AI models on the same GPU during inference time. There are many reasons for doing this — perhaps our AI pipeline includes computer vision tasks like object detection or segmentation via <a href="https://ai.meta.com/dinov3/">DINO</a>, higher-level planning or reasoning via on-device VLMs, voice AI systems for user interaction, <a href="https://villekuosmanen.medium.com/building-a-simple-out-of-distribution-detector-for-physical-ai-models-using-lerobot-bfa02b4a3876">error detection systems</a> or anything else a robotic application might need.</p><p>The most obvious approach for integrating these models into your system is to run them sequentially, once every control loop:</p><pre>while:<br>  # Record start time<br>  start_loop_t = time.perf_counter()   <br>  <br>  # Capture observations, inc. cameras<br>  obs = robot.capture_observation()<br><br>  # Predict and send actions - real-time requirement<br>  action = policy.select_action(obs)<br>  robot.send_action(action)<br><br>  # Detect objects - no real-time requirements<br>  objects = vision_model.detect_objects(obs)<br><br>  # Wait to match control frequency<br>  dt_s = time.perf_counter() - start_loop_t<br>  busy_wait(1 / fps - dt_s)</pre><p>This works, if the combined inference time of our action and vision models comfortably fits into a single control loop (50ms at 20Hz). What should we do if this is not possible. Most obvious solution — use threads!</p><h4>Concurrency 101 in Python</h4><p>Python’s <a href="https://docs.python.org/3/library/threading.html">threading</a>-package is the the simplest way to add concurrency to an application. We’ll simply create a new thread for our vision model, define some shared variables for cross-thread communication, and we’re sorted. The vision model will run in the background, decoupled from our control frequency, allowing our control policy to meet its time requirements.</p><pre>import threading<br><br># Shared variables for communication between threads<br>latest_obs = None<br>latest_objects = None<br>obs_lock = threading.Lock()<br>objects_lock = threading.Lock()<br><br>def vision_thread():<br>    &quot;&quot;&quot;Background thread for object detection&quot;&quot;&quot;<br>    global latest_obs, latest_objects<br>    <br>    while True:<br>        # Get latest observation<br>        with obs_lock:<br>            obs = latest_obs<br>        <br>        if obs is not None:<br>            # Detect objects - no real-time requirements<br>            objects = vision_model.detect_objects(obs)<br>            <br>            # Update latest objects<br>            with objects_lock:<br>                latest_objects = objects<br><br>        # Sleep for a bit to avoid thread running constantly      <br>        time.sleep(0.01)<br><br># Start background thread<br>thread = threading.Thread(target=vision_thread, daemon=True)<br>thread.start()<br><br>while True:<br>    # Record start time<br>    start_loop_t = time.perf_counter()<br>    <br>    # Capture observations, inc. cameras<br>    obs = robot.capture_observation()<br>    <br>    # Share observation with vision thread<br>    with obs_lock:<br>        latest_obs = obs<br>    <br>    # Predict and send actions - real-time requirement<br>    action = policy.select_action(obs)<br>    robot.send_action(action)<br>    <br>    # Get latest detected objects (may be from previous frame)<br>    with objects_lock:<br>        objects = latest_objects<br>    <br>    # Wait to match control frequency<br>    dt_s = time.perf_counter() - start_loop_t<br>    busy_wait(1 / fps - dt_s)</pre><p>Imagine us trying this — what you’ll see is a stuttering robot, unable to meet its control frequency. Why is this happening?</p><p>Threads are great at distributing CPU-bound work across CPU cores. Nearly every modern processor benefits from multiple cores — thus CPU (or IO) bound work can be neatly parallelised. GPUs work differently.<strong> </strong>By default,<strong> PyTorch queues all GPU operations sequentially, even when they come from different threads.</strong> When multiple threads submit GPU work, the operations execute one after another — no parallelism. In the example above, the GPU executes either our vision model or control policy. While one runs, the other waits.</p><p>Hmm. If <strong>threads</strong> don’t work, maybe <strong>processes</strong> will? Multi-processing often avoids many of the practical challenges in threading. Besides replacing threading with multiprocessing, we also need to ensure we initialise CUDA and reload the vision model weights in our child process, and move all tensors to CPU when communicating across processes.</p><pre>import multiprocessing as mp<br><br># Shared variables (requires special multiprocessing types)<br>latest_obs = mp.Manager().dict()<br>latest_objects = mp.Manager().dict()<br><br>def vision_process():<br>    &quot;&quot;&quot;Background process for object detection&quot;&quot;&quot;<br>    # Initialize CUDA in child process<br>    import torch<br>    torch.cuda.init()<br>    <br>    # Reload model weighs in child process<br>    vision_model_local = load_vision_model()<br>    <br>    while True:<br>        # Get latest observation<br>        obs = latest_obs.get(&#39;obs&#39;, None)<br>        <br>        if obs is not None:            <br>            # Detect objects - no real-time requirements<br>            objects = vision_model_local.detect_objects(obs)<br>            <br>            # Move results to CPU for sharing<br>            objects_cpu = {k: v.cpu() for k, v in objects.items()}<br>            <br>            # Update latest objects<br>            latest_objects[&#39;result&#39;] = objects_cpu<br><br>        # Sleep for a bit to avoid thread running constantly      <br>        time.sleep(0.01)<br><br># Start background process<br>process = mp.Process(target=vision_process, daemon=True)<br>process.start()<br><br>while True:<br>    # Record start time<br>    start_loop_t = time.perf_counter()<br>    <br>    # Capture observations, inc. cameras<br>    obs = robot.capture_observation()<br>    <br>    # Share observation with vision process - moving them to GPU<br>    latest_obs[&#39;obs&#39;] = {k: v.cpu() for k, v in objects.items()}<br>    <br>    # Predict and send actions - real-time requirement<br>    action = policy.select_action(obs)<br>    robot.send_action(action)<br>    <br>    # Get latest detected objects (may be from previous frame)<br>    objects = latest_objects.get(&#39;result&#39;, None)<br>    <br>    # Wait to match control frequency<br>    dt_s = time.perf_counter() - start_loop_t<br>    busy_wait(1 / fps - dt_s)</pre><p>If we run this, our robot still stutters. Why?</p><p>Firstly, the overhead of moving tensors between CPU and GPU many times a second adds to latency. Moving images in particular is going to be costly. The communication overhead, as well as complexity in debugging and initialising CUDA in each process separately means CUDA operations are not the best uses for multiprocessing. However, we <em>should</em> have solved the concurrency problem — our processes are now fully separate so they should not interfere with each other. Yet they clearly do. Why is this the case?</p><h4>CUDA Streams to the rescue</h4><p>Turns out concurrency in GPUs is quite different to what we’re used to with CPUs. A GPU can overlap operations that use different hardware units (like memory transfers and compute), but operations competing for the same resources (like two matrix multiplications on tensor cores) must execute sequentially, as though the tensor cores are “single-threaded”.</p><p>Luckily for us, GPU access operations typically consist of multiple stages such as memory copy and matrix multiplication. <strong>Nvidia GPUs can run CUDA operations concurrently, as long as they use different parts of the GPU — a matmul can be executed concurrently to a memory copy instruction, but two matmuls have to be executed sequentially, regardless of where they came from</strong>. <a href="https://developer.download.nvidia.com/CUDA/training/StreamsAndConcurrencyWebinar.pdf">This diagram from a presentation by Steve Rennich of Nvidia explains how this interleaving</a> works in practice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Cy0sAFe-jZXQvaeFrC-1Q.png" /><figcaption>Diagram by <a href="https://developer.download.nvidia.com/CUDA/training/StreamsAndConcurrencyWebinar.pdf">Steve Rennich and Nvidia</a></figcaption></figure><p>Our multiprocessing example failed because the two inference calls are not truly independent. If the vision model is accessing tensor cores, our control policy might have to wait in line. Does it mean this problem is impossible to solve?</p><p>Turns out we have one trick up our sleeve we can apply here: CUDA Streams. The CUDA Stream API allows us to create separate, isolated execution queues within a single process. This potentially enables the two streams to run concurrently [2]. But if we saw no improvement in the multiprocessing setup, how will CUDA streams help us here?</p><p>Firstly, multithreading avoids moving tensors between the CPU and GPU. This memory copying, illustrated in yellow in the above diagram, is costly and unnecessary. But more importantly, CUDA streams offer a key piece of information we can communicate to the GPU scheduler— <strong>stream priority</strong>.</p><p>The GPU scheduler is part of the GPU firmware that decides which instructions should execute. Within a single stream, work from different threads can’t be interleaved. With multiple independent streams, <strong>the scheduler is free to prioritise pending kernels from high-priority streams over low-priority streams</strong>. However, once a kernel starts executing, it runs to completion. Unlike CPU threads, GPUs offer no mechanism for interrupting a kernel.</p><p>Thus, we return to our threading-based example and <strong>create two CUDA streams with different priorities </strong>[3].</p><pre>import threading<br>import torch<br><br># Shared variables<br>latest_obs = None<br>latest_objects = None<br>obs_lock = threading.Lock()<br>objects_lock = threading.Lock()<br><br># Create streams with different priorities<br>policy_stream = torch.cuda.Stream(priority=-1)  # High priority (smaller integer indicates higher priority)<br>vision_stream = torch.cuda.Stream(priority=0)   # Low priority<br><br>def vision_thread():<br>    &quot;&quot;&quot;Background thread for object detection&quot;&quot;&quot;<br>    global latest_obs, latest_objects<br>    <br>    while True:<br>        # Get latest observation<br>        with obs_lock:<br>            obs = latest_obs<br>        <br>        if obs is not None:<br>            # Detect objects on low-priority CUDA stream<br>            with torch.cuda.stream(vision_stream):<br>                objects = vision_model.detect_objects(obs)<br>            <br>            # Update latest objects<br>            with objects_lock:<br>                latest_objects = objects<br><br>        # Sleep for a bit to avoid thread running constantly<br>        # Ensure the workload of this thread is complete before sleeping<br>        vision_stream.synchronize()<br>        time.sleep(0.01)<br><br># Start background thread<br>thread = threading.Thread(target=vision_thread, daemon=True)<br>thread.start()<br><br>while True:<br>    # Record start time<br>    start_loop_t = time.perf_counter()<br>    <br>    # Capture observations, inc. cameras<br>    obs = robot.capture_observation()<br>    <br>    # Share observation with vision thread<br>    with obs_lock:<br>        latest_obs = obs<br>    <br>    # Predict and send actions on high-priority CUDA stream<br>    with torch.cuda.stream(policy_stream):<br>        action = policy.select_action(obs)<br>    robot.send_action(action)<br>    <br>    # Get latest detected objects (may be from previous frame)<br>    with objects_lock:<br>        objects = latest_objects<br>    <br>    # Wait to match control frequency<br>    dt_s = time.perf_counter() - start_loop_t<br>    busy_wait(1 / fps - dt_s)</pre><p>Running this example solves our problem — the robot policy runs consistently at 20Hz, while our vision model is run at best-effort guarantees. There’s no free lunch here, we didn’t magically develop new tensor cores, all we did was ensure these cores are allocated to our control policy, occasionally delaying the vision model’s execution.</p><p>Note that since CUDA kernels cannot be interrupted mid-execution, models with long-running kernels may not allow sufficient the control policy’s high-priority workloads to be prioritised. If your model contains many small tensor operations, the scheduler will have ample opportunities for scheduling. You can also use a combination of stream.synchronise() and time.sleep(0.005) # or other small time period to force a stream to “take a time out”, allowing the GPU scheduler to schedule other kernels.</p><p>[1]: Operating-system level guarantees like <a href="https://ubuntu.com/real-time">Real-time Linux</a> are out of scope for this blog post. Even when running a single Python program controlling a robot via ACT, there are many reasons why the program might be interrupted in a traditional Linux system (such as other programs consuming CPU time, memory allocation slowness etc.) — RTLinux aims to solve many of them and is usually used in serious, enterprise-grade robotics. I have never designed systems like these and don’t know much about them.</p><p>[2]: To expand on the technicalities of this point: each process gets its own CUDA context, which includes a default CUDA stream. This enables their operations to run concurrently by default. The CUDA stream API allows us to approximate this behaviour within a single process.</p><p>[3]: We start two streams not only to define the priorities — only starting one stream for the priority work might not work because CUDA’s default stream has special rules where it <em>can</em> intervene with other streams (<a href="https://www.hpcadmintech.com/wp-content/uploads/2016/03/Carlo_Nardone_presentation.pdf">more information here</a>). I don’t know the exact details on how a default stream interferes with a non-default stream with lower priority, but it’s best to start two or more non-default streams and avoid any issues.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2c56b2d5988b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a simple out-of-distribution detector for Physical AI models using LeRobot]]></title>
            <link>https://villekuosmanen.medium.com/building-a-simple-out-of-distribution-detector-for-physical-ai-models-using-lerobot-bfa02b4a3876?source=rss-1a700a66e281------2</link>
            <guid isPermaLink="false">https://medium.com/p/bfa02b4a3876</guid>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[physical-ai]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[lerobot]]></category>
            <dc:creator><![CDATA[Ville Kuosmanen]]></dc:creator>
            <pubDate>Mon, 15 Sep 2025 14:48:40 GMT</pubDate>
            <atom:updated>2025-09-15T14:48:40.242Z</atom:updated>
            <content:encoded><![CDATA[<h4>In order to solve problems in our robots, we first have to detect them.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*UZUyFjumYD0nTZMGpjy-LQ.png" /></figure><p>You know the ChatGPT moment for robotics is coming, and have assembled your first robot arm and trained your first policies. Excited to try it out, you roll out the policy… and it doesn’t work.</p><iframe src="https://cdn.embedly.com/widgets/media.html?type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=twitter&amp;url=https%3A//x.com/VilleKuosmanen/status/1899032537898287445&amp;image=" width="500" height="281" frameborder="0" scrolling="no"><a href="https://medium.com/media/b31bdbcef8e6771c21badca8f4189eba/href">https://medium.com/media/b31bdbcef8e6771c21badca8f4189eba/href</a></iframe><p>There are many approaches towards improving the performance of end-to-end learned robot models — this blog post is not for them. Instead, I’d like to propose tackling a related, yet distinct, problem — <strong>can we detect when a robot is about to make a mistake, or fail in its task?</strong></p><p>Out of distribution (OOD) detection is a common approach for predicting failure conditions. AI models in general perform poorly outside their training distributions — robotics is no exception. New environments, lights, target objects, and even the positioning of objects on the screen can dramatically reduce the success rates of robot control policies, especially in limited data regimes. Detecting OOD conditions can help us stop the robot safely before anything bad happens, and potentially change from AI to human controls, helping us gather more human data from such conditions.</p><p>This blog post proposes a method that builds on my <a href="https://medium.com/@villekuosmanen/opening-the-black-box-of-robot-ai-983ab2754eec">previous work in mechanistic interpretability for robot AI models</a>, where the reconstruction loss of the sparse auto-encoder (SAE) is used as a useful proxy for determining whether given observations are similar to ones in the training set. This unsupervised method is simple and requires no data labelling.</p><h4>Mechanistic Interpretability and SAEs</h4><iframe src="https://cdn.embedly.com/widgets/media.html?type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=twitter&amp;url=https%3A//x.com/VilleKuosmanen/status/1935336920986333530&amp;image=" width="500" height="281" frameborder="0" scrolling="no"><a href="https://medium.com/media/51310572e06115a7f61ae20da8a5dc35/href">https://medium.com/media/51310572e06115a7f61ae20da8a5dc35/href</a></iframe><p>The key idea behind SAEs for mechanistic interpretability is to build a small, easy to interpret AI model to explain the attention patterns of a large, complex foundation model. I previously <a href="https://medium.com/@villekuosmanen/opening-the-black-box-of-robot-ai-983ab2754eec">built and trained such a model for the ACT policy architecture — with mixed results</a>.</p><p>The main purpose of a SAE is the activation of atomic features at test-time, which we can assign for arbitrary concepts. However, as the model is an auto-encoder, we have access to an additional output — the reconstruction loss. Sparse auto-encoders are trained with the unsupervised learning objective of reconstructing the original input from the bottleneck layer’s features (alongside the sparsity penalty and other regularisers). Thus <strong>the closer a given observation is to examples found in the training set, the better our model should be at reconstructing it</strong>.</p><p>Using a SAE trained on the attention patterns of the AI model has several useful properties. If our model natively generalises well to unseen environments for instance, the reconstruction loss should not increase as the model’s internal representations should represent the relevant parts of the changing environment in similar ways. This can be tested, by placing an unseen pineapple 🍍 to the back of the scene, versus to the front — in an accurate system, the latter should significantly increase the reconstruction loss, while the former should have little to no impact as objects in the background are not relevant when completing simple manipulation tasks.</p><iframe src="https://cdn.embedly.com/widgets/media.html?type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=twitter&amp;url=https%3A//x.com/VilleKuosmanen/status/1958159492329189503&amp;image=" width="500" height="281" frameborder="0" scrolling="no"><a href="https://medium.com/media/ef97046f2d0254ff594cddde2e3bcbdf/href">https://medium.com/media/ef97046f2d0254ff594cddde2e3bcbdf/href</a></iframe><h4>Step-by-step instructions</h4><ol><li>Follow the <a href="https://github.com/huggingface/lerobot">LeRobot getting started guide</a> to record <strong>three datasets</strong> — one larger training set, and a smaller validation set. Alternatively, you can choose to ring-fence a number of episodes from your training set for validation, if you only have a single dataset. This blog post assumes basic knowledge of the LeRobot project — if this step is challenging I recommend spending some time looking into how LeRobot and robot AI models work first.</li><li>Train a small version of an ACT model using LeRobot — to do this we need to set <em>dim_model=128</em> (default is 512) and increase training steps by ~50%. This policy will not perform as well as your regular robot control policy. It is not used for predicting actions, but purely for the out of distribution detection.</li><li>Create a Python virtual environment and clone and install the <a href="https://github.com/villekuosmanen/physical-AI-interpretability">Physical AI Interpretability toolkit</a>.</li></ol><pre>mamba create -n ood-test python=3.10 pip # you can also reuse an existing environment<br><br>git clone https://github.com/villekuosmanen/physical-AI-interpretability.git<br>pip install physical-ai-interpretability<br><br># Alternatively, you can install the package in editable mode if you&#39;d like to make local changes<br>cd physical-AI-interpretability<br>pip install -e .</pre><p>4. Run `python scripts/train_sae.py` to train a sparse autoencoder. You will need the following parameters, all others can use default values.</p><ul><li><em>— repo-id</em> — the <strong>train dataset</strong> for your task</li><li><em>— policy-path</em> path to the small <strong>ACT policy</strong> you trained on Step 2</li></ul><p>(note: Medium’s formatting automatically replaces dashes with em dashes — ensure you format the commands with two dashes rather than with an em dash)</p><p>Before training, the system will collect and cache model activations in your <em>.cache</em> folder — if something changes on the datasets you are using, you can delete these caches manually or use the <em>— force-cache-refresh</em> flag to invalidate the cache. You can also set <em>— activation-cache-path</em> to a suitable location if you’d prefer not using the default location in <em>.cache</em>.</p><p>5. You are now ready to calibrate and demo the out of distribution detector, using python <em>scripts/demo_ood_detector.py</em>. Again, we need a few parameters<br> 1. <em>— policy-path</em> — same as previously, path to the <strong>small ACT policy</strong> you trained on Step 2<br> 2.<em> — sae-experiment-path</em> — the path to the SAE model trained in Step 4, by default inside the <em>output</em> folder<br> 3. <em>— validation-repo-id</em>— the <strong>validation dataset</strong> for your task — do not reuse the training set as this will not give accurate results<br> 4. <em>— test-repo-id</em> — the <strong>test dataset</strong> to evaluate your OOD detector against. Ideally this is different from training or validation data, but if you only have 2 datasets you should use the validation dataset or an episode in it here.</p><p>This script first calibrates the threshold for reconstruction loss that defines whether a sample is considered in or out of distribution based on the validation dataset and the <em>— std-threshold</em> flag (default=2.5). Setting this to a higher value will make the model more discerning, reducing false positives while increasing false negatives. Note that the use of standard deviations assumes a Gaussian distribution of reconstruction loss values, which is probably not accurate — future work should explore alternative calibration methods. Calibration results are saved automatically so you should only need to complete it once.</p><p>After calibration, the OOD Detector is ready to be used, and has a very simple API.</p><pre>is_ood, recon_error = ood_detector.is_out_of_distribution(observation)</pre><p>The test script runs all samples in the test dataset through the OOD Detector, and reports some basic metrics.</p><p>For more information on usage, <a href="https://villekuosmanen.github.io/physical-AI-interpretability/">check out the docs page</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bfa02b4a3876" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Opening the black box of robot AI]]></title>
            <link>https://villekuosmanen.medium.com/opening-the-black-box-of-robot-ai-983ab2754eec?source=rss-1a700a66e281------2</link>
            <guid isPermaLink="false">https://medium.com/p/983ab2754eec</guid>
            <category><![CDATA[ai-safety]]></category>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[anthropic-claude]]></category>
            <category><![CDATA[humanoid-robot]]></category>
            <dc:creator><![CDATA[Ville Kuosmanen]]></dc:creator>
            <pubDate>Mon, 23 Jun 2025 16:00:45 GMT</pubDate>
            <atom:updated>2025-06-23T16:00:45.384Z</atom:updated>
            <content:encoded><![CDATA[<h4>Mechanistic interpretability and attention maps offer solutions for understanding transformer-based Physical AI models</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*cpqO-9CHiPekUBx2-EARRQ.gif" /></figure><p>In a few years, helpful service robots will be available for purchase in your local store. Will you accept a robot in your home if you don’t understand how it behaves? Will you trust your cat, your children, your elderly parents or grandparents, to walk next to it?</p><p>Physical AI is fundamentally different from AI models living inside digital devices. While language models can cause harm through biased or hateful outputs, or when given agency to make decisions independently from humans, cars and robots controlled by Physical AI can directly cause serious injuries or death with a single mistake.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FNRY1lvH4izRbUsi.jpg" /><figcaption>Unitree, a humanoid robot manufacturer, is confident in showing their robots interacting with people. But the powerful electric motors and unpredictable AI models controlling the robot raise questions over safety. Image source: <a href="https://www.chinadailyhk.com/hk/article/584358">China Daily Hong Kong</a></figcaption></figure><p><strong>In Physical AI, black boxes are no longer enough</strong>. We won’t accept robots in our society if we can’t understand how they work. This article introduces two recent projects I’ve worked on to make transformer-based AI robots interpretable for humans: attention mapping and feature extraction through mechanistic interpretability. Let’s go!</p><h3>Attention Maps</h3><p>Do AI robots see the world like we do?</p><p>Transformer-based AI models use attention mechanisms to compute how different parts of the input relate to and influence each other. Visualising regions of high and low attention can therefore help us understand what parts of the input the AI model uses in predicting actions.</p><p>Since most inputs to the robot AI model we are looking at (called <a href="https://arxiv.org/abs/2304.13705">ACT</a>) originate from images taken via the robot’s cameras, we can map the attention scores measured at the last layer of the transformer encoder¹ back into the pixels of the camera images the tokens originated from. We can visualise this by overlaying the scores on the image as a heatmap to produce the below visualisation, while also normalising the scores per time step to make the values more meaningful. The attention mapper and visualiser can be used with any pre-trained ACT policy, and only requires minimal changes to existing code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*cpqO-9CHiPekUBx2-EARRQ.gif" /></figure><p>The visualiser shows that the attention maps are not random, and indeed correspond roughly with the points of interest humans would pay attention to. We can also observe that the model learns to use data from both images, changing its attention between one or the other as is necessary.</p><p>The attention maps are not always what you expect, though. In my task, the model seems to mostly focus on the gripper when approaching the coffee capsules to pick.</p><p>Attention maps can also be used to gauge overfitting to joint states. My robot models are trained with a variety of object positions — a model trained mostly with data from a single pick position tends to constantly attend to the joint state (also known as proprioceptive) token.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AtdJOF-TBQZcGPxTL3CBgg.gif" /><figcaption>Original robot dataset and AI model is from <a href="https://github.com/imstevenpmwork">Steven Palma</a> (Hugging Face)</figcaption></figure><p>Transformer models are also complex — while the ACT encoder used only has 4 layers, that’s still enough to learn complex, higher level representations for the task that don’t necessarily relate to the visual features in the original images. This kind of reasoning happens in the “latent space” we are mapping back into the images, so areas lighting up doesn’t necessarily mean that the robot is looking at that region, but rather that region might be used for a higher-level feature like “we are holding the coffee capsule but not aligned with the coffee cup yet”.</p><h3>Feature extraction</h3><p>We now have an idea of how AI robots see the world. But do they learn to represent specific scenarios or concepts in their brain? In the language model work, such feature-centric interpretability work has been led by Anthropic through their work on <a href="https://transformer-circuits.pub/2023/monosemantic-features">dictionary learning with sparse autoencoders</a> (SAEs), culminating in the release of <a href="https://www.anthropic.com/news/golden-gate-claude">Golden Gate Claude</a>, a language model obsessed with the San Francisco bridge achieved by amplifying the Golden Gate Bridge Feature. I highly recommend reading through these blog posts before diving into the following section — the authors explain the theory and method much better than I ever could!</p><p><em>this part gets more technical, non-engineers you have been warned…</em></p><p>I applied Anthropic’s approach to robot AI models by first training a smaller version of my coffee capsule picking ACT model². Small models contain fewer unique internal states than large ones, thus making them ideal for interpretability work, at the cost of real-world task success rates of course. Secondly, we train a SAE to reconstruct the internal attention patterns of the ACT model, targeting the 3rd (out of 4) encoder layers on the ACT transformer encoder. We choose an internal layer in order to capture internal, mid-level feature representations. Besides reconstruction loss, the SAE is trained with L1 regularisation on activated features, encouraging sparsity.</p><p>Robot and language models mainly differ in the number of tokens analysed at each time step. Language models can be interpret separately for each token, while in robot models all tokens should be interpret as one time step. This creates a problem — my simple ACT model contains 602 tokens with 128 dimensions each, equalling to <em>602 * 128 = 77,056</em> independent dimensions. If we used Anthropic’s suggested 8x expansion factor in the SAE, we would end up with over half a million features!</p><p>I propose solving this problem by introducing sampling or block averaging for the tokens originating from image encoders. Sampling 1 out of 8 tokens loses some information but maintains specific, “spiky” tokens, while averaging the values of each 8 token block pushes them towards the mean in all dimensions. Using one of these approaches leaves us with 77 tokens (the VAE and joint state token are not compressed), and <em>128 * 77 = 9,856</em> total dimensions. By using “just” 1.25x expansion factor, the SAE can be trained using an RTX 4090 consumer GPU³!</p><p>Training the SAE gives us ~12k independent features. To know the meaning of each feature, we must visualise frames that most activate them. Ideally this would be done during inference, but robots being physical machines makes this inconvenient. We can work around this by running our ACT model on existing, offline data, and recording top activating frames for each feature. We can record other stats too like mean activation, percentiles, and variance of activation. We can visualise the results in a <a href="https://huggingface.co/spaces/villekuosmanen/act-feature-visualiser">Hugging Face space I created</a>.</p><p>Are the features interpretable? Kind of — below are examples I discovered that activate for particular scenarios somewhat consistently.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g45U57tJZO0fol_D5BO8Ng.png" /><figcaption>This feature activates when the gripper is close to the coffee capsule, but seemingly does not activate during the grasp itself</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VlNh6PqV5nptetrkgiWPNg.png" /><figcaption>This feature activates on the Living Room Chair recording environment.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VE8nWM3ToqW6T84jsTJrdA.png" /><figcaption>This feature activates during the first frames of an episode when the coffee capsule is placed in the far left of the front camera. It seems to tell the robot to seek the capsule from the left side of the manipulation environment.</figcaption></figure><p>It’s interesting how these features seem to capture details of the environment, the location of the target objects in it, as well as the specific motions or types of grasps or motions required. Increasing the expansion factor from 1.25x to 8x would likely reveal even more granular features. I am especially interested if such features just “collapse” to individual episodes, datasets, or sections in them — doing so would be evidence for the memorisation hypothesis recently proposed <a href="https://arxiv.org/abs/2505.05787">in this paper</a>.</p><p>While we can extract features from the robot AI model, borrowing a method directly from language models is likely not optimal. I’d love to see new research ideas specifically for extracting features off of multi-token visual components, which would likely work much better for both robot AI models and VLMs alike.</p><h3>Future work</h3><p>This interpretability work only covers ACT models, which in the robotics context are small single-task models. A far more interesting prospect is studying large robotics foundation models, such as vision-language-action models like pi0, GR00T N1, or SmolVLA. These models are similar to VLMs, so many learnings from VLM interpretability would likely apply to them as well. The deeper a neural network gets, the harder it is to interpret — hence why starting with ACT models is a good idea. Work on identifying and labelling features automatically would be valuable as well — I can’t possibly go through 10,000 features by hand!</p><p>While the attention maps and interpretable features are interesting by themselves, they can also potentially be used for downstream applications. In language models, interpretable features have been used for <em>steering</em> a model’s output away from dangerous concepts and towards safe, desirable features (or Golden Gate Bridges…). Feature activations can also be used to detect potential abuse or undesirable model behaviour. Such applications could have many benefits for robotics as well, and take us closer to safe, friendly service robots in our homes!</p><p>Code to replicate all experiments and visualisations mentioned in this blog post is open source, and can be accessed in <a href="https://github.com/villekuosmanen/physical-AI-interpretability">https://github.com/villekuosmanen/physical-AI-interpretability</a>. Happy coding!</p><h4>Footnotes</h4><p>¹ Because each token attends to all other tokens, we need to take an average of all these activations to calculate a single score for each token.<br>² Smaller in this context means the internal states of the transformer model have fewer dimensions. We reduce the <em>dim_model</em> hyperparameter from the default of 512 to 128 — a 4x reduction of internal parameters.<br>³ The SAE model training itself “only” requres ~12GB VRAM. The remaining 10–12GB is used by the Adam optimiser. I tried using VRAM-saving optimisers like SGD without success — Adam seems to be required for this task.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=983ab2754eec" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Manifesto for Open-Source Physical AI]]></title>
            <link>https://villekuosmanen.medium.com/a-manifesto-for-open-source-physical-ai-94b0cb38f5d8?source=rss-1a700a66e281------2</link>
            <guid isPermaLink="false">https://medium.com/p/94b0cb38f5d8</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[humanoid-robot]]></category>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Ville Kuosmanen]]></dc:creator>
            <pubDate>Mon, 02 Jun 2025 11:46:39 GMT</pubDate>
            <atom:updated>2025-06-02T16:51:59.581Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to build the open-source intelligence powering millions of robots</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SEib4EKdIssZulYUeikcCQ.jpeg" /></figure><h4>Open source is an on-ramp</h4><p>Robotics is hard to get started with. Working effectively with robots requires expertise in many skills, from mechanical and electric engineering to software, controls, and AI. I know this first hand — I pivoted into the field ~2 years ago from a pure software background. The robots themselves are expensive as well — the barrier of entry into buying a physical robot has traditionally been high and inaccessible for most people.</p><p>As a result, software and AI has seen much progress over the past decade while robots and the physical world in general has lagged behind.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*t6AvvqeqU8FtsLtuZcyj3A.jpeg" /></figure><p><strong>Open source offers an on-ramp for millions of software engineers into robotics</strong>. Open-access journals and <a href="https://arxiv.org/">arXiv</a> pre-prints give anyone with an internet connection access to cutting-edge AI research, while podcasts and commentary on Twitter and Bluesky gets you involved in casual conversations between researchers you’d only hear on conference after-hours. Simulators let you control robots without having to own the physical hardware yourself. Work by <a href="https://github.com/AlexanderKoch-Koch/low_cost_robot">Alexander Koch</a>, <a href="https://github.com/TheRobotStudio/SO-ARM100">Rob Knight</a>, <a href="https://github.com/jess-moss/moss-robot-arms">Jess Moss</a>, and many others has given rise to new low-cost robots with bills-of-material well below the cost of an iPhone.</p><p>Open source helped me get started. After reading many robotics papers, I got my hands dirty by building on Meta’s <a href="https://github.com/facebookresearch/home-robot">Home Robot</a> project based on the <a href="https://github.com/facebookresearch/habitat-sim">Habitat simulator</a>, project at the time lead by the great <a href="https://cpaxton.github.io/about/">Chris Paxton</a>. <strong>Without open source, I wouldn’t be working in the field</strong>.</p><h4>A monopoly on intelligent robots is forming</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PgPv-Vn1IABcDhx6LjC4RQ.png" /></figure><p>AI is a centralising technology — building cutting-edge models requires clean internet-scale datasets and data centres worth of compute to train. Peter Thiel has gone as far as saying <a href="https://www.axios.com/2018/02/13/peter-thiel-ai-is-communist-1518541570">“Crypto Is libertarian, AI Is communist”</a>, though in today’s world, cutting-edge AI looks to be controlled more by giant corporations than authoritarian governments.</p><p>These dynamics apply even more to robotics — at least in language and image modelling, anyone can access an internet’s worth of pre-training data — data we don’t have for robots. Scarcity of public data concentrates power further into the hands of giant companies who can afford to pay an army of human data collectors to control and “teach” their robots.</p><p>At the same time, the impact of intelligent robots would be much greater than language models. On the internet, “competition is just a click away”. There’s no clicking away from a humanoid robot cleaning your house or office. Unless something changes, the vast majority of the world’s service robots will be directly controlled by a small number of powerful men in charge of the AI supermajors. What could go wrong with that, right??</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*QV6cuauhM0gR8bO1wBzD0g.jpeg" /><figcaption>Generated using Grok — <em>you dare use my spells against me, Potter</em>?</figcaption></figure><h4>Open source — the People’s AI</h4><p>Open source presents an alternative — the <em>only</em> alternative — for the previous scenario. <a href="https://ai.meta.com/blog/llama-4-multimodal-intelligence/">Llama</a> and <a href="https://github.com/deepseek-ai/DeepSeek-R1">DeepSeek</a> LLMs have proven open-source AI can compete with commercial alternatives in raw intelligence. More importantly, these models have created an ecosystem of hackers, researchers, and startups building new models and products on top of them.</p><p>In robotics, work in open-source is lead by Hugging Face’s <a href="https://github.com/huggingface/lerobot">LeRobot team</a>. In a year, they have built a community with 14k GitHub stars, 7.5k members on Discord, and thousands of new public robotics datasets uploaded to the Hugging Face Hub. Other companies are helping too — Physical Intelligence open-sourced their powerful <a href="https://github.com/Physical-Intelligence/openpi">Pi0 foundation model</a>, with NVIDIA (<a href="https://github.com/NVIDIA/Isaac-GR00T">Gr00t N1</a>) and Meta (<a href="https://github.com/facebookresearch/sam2">SAM-2</a> and many others) among other companies contributing to open source. But above all, open source is advanced by the thousands of community members around the world testing, remixing, building and sharing their work on top of these projects.</p><p>The power of the open-source community should not be underestimated. <strong>If the LeRobot community was a company, it’s 7.5k “employees” would make it the largest AI lab in the world</strong>, edging out DeepMind and OpenAI at ~5k and ~3.5k employees, respectively. There are likely 100s of open-source AI researchers and hackers for every employee working in closed-source AI labs.</p><p>I’ve seen the community grow first-hand. I joined the LeRobot Discord in the first few days of it being opened to public in Summer 2024, and since then have had the pleasure of collaborating with the Hugging Face team on several initiatives — from helping teams as a <a href="https://www.linkedin.com/posts/ville-kuosmanen-891889154_i-joined-friends-at-hugging-face-for-a-activity-7316736706202791936-eGjZ?utm_source=social_share_send&amp;utm_medium=member_desktop_web&amp;rcm=ACoAACUmKa0B3zFGtiS0LMTwR4lsSCQTCveauoc">Hugging Face mentor in the Paris hackathon in April</a> to contributing to the <a href="https://huggingface.co/blog/lerobot-datasets">curation of the world’s largest crowd-sourced robotics dataset</a>. Beyond, I’ve contributed data cleaning and curation tools with <a href="https://github.com/villekuosmanen/pikodata">Pikodata</a>, and tools to visualise the internal attention states of robot AI models through <a href="https://github.com/villekuosmanen/physical-AI-attention-mapper">Physical AI Attention Mapper</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*cpqO-9CHiPekUBx2-EARRQ.gif" /><figcaption>Ever wondered how AI robots see the world? With <a href="https://github.com/villekuosmanen/physical-AI-attention-mapper">Physical AI Attention Mapper</a>, you can find it out!</figcaption></figure><p>Not only can open source win in Physical AI, it <em>has</em> to win. A world of extreme power concentration in robotics into the hands of the few is fundamentally undemocratic and far too dangerous to let happen. Open source doesn’t have to mean charity either — companies like Hugging Face, DeepSeek, MongoDB, and Docker have all built successful businesses on open-source foundations. An open source Physical AI ecosystem offers an alternative to commercial models, and allows thousands of robotics startups around the world to compete on equal footing with Goliaths hundreds of times their size. Is this not a world we should all be fighting for?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*ZOVJ22ElgU-i-oyb_xIeJA.png" /><figcaption>Buy this beautiful frag from the <a href="https://levelsio.com/products/come-and-take-it-flag">levelsio store</a>, we just need a robot variant of it!</figcaption></figure><p>If you are as passionate about open-source Physical AI as I am, let’s talk! I’m building something big in this space and am looking for co-founders and partners to share the journey with. Let’s build the future of open-source Physical AI together!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/0*C3-pkm4b4haQTHpf" /><figcaption>“words importing the masculine gender shall include all other genders”</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=94b0cb38f5d8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Aldix and the Book of Life]]></title>
            <link>https://villekuosmanen.medium.com/aldix-and-the-book-of-life-5da3f0001b6c?source=rss-1a700a66e281------2</link>
            <guid isPermaLink="false">https://medium.com/p/5da3f0001b6c</guid>
            <category><![CDATA[aging]]></category>
            <category><![CDATA[rationalism]]></category>
            <category><![CDATA[biotechnology]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Ville Kuosmanen]]></dc:creator>
            <pubDate>Sat, 30 Dec 2023 11:43:49 GMT</pubDate>
            <atom:updated>2023-12-30T11:43:49.561Z</atom:updated>
            <content:encoded><![CDATA[<h4>A fable on the dangers of measures becoming targets</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ohJI8TuASEtKkTSE8kyQlg.png" /></figure><p>When on a crisp spring night you look at a night sky, you can see countless stars, each with their own planetary systems. But there are also planets without suns, forever bound to travel across space in darkness. What if such a planet developed intelligent life? Let’s imagine so, for it may help us find answers to an important question.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4YLYxM2LJYt1Sp_8O3GkIw.png" /><figcaption>Rogue planet Tempus Vagaris as seen by a distant observer.</figcaption></figure><p>You have now left Earth, and landed on Tempus Vagaris, a planet without a sun. But don’t worry, it won’t be all cold and dark — thermal vents from the planet’s molten core have created ancient hot springs. These pockets of warmth feed an ecosystem of most wondrous creatures. There are giant mushrooms and fungi with an eerie blue glow, providing light for small flowers, grasses, and bushes. The air buzzes with fireflies of different colours, creating an ambient twinkling of lights like sparks, and the grasses are home to small frogs catching flies that stray too close to the ground. Within the spring waters swim small fish, eating the spores and algae growing at the surface. Beyond a few meters, the waters get too murky and hot to explore, and the creatures hiding in the depts remain a mystery.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vo0VKxbjzroo0-nq5lScYA.png" /><figcaption>The hot springs of Tempus Vagaris are full of life.</figcaption></figure><p>But the most special creatures on Tempus Vagaris are the Aevosaurs, a race of small dinosaur-like creatures living inside the fungi. The Aevosaurs have small hands, two feet they use for walking, and a long tail with a bioluminescent bulb they use for communicating their feelings. Unlike other inhabitants of Tempus Vagaris, the intelligent Aevosaurs can use tools and live in small villages where they study and catalogue the wondrous world around them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HBaWn4QkmMbM0ZlLzcbt9g.png" /><figcaption>An Aevosaur walking in the mushroom forests.</figcaption></figure><p>The centre of the Aevosaur village is the tallest mushroom of the forest, which over generations has been hollowed out with a staircase that reaches all the way to the top. The cap offers a beautiful vista across the hot springs, while also acting as the an observatory for the eternal night sky. The Aevosaurs are master astronomers, and have named every constellation of stars in the sky. This allowed them to understand a fundamental truth about their home world.</p><p>It rotates.</p><p>Unlike our Earth, Tempus Vagaris has no days and nights, no months, and no years either. Stories tell of a past where Aevosaurs slept whenever they tired. But a growing civilisation needs a schedule to function, and thus a Day was defined as a full rotation of the stars in the sky. Lamps made of glowing fungi were placed across the village to distinguish the day from the night. The future could now be planned in advance: when a field was planted with spores, a harvest of fully grown mushrooms could be expected in 40 days. The passage of time was tracked by marking a small cross, first on wood or stone and later on papyrus made of reeds, another famous invention of the Aevosaurs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NH240zqnLuUHtKQkFbFyvA.png" /><figcaption>An aevosaur checking on the progress of the upcoming harvest.</figcaption></figure><h4>The Book of Life</h4><p>While the Day was useful for tracking harvests, the Aevosaurs started using it to track their own lifespans as well. Aevosaurs did not know how old or young they were: some were taller than others, or had wrinkles, scars, and stories reminiscent of a life long lived. But just as a harvest could be tracked by making a cross on papyrus, so could the age of an Aevosaur. And as legend tells, the Elders of the tribe so created the Book of Life: a personal book of papyrus scrolls, where each Aevosaur was to mark a cross every morning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ohJI8TuASEtKkTSE8kyQlg.png" /><figcaption>A young Aevosaur crossing off another day on a hand-bound Book of Life.</figcaption></figure><p>Days on Tempus Vagaris were long: after a shift in the fields, an Aevosaur deserved their rest. A single scroll in the Book of Life could accommodate around 60 standard marks. Thus it became convenient to count an Aevosaur’s age using the number of scrolls they had filled up in their Book. A newly hatched Aevosaur’s Book would be marked by their parents, until they had filled six full scrolls, after which the young one would join the village school and be taught how to write their own marks. Since progress at school was tied to scrolls, some mischievous Aevosaurs tried to fill theirs with marks larger than usual. They had to tread carefully: transgressions against the sacred practice would not be tolerated by the Council of Elders.</p><p>At fourteen scrolls, an Aevosaur could take the Challenges, a final exam of sorts, and enter the society as an adult. No more homework or being bossed around by Mum and Dad! After that, the daily marking became seen as a chore, and filling up a scroll could trigger a small age crisis. The 30th and 40th scrolls were seen as especially difficult, while the few Aevosaurs making it to their 50th scroll were revered for their wisdom and could join the Council of Elders to lead the tribe.</p><h4>Life of Aldix</h4><p>Out of all Aevosaurs, Aldix was born different. In school, he was always looking to pass his assignments with the least effort possible. Aldix was reprimanded multiple times by the Elders for marks too large on his Book of Life, until he had perfected handwriting that was just small enough to pass the scrutiny of his teachers. At fourteen pages, Aldix was the shortest Aevosaur to take on the Challenges, which he passed by the skin of his teeth.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U5NyPytrBEbmuA1lm6I4wA.png" /><figcaption>Schoolteachers would often reprimand Aldix for not filling his Book of Life correctly.</figcaption></figure><p>Aldix seemed much happier as a grown-up Aevosaur. He started working as a farmer, and even got married. But as his scrolls started to fill up, a new form of anxiety started to creep up: Aldix was getting older and approaching 30 scrolls of age.</p><p>Aldix started filling his scrolls with smaller marks. Gone were the large marks of his youth: the new ones were distinctively smaller, though not so small as to stick out. Aldix would also “forget” to fill out a mark every now and then. This did have an impact: other Aevosaurs started noticing that his wife Vitella was aging faster than him. But Aldix’s 30th scrolldate came and went, and his anxiety only became stronger.</p><p>Finally Aldix had an idea. Other Aevosaurs were mostly paying attention to the current position of marks in the scroll. If he started filling out multiple scrolls in parallel, he could show progress each day, while discreetly ripping out incomplete scrolls from the Book. As others started asking questions about his slow aging, Aldix claimed it was due to the foul-tasting elixir he made from fermenting seaweed growing at the bottom of the springs. Few believed the eccentric Aevosaur’s words as no one else had ever dived deeper than a few meters, yet his slow aging did become a hot topic. As Vitella turned 50 and joined the Council of Elders, Aldix was still at his 37th scroll. At this rate he could be overtaken by his own son and daughter!</p><p>Because of his young age, it was a shock for everyone when Aldix passed away at the age of 38 scrolls. The loss of such a young life touched everyone, though wise Vitella, now 55 and one of the village’s oldest had caught on to Aldix’s tactics. At her 56th scrollday, she asked the village to move away from measuring age and experience in scrolls, as <a href="https://en.wikipedia.org/wiki/Epigenetic_clock">Epigenetic clocks</a> like the Book of Life can be useful measures yet will always break when the measure becomes a target.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*236f3QXAmZZ407VwzmV4Lg.png" /><figcaption>Vitella was the wisest Aevosaur, as she understood the dangers of measures becoming targets.</figcaption></figure><p><em>The tale of Aldix is part 2 of my series of two fables. You can check out part 1, the story of Googlopolis, </em><a href="https://villekuosmanen.medium.com/the-fall-of-googlopolis-76140339c3fc"><em>here</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5da3f0001b6c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The fall of Googlopolis]]></title>
            <link>https://villekuosmanen.medium.com/the-fall-of-googlopolis-76140339c3fc?source=rss-1a700a66e281------2</link>
            <guid isPermaLink="false">https://medium.com/p/76140339c3fc</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[rationalism]]></category>
            <category><![CDATA[search-engine-optimizati]]></category>
            <category><![CDATA[google]]></category>
            <dc:creator><![CDATA[Ville Kuosmanen]]></dc:creator>
            <pubDate>Sun, 24 Dec 2023 12:18:34 GMT</pubDate>
            <atom:updated>2023-12-30T11:45:26.155Z</atom:updated>
            <content:encoded><![CDATA[<h4>A fable on the dangers of measures becoming targets</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F_ukw-pWepZc7MP8885wFA.png" /></figure><p>Once upon a time, there was a market town known for the quality of its tailoring named Googlopolis. Sitting on a calm bay at the mouth of a river, cloth and textiles from all corners of the world passed through the town’s harbour. Cotton, cashmere, silk, each dyed with more exotic colours.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OFsqw_T5QJ1VMAGCXAg0ug.png" /><figcaption>The harbour of Googlopolis</figcaption></figure><p>Besides the variety of textiles, Googlopolis also had many tailors, dressmakers, and haberdashers crafting them into clothing of all kinds. Citizens of the town knew who to go to for with a particular request, and what the fair price for each type of service was. Life was good.</p><p>As the kingdom grew prosperous, the reputation of Googlopolis attracted shoppers far and wide. But with so many textiles and merchants to choose from, the visitors had problem: they did not know who to shop with and what a fair price for each product would be. Most customers had never seen fabrics as fine and colours as vibrant as the ones sold here: how could they known which ones are cheap and which ones are worth paying for? So daunting was the paralysis of choice that many went home empty handed, choosing to do business with their local village tailor and their fabrics made of wool instead.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZRXmEzr9-OlGHGCXS1PkJg.png" /><figcaption>Shoppers from every corner of the kingdom liked to visit the tailors of Googlopolis.</figcaption></figure><p>The tailors’ guild needed a solution, as the problems of their customers were taking a toll on the reputation of the whole town. One day, a royal scribe named Larry had an idea. You see, tailors already had certain informal ways to signal the quality of their products. The most prestigious tailors would use thread made of gold to monogram the customer’s initials inside men’s jackets. Middle-market tailors used silk instead of gold, while the most humble haberdashers skipped monogramming altogether. Top tailors also hired their apprentices amongst graduates of the prestigious Royal College located in the kingdom’s capital city, whereas smaller shops tended to go for self-trained sewers and cutters from the town’s youth. Larry proposed a method for rating each tailor based on these attributes, knowing that they signalled the quality, and price, of their work. All members of the guild rejoiced, and a board displaying the rank of each tailor was placed at the centre of the market square in front of Googlopolis Cathedral.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F_ukw-pWepZc7MP8885wFA.png" /><figcaption>A ranking of every tailor in town was displayed prominently in front of the Googlopolis Cathedral.</figcaption></figure><p>Now the visiting shoppers knew exactly which tailors’ prices were worth paying for, and which ones were trying to rip them off. Life was good. The reputation of Googlopolis grew with every wedding dress and suit brought to the countryside.</p><p>However, not all locals were happy with the rankings. Many found their trusted tailor ranked towards the bottom of the list. “<em>Have they been ripping me off during all these years?</em>” Customers thus started to move towards the few shops at the top of the list. Their masters became wealthy as they could raise prices and hire more staff, while the families at the bottom had to let go their apprentices and seek extra jobs to put food on the table.</p><p>The struggling tailors at the bottom of the rankings had to do something to change the situation. While the exact method of assessing tailors was a tightly guarded secret, they started to understand how it worked. Only superficial markers of quality mattered: you could use less expensive materials to make a dress, as long as it was dyed with vibrant colours. You could also spend less time measuring each customer and making a pattern for them, as fit quality, a point of pride for all tailors, was not assessed at all.</p><p>Tailors also kept adding cosmetic details like monogramming with gold, and hiring their apprentices exclusively from the Royal College over hard-working self-trained ones. As it became harder to work your way into the tailoring profession without a degree, the cost of studying in the College increased until only children from rich families could afford to enrol. This was partly by design, as it allowed Googlopolis’s richest shopkeepers to only hire their friends’ children without any pushback! The guarantee of a well-paying job also made the students lazy — why work hard to learn the craft of tailoring if you can get a job anyway?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iZm0A95su83BrYF8tMqzBA.png" /><figcaption>Students of the Royal College were known for working hard for their degrees (as pictured).</figcaption></figure><p>Thus while the ratings of shopkeepers kept improving, the locals started to notice the declining quality of garments. The foreign traders also stopped importing exotic textiles as nobody wanted to spend money on them. They went to the nearby Great Penn Town instead, where local tailors were ecstatic to finally be able to buy such rare fabrics. As word of Googlopolis’s declining quality spread, shoppers from the kingdom started to visit Great Penn Town instead. At the same time, talented and opportunistic youth from Googlopolis chose to seek apprenticeships in Great Penn Town’s shops, whose masters did not rate expensive qualifications over talent, work ethic, and eagerness to learn on the job. The few remaining master tailors focusing on quality in Googlopolis also moved to Great Penn Town, being accepted by the locals with open arms for their knowledge in the trade.</p><p>A few decades on, the reputation of Great Penn Town as the new centre of tailoring had spread far across the kingdom. Few visitors bothered to visit the tailors of Googlopolis, who could not understand the decline of their town: when every tailor had a perfect rating and every apprentice had a college degree, why would anyone want to shop anywhere else? For everyone else in the kingdom, the fate of Googlopolis was a lesson on what can happen when <a href="https://en.wikipedia.org/wiki/Goodhart&#39;s_law">Goodhart’s law</a><strong> </strong>is broken and simple measures become targets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TszO-ny2Gy75JBPgXE3KHA.png" /><figcaption>“We have a five star rating, yet are going bankrupt. How?”</figcaption></figure><p><em>The tale of Googlopolis was part 1 of a series of two fables. Check out Part 2 and the story of Aldix and the Book of Life </em><a href="https://villekuosmanen.medium.com/aldix-and-the-book-of-life-5da3f0001b6c"><em>here</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=76140339c3fc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[It needs to get worse before it gets better]]></title>
            <link>https://villekuosmanen.medium.com/it-needs-to-get-worse-before-it-gets-better-10bbf5ce36cc?source=rss-1a700a66e281------2</link>
            <guid isPermaLink="false">https://medium.com/p/10bbf5ce36cc</guid>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[digital-transformation]]></category>
            <dc:creator><![CDATA[Ville Kuosmanen]]></dc:creator>
            <pubDate>Sat, 28 Oct 2023 20:17:25 GMT</pubDate>
            <atom:updated>2023-10-28T20:17:25.670Z</atom:updated>
            <content:encoded><![CDATA[<h4>One way to approach the benefits and costs of changing your ways of working.</h4><figure><img alt="Cartoon illustration set in Victorian London at dusk. A man in a suit, top hat, beard, and monocle, and a woman in a long lace dress, both read by candlelight. Beside them, an engineer struggles with a burned-out prototype of an electric light, a hole burned in his glove. The gentleman remarks to the lady, ‘I knew it, these electric lightbulbs would never take off!’" src="https://cdn-images-1.medium.com/max/1024/1*bMNcBnuVyZ1KEj2VvrP5vA.png" /></figure><p>Almost all changes require a period of lower productivity before the new, more advanced workflow becomes more efficient. This is because The Old Ways have been optimised to a local maximum — while the new workflow has a higher potential, it starts out unoptimised. New ways of doing things also require a period of learning. Before reaching competence, you will have go through a period where every basic action requires thinking or searching through the manual. Us humans are biased towards short-term thinking, so such inefficiency feels like a waste of time, leading to a a painful transition period before the efficiencies start bearing dividends.</p><figure><img alt="I could do my job so much faster with my old spreadsheet than with the new cloud-based system IT installed" src="https://cdn-images-1.medium.com/max/640/1*Zun0gkkART07jJaYT-8eiw.png" /><figcaption><em>Illustration generated using matplotlib and the </em><a href="https://github.com/ipython/xkcd-font"><em>xkcd-script font</em></a><em>.</em></figcaption></figure><p>Is it always worth upgrading to a new system? After all, you wouldn’t want to be a Luddite holding back the engines of progress? Before you jump all in abroad the Current Thing™ hype train, do consider a few more arguments.</p><p>Firstly, just because a product is new or more advanced than an old one doesn’t mean it’s automatically better. For example, traditional paper-based books still have many advantages over e-books: they won’t run out of battery, can be read in bright sunlight better than backlit screens, are less likely to be stolen, won’t expire due to format changes or the sales platform going out of business, and can easily be skimmed through.</p><p>Secondly, even if a new product or system is better than the existing one, one must consider the opportunity cost in changing to it. The decline in productivity until it gets better should be estimated and evaluated against other investments. If the changing costs are high, you should probably stick to your old system. This is why core business software such as CRMs, databases, and programming languages are considered “sticky”, and their annual recurring revenues (ARRs) are valued highly by investors. <em>Once it’s in, it’s never coming out</em>.</p><figure><img alt="Cartoon illustration in Victorian London style. The bearded man with a top hat and monocle from the previous image sits in a candlelit drawing room, reading a newspaper. The news article shows a zig-zagging line taking a steep dive, symbolizing a falling share price. He looks shocked with sweat dripping from his forehead. The room has subtle hints of his failing investments, indicating he’s losing money." src="https://cdn-images-1.medium.com/max/1024/1*k8HsazYg3uYKGIuIN6bDUw.png" /><figcaption><em>”What do you mean my Metaverse NFT land parcel is worthless? It’s Artificially Scarce™! Even Snoop Dogg promoted it!”</em></figcaption></figure><h4>On AI investment</h4><p>This post began its life as a commentary on the recent advances in AI. AI-powered tools have the potential to disrupt workflows and processes across industries. While some products like Google Docs promise to mix AI seamlessly into existing tools, many others will require large changes to existing workflows. Not every company will choose to adopt them. It will be too expensive and difficult to do so. “<em>Blah blah regulation blah intellectual property blah blah legal uncertainty blah copyright blah blah GDPR blah blah safety”</em>.</p><p><strong>To adopt AI in your organisation, your workflow needs to get worse before it gets better</strong>.</p><p>Many fear AGI will take over jobs from humans. The more likely scenario in the short term is that the jobs will be taken by users of AI. Generative AI tools can make a junior employee perform at a level of a senior, by reducing the time taken to learn internal processes and master technical skills, while senior employees can work as a small team with AI automating the most menial parts of a job. Would you like to double your organisation’s output?</p><p>This time, take the time to learn the Current Thing™ and embrace disruption. It will get so much better.</p><p>But first, it must get worse.</p><p><em>Images generated by Dall-E 3 and GPT-4. All writing is my own.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=10bbf5ce36cc" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>