<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Siamak Yousefi  on Medium]]></title>
        <description><![CDATA[Stories by Siamak Yousefi  on Medium]]></description>
        <link>https://medium.com/@siamak.yousefi.1984_58741?source=rss-024a4ac34f32------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*JwGUPV8qu9c1gpQ6KtKpBw.jpeg</url>
            <title>Stories by Siamak Yousefi  on Medium</title>
            <link>https://medium.com/@siamak.yousefi.1984_58741?source=rss-024a4ac34f32------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 18:52:09 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@siamak.yousefi.1984_58741/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Overview of Reinforcement Learning Methods for Enhancing VLA Robot Training]]></title>
            <link>https://medium.com/@siamak.yousefi.1984_58741/overview-of-reinforcement-learning-methods-for-enhancing-vla-robot-training-e33ef1f3d34e?source=rss-024a4ac34f32------2</link>
            <guid isPermaLink="false">https://medium.com/p/e33ef1f3d34e</guid>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[flowmatching]]></category>
            <category><![CDATA[robot-training]]></category>
            <category><![CDATA[vla]]></category>
            <category><![CDATA[diffusion-models]]></category>
            <dc:creator><![CDATA[Siamak Yousefi ]]></dc:creator>
            <pubDate>Sun, 24 May 2026 21:35:33 GMT</pubDate>
            <atom:updated>2026-05-24T21:35:33.761Z</atom:updated>
            <content:encoded><![CDATA[<p>The Vision Language Action (VLA) Models have gained tremendous attention in the past couple of years, thanks to the great progress made in the field of large language model (LLM) and vision language model (VLM). Among the very first VLA models are OpenVLA <strong>[1]</strong> proposed by Stanford researchers. The generative model used by OpenVLA is a diffusion based model that is slightly unstable and also slow during inference. Physical Intelligence addressed these limitations with <strong>π0</strong> [2] and <strong>π0.5 </strong>[3] models, which combine VLM backbones with flow matching for continuous action generation — which has shown amazing performance if the model went through a phase of supervised fine-tuning on the robot trajectory data in the environment under deployment.</p><p>The problem formulation of SFT loss is shown below where the network tries to minimize the distance between ground truth action chunks of length H and the generated actions chunks from the diffusion policy for K_i observed trajectories:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-vH5sggR71pivbu43TbZ7w.png" /></figure><p>If you are interested you can check my previous blog on <strong>π0</strong> and <strong>π0.5</strong> on medium.com here:</p><p><a href="https://medium.com/@siamak.yousefi.1984_58741/robotics-foundation-models-by-physical-intelligence-openpi-663dc55aa9d3">Robotics Foundation Models by Physical Intelligence: OpenPi</a></p><p>The SFT seems a high burden in new environments and without that the robot seems to fail miserably when doing new tasks on new objects and in new environments that was not used during fine-tune step. The robot somehow only memorized the actions during SFT and doesn’t learn them as preferred when tested in new environments and on new objects! Therefore, there has been some efforts to enhance the performance of VLA models using reinforcement learning (RL) which is the traditional approach to robot training. In the following I describe a few recent papers and research works in this domain [4–6].</p><h3>RL-based VLA Training</h3><h3>1- RL-VLA :</h3><p>One of the recent works in this domain, which is published in Neurips 2025 [4], is a PPO-based model applied to OpenVLA foundation model. It uses an actor critic architecture where the Value is predicted using an MLP network applied on the last transformer block of OpenVLA before generating action tokens [4].</p><p>The PPO is among the policy gradient methods composed of two neural networks; actor and critic. The actor network predicts the actions based on the recent policy. Then the critic tries to predict the state values. A reward is assigned to the actions taken and then an advantage value is calculated which shows how much the return is better than state value estimated by critic network. During train the critic tries to minimize the loss between true return values and it’s prediction. The critic is not independent than actor as it’s prediction of value affects the advantage.</p><p>In RL-VLA [4], The actor is the openVLA robot policy that is being fine-tuned using LoRA. To save VRAM and speed up training, the actor (policy) and critic (value function) share the same Transformer backbone. In summary the RL-VLA algorithm is trained as follows:</p><p>The robot uses its current policy to rollout an action and the log probability, entropy and rewards are returned.</p><p>The relative return and generalized advantage estimation (GAE) which is the relative improvement of an action compared to average of other actions are estimated through the trajectory for each time-step t as:</p><pre>for t in reversed(0 to T-1):<br>    # TD error<br>    δ_t = r_t + γ × V(s_{t+1}) × mask_{t+1} - V(s_t)<br>    <br>    # GAE accumulation<br>    GAE_t = δ_t + γ × λ × mask_{t+1} × GAE_{t+1}<br>    <br>    # Returns (for value function target)<br>    Returns_t = GAE_t + V(s_t)<br><br>Advantages = Returns - V(s_T) <br><br># Normalize advantages<br>Advantages_normalized = (Advantage - mean(Advantages)) / (std(Advantages) + 1e-5)</pre><p>The advantage is then calculated as the difference between the returns and the last value estimate. The advantage is an indicator for policy; if positive the action is considered a good move, else if negative the action was bad and will be penalized. Based on the normalized estimated advantage of an action, A, and the log probability, the surrogate policy loss is calculated.</p><p>Loss_policy= -min(r·A, clip(r)·A)</p><p>where r = π_new(a|s) / π_old(a|s) is the ratio of new policy to old policy and clip keeps the parameter within [1-ε, 1+ε]. The negative of loss is minimized which is equivalent to maximizing the reward policy of actions, as the actor tries to maximize the policy and take actions that have high advantage.</p><p>The value is predicted by critic network (MLP) from the hidden layer output of openVLA right before action tokens. The critic tries to minimize the distance between the estimated return coming from the MLP head and the return calculated from rewards. The value loss is calculated using a clipped huber loss, which is a robust L1-L2 trade off as</p><p>L_value: Huber( V_clipped — Returns_t)</p><p>where V_clipped = V_old + clamp(V — V_old, -ε, ε) and clamp is clipping the value between [-ε, ε].</p><p>Finally, the loss is formed as the sum of policy, value and an optional entropy loss for exploration. The architecture of the RL-VLA is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/613/1*sDrXg25iTdzwpmpyx08m9Q.png" /><figcaption>RL-VLA model architecture applied to OpenVLA</figcaption></figure><p>The findings show that the RL-based training can gain significantly on execution of tasks compared to SFT. RL-trained models can recover from mid-episode object repositions and handle unseen robot initial poses. While SFT agents often “march on” despite missing a grasp, RL agents learned to retry and adjust their end-effector poses. On semantic grounding where robot is supposed to generalize to unseen objects and novel instruction phrasing, there was moderate gains. On the vision side the RL didn’t provide much advantage over SFT.</p><h3>2-π_RL [6]</h3><p>This work is from some of the authors of RL_VLA so it is an improvement by exploiting <em>π</em>0.5​ instead of OpenVLA as SOTA and applying RL. Since flow matching output tokens are deterministic and not suited for RL exploration during rollout, the framework introduces two solutions to characterize the logarithmic likelihood of executed actions:</p><ul><li><strong>Flow-Noise:</strong> This approach integrates a <strong>learnable noise network</strong> into the denoising process. By modeling the denoising stage as a <strong>discrete-time Markov Decision Process (MDP)</strong>, the system can directly compute the exact log-likelihood of the de-noised action sequence for policy optimization.</li><li><strong>Flow-SDE:</strong> This method is based on the idea of flow-GRPO that uses RL for enhancement of flow matching, and it converts the deterministic Ordinary Differential Equation (ODE) process of flow matching into a <strong>Stochastic Differential Equation (SDE)</strong>. It creates a <strong>two-layer MDP</strong> that couples the internal denoising steps with the external agent-environment interaction, allowing for efficient exploration while maintaining equivalent marginal distributions.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-Ts_oKxGVW7_Tnb4teSZ3g.png" /></figure><p>The <em>π</em>_RL has these steps:</p><ul><li><strong>Selective Fine-tuning:</strong> To ensure GPU memory efficiency, the large <strong>VLM backbone is frozen</strong>, and RL fine-tuning is applied exclusively to the <strong>action expert</strong>.</li><li><strong>Critic Design:</strong> In the <em>π</em>0.5​ variant, the <strong>critic network</strong> (responsible for value prediction) is attached directly to the <strong>VLM output</strong>. This is because <em>π</em>0.5​ merges proprioceptive state information with prompt embeddings within the VLM, rather than feeding it separately into the action expert.</li><li><strong>Policy Optimization:</strong> The framework uses the <strong>Proximal Policy Optimization (PPO)</strong> algorithm to optimize the policy parameters. For <em>π</em>0.5​, a <strong>learning rate scheduler with cosine annealing</strong> is often employed to stabilize training and prevent the escalation of KL divergence.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*l1GdPaHFkrMbzYOqfXsYBw.png" /></figure><p>The GAE is estimated similar to RL-VLA.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-1ab3iRQ8DxHwdkyq7lrqw.png" /></figure><p>Experimental results show that applying <em>πRL</em>​ to <em>π</em>0.5​ yields substantial improvements over supervised fine-tuning (SFT) baselines:</p><ul><li><strong>Success Rates:</strong> In the LIBERO benchmark, <em>πRL</em>​ achieved a <strong>98.3% success rate</strong>, surpassing the full-dataset SFT baseline of 96.9%.</li><li><strong>Generalization:</strong> The framework effectively enhances the model’s ability to handle <strong>out-of-distribution (OOD)</strong> visual and execution variations, suggesting the acquisition of generalized action representations rather than narrow overfitting.</li><li><strong>Efficiency:</strong> It significantly improves <strong>temporal efficiency</strong>, with episode lengths converging to expert-level ranges after RL training.</li></ul><h3><em>3- π</em>∗0.6:</h3><p>The Physical Intelligence group introduced π0 and π0.5 and later an enhancement called π0.6 [4]. An RL-based enhancement to it called π*0.6 is released also named <strong>RECAP</strong> (<strong>R</strong>einforcement learning with <strong>E</strong>xperience and <strong>C</strong>orrections via <strong>A</strong>dvantage-<strong>C</strong>onditioned <strong>P</strong>olicies), a general-purpose recipe for RL training of large-scale VLAs. The idea of <em>π</em>∗0.6 is based on <em>π</em>0.6 by conditioning the model on a binarized advantage indicator (I_t), which tells the model if the action the robot is taking is optimal or not, in the sense of task completion.</p><p>Step I: data collection with human correction:</p><p>The human in the loop starts introducing randomness in the scene by moving objects while the robot is doing actions. This helps increase robustness of the robot to failures and learning to take steps to get rewarded.</p><p>Step II: Value function training:</p><p>To act as a reliable critic, <em>π</em>∗0.6 uses a <strong>distributional value function</strong> <em>pϕ</em>​(<em>V</em>∣<em>o_t</em>​,ℓ) that maps observations and language commands to a distribution over <em>B</em> discretized value bins. Let R_t(<a href="https://en.wikipedia.org/wiki/%CE%A4">τ</a>) be the empirical return of a trajectory <a href="https://en.wikipedia.org/wiki/%CE%A4">τ</a> from step t until T. The empirical return is discretized into B=201 bins, and then cross-entropy loss <em>H</em> is minimized over empirical returns <em>R_t</em>​(<a href="https://en.wikipedia.org/wiki/%CE%A4">τ</a>) as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/465/1*3a7mY12kQkdrCHORPXWGMg.png" /></figure><p>The value function is then estimated from the learned value distribution</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*cicwR2dgYMOr1R-3DdpE7w.png" /></figure><p>where v(b) is the value for bin b. This value function identifies mistakes on-the-fly, allowing the system to estimate the “advantage” of any given action — essentially asking, “Is this action better than what robot usually does?”</p><p>Step III: Advantage conditioning training:</p><p>Once the value function is available, it should be used to train an improved policy, which is known as policy extraction method. While there are some well-known RL methods, the authors claim that those are apparently hard to be applied to flow matching models. The authors proposed an advantage conditioning where an additional input indicating how optimal the action might be, is introduced as indicator:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/1*tfMzTAj2Xty6z8koaW3xBw.png" /></figure><p>where the indicator <em>I(.)</em> is modeled as a delta distribution where it is 1 when the advantage is greater than threshold <em>ϵ_ℓ </em>and 0 otherwise. Therefore, the policy objective is to minimize</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*XzhWV4fo3kWhUNq4mY0PsQ.png" /></figure><p>The advantage value <em>A(o_t, a_t, l) </em>are obtained from the value function calculated earlier.</p><p>Step IV : Continuous control with flow matching</p><p>Because <em>π</em>∗0.6 must generate smooth, continuous actions for dexterous tasks like tamping espresso or folding laundry, it integrates the flow-matching loss into its RL objective. The model optimizes a lower bound on the action likelihood:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*EKQ8NP1McpFA44mI3qv_8g.png" /></figure><p>Here, <em>f_θ</em>​ represents the continuous output of the action expert, and <em>ω</em> is sampled noise. This formulation allows the VLA to maintain the benefits of expressive action distributions while benefiting from RL-based improvement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*-60RHqEOa68hOCfeQ8yIKA.png" /><figcaption><em>The π</em>*06 diagram of training architecture.</figcaption></figure><p>By deploying <em>π</em>∗0.6 in a continuous loop — collecting autonomous rollouts, receiving human corrections, and retraining — the model achieves performance levels far beyond standard supervised learning in terms of throughput gain, failure reduction and robustness for long period of time.</p><p>In Conclusion, <em>π</em>∗0.6 demonstrates that RL for VLAs doesn’t have to be unstable or complex. By treating RL as a conditioning problem and leveraging the power of flow matching, we can move from generalists that <em>can</em> do a task to experts that do it <em>well</em>.</p><p>The only limitation is that the robot needs to be trained with a human supervision for many examples so it can master the tasks.</p><h3>Conclusion</h3><p>The use of RL as en enhancement to the VLA is a promising trend in recent years and seems to be a preferred approach as compared to pure SFT. Since applying RL to VLA is not as straightforward as conventional robot learning, the papers discussed have applied novel RL ideas to the VLAs. While these approaches seem to improve the results compared to SFT, still there is room for developing more optimal RL formulation that can generalize to unseen environments and tasks.</p><h3>References</h3><p>1- <a href="https://arxiv.org/abs/2406.09246">https://arxiv.org/abs/2406.09246</a></p><p><a href="https://www.pi.website/blog/pi0">2-https://www.pi.website/blog/pi0</a></p><p>3-<a href="https://www.pi.website/blog/pi05">https://www.pi.website/blog/pi05</a></p><p>4-<a href="https://arxiv.org/pdf/2505.19789">What Can RL Bring to VLA Generalization? An Empirical Study</a></p><p>5-<a href="https://www.pi.website/download/pistar06.pdf">https://www.pi.website/download/pistar06.pdf</a></p><p>6-<a href="https://arxiv.org/pdf/2510.25889">https://arxiv.org/pdf/2510.25889</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e33ef1f3d34e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Robotics Foundation Models by Physical Intelligence: OpenPi]]></title>
            <link>https://medium.com/@siamak.yousefi.1984_58741/robotics-foundation-models-by-physical-intelligence-openpi-663dc55aa9d3?source=rss-024a4ac34f32------2</link>
            <guid isPermaLink="false">https://medium.com/p/663dc55aa9d3</guid>
            <dc:creator><![CDATA[Siamak Yousefi ]]></dc:creator>
            <pubDate>Sat, 29 Nov 2025 15:15:32 GMT</pubDate>
            <atom:updated>2025-11-29T15:15:32.924Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>The robotics field has experienced remarkable progress over the past few years, thanks to the advances in foundation models for text and image generation — namely Large Language Models (LLMs) and Vision-Language Models (VLMs). Traditional approaches to robot training relied on mathematical formulation of rules and explicit programming, which proved inaccurate for the complexity demanded by humanoid robots and manipulators in home and industrial environments.</p><p><strong>Vision-Language-Action (VLA) models</strong>: a paradigm that embeds multimodal inputs — visual observations from cameras, natural language commands, and proprioceptive states — into a unified representation, then generates continuous robot actions over time. This approach enables robots to perform complex, multi-stage tasks by leveraging the semantic reasoning capabilities inherited from internet-scale pre-training.</p><p>Among the pioneers in this space, <strong>OpenVLA [1] </strong>from Stanford introduced a 7-billion parameter VLA fine-tuned for robot control. While groundbreaking, OpenVLA’s autoregressive discrete action representation struggled with fast robot tasks. The generative model used by OpenVLA is a diffusion based model that is also slow during inference. Physical Intelligence addressed these limitations with their <strong>π0</strong> [2] and <strong>π0.5 </strong>[3]<strong> </strong>models, which combine VLM backbones with flow matching for continuous action generation — achieving state-of-the-art performance on complex manipulation tasks.</p><p>The mind blowing fact about these models is that they are generalizable to any robot. Only a set of ground truth data is needed for a new robot in order to do supervised fine tuning (SFT) using low-rank adaptation (LORA) or other techniques. Afterwards, the robot is ready for deployment in any environment.</p><p>In this article I provide a technical comparison of π0 and π0.5, examining their architectures, training recipes, and the design choices that enable broad generalization.</p><h3>The VLA Paradigm</h3><p>The core objective of VLA models is to learn the conditional action distribution given observation, i.e., p(A_t | o_t), where:</p><p><strong>A_t = [a_t, a_{t+1}, …, a_{t+H-1}]</strong> represents an <em>action chunk</em> — a sequence of H future actions (typically H = 50),</p><p><strong>o_t = [I_t¹, …, I_t^n, ℓ_t, q_t]</strong> is the observation consisting of:</p><ul><li>Multiple RGB images I_t^i from robot cameras (2–3 views)</li><li>Language command ℓ_t as tokenized text</li><li>Proprioceptive state q_t including joint positions, velocities, and gripper state</li></ul><p>The images I_t^i and state q_t are encoded via corresponding encoders and then projected via a linear projection layer into the same embedding space as the language tokens.</p><h3>Physical Intelligence Foundation Model: <em>π</em>0</h3><p>Physical intelligence started developing models that are foundational for VLA similar to the way LLMs are for text. Among the recent works published, the one called <em>π</em>0 and an upgraded one <em>π</em>0.5 provide a general architecture that can transfer the embedding of the prompt and the observation from cameras to a set of continuous robot actions.</p><p>The<em> π</em>0​ model utilizes a multi-stage training recipe, mirroring those used for large language models (LLMs): a pre-training phase followed by a post-training phase. In pertaining phase, it uses a massive, diverse, cross-embodiment dataset consisting of approximately 10,000 hours of robot data collected across 7 different robot configurations and 68 tasks, supplemented by open-source datasets like OXE. Post training phase fine-tunes the model using smaller, high-quality, curated data to specialize it for skillful and fluent execution of specific, complex downstream tasks, such as laundry folding, table cleaning, and assembling boxes.</p><p>For vision language understanding a Paligemma transformer model is used. There are 3 cameras; base, left wrist and right wrist at 224*224 resolution. The state of the robot consist of robot joint positions, velocities, and gripper state.</p><p>Unlike the openVLA which uses diffusion models for the action training, <em>π</em>0 and <em>π</em>0.5 use flow matching algorithm. Flow matching is a generative model based on ordinary differential equation (ODE), while diffusion score matching is based on stochastic differential equation (SDE).</p><p>A good blog and video explaining the two models can be found here:</p><p><a href="https://harshm121.medium.com/flow-matching-vs-diffusion-79578a16c510">https://harshm121.medium.com/flow-matching-vs-diffusion-79578a16c510</a></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FfirXjwZ_6KI%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DfirXjwZ_6KI&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FfirXjwZ_6KI%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/dbefb079e6141d8580f7f1db6410c8b2/href">https://medium.com/media/dbefb079e6141d8580f7f1db6410c8b2/href</a></iframe><p>Flow matching has some advantages for robotics applications such as:</p><ol><li>Smooth action generation: No discrete steps that cause jerky motions. It is suited for continuous movements.</li><li>Efficient sampling: Requires fewer integration steps than discrete diffusion. Faster inference for real-time robot control.</li><li>Stable training: More stable gradients than adversarial methods.</li><li>Multimodal conditioning: Natural integration of vision, language, and proprioception.</li></ol><h3>Training Networks</h3><p>For each action in the action chunk A_t, a corresponding action token is fed through the action expert. In practice, the flow matching network is trained by sampling random noise ϵ ∼ N(0, I), computing the “noisy actions” at step τ ∈ [0, 1] is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/494/1*j5mgqgzWa75mmL0kgI8tSg.png" /></figure><p>This can also be formulated as A_t^τ = ϵ — τ(ϵ — A_t), where the noisy action consists of noise minus the rate of change of noise (ϵ − A_t) times time-step τ, hence the term (ϵ − A_t) is also called the target velocity:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/494/1*khw0iwJbfCZl7UrdzwiaGg.png" /></figure><p>During training, these action tokens are supervised using a conditional flow matching loss. The loss function of the action model is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*-CqhtrCnx4iJnp5P1p2z-w.png" /></figure><p>where subscripts denote robot time-steps and superscripts denote flow matching time-steps, with τ ∈ [0, 1] and where q(A_t^τ |A_t) = N (τA_t,(1 − τ )I). In practice, the network is trained by sampling random noise ϵ ∼ N(0, I), computing the “noisy actions”. During training, the network iteratively learns to output v_θ(A_t^τ , o_t) such that it is closest to the target velocity vector field u(A_t^τ |A_t).</p><p>The action expert transformer model uses a full bidirectional attention mask, so that all action tokens attend to each other. During training, the flow matching time-step τ is sampled from a beta distribution that emphasizes lower (noisier) time-steps. The attention mechanism is a bit complicated! The model is using 3 block-wise causal attention for these three blocks [I_t¹ , …, I_t^n , ℓ_t], [q_t], and [a_t^τ , …, a_{t+H−1}^τ ]. Within each block there is a full bidirectional attention whereas the tokens in each block cannot attend to the next block. The first one which is the input modalities from VLM, attends to itself and avoids attending to future blocks to minimize distribution shift. The state q_t uses causal attention so it doesn’t attend to future blocks. The final blocks is the noisy actions that also uses causal attention that can attend to the full input sequence.</p><p>The π0 employs a two-stage training procedure mirroring LLM practices:</p><p><strong>Pre-training Phase</strong>:</p><ul><li>~10,000 hours of robot data across 7 robot configurations and 68 tasks</li><li>Supplemented with open-source datasets (OXE, Bridge v2, DROID)</li><li>Diverse language labels combining task names and fine-grained segment annotations</li></ul><p><strong>Post-training Phase</strong>:</p><ul><li>High-quality curated data for specific downstream tasks</li><li>Teaches fluent, efficient execution strategies</li><li>Enables recovery from mistakes while maintaining skillful behavior</li></ul><h3>Inference Pipeline</h3><p>During inference, the actions are generated by integrating the learned vector field from τ=0 to τ=1 with 10 fractional steps, starting from standard normal action vector A_t⁰ ∼ N (0, I) and evolving as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/1*Rmb__YB_kIdWilT4f-XVMA.png" /></figure><p>where δ=0.1 is the integration step size. Note that given the same initial noise, the action is deterministic and hence trajectory is smoother as compared to diffusion score matching. Note that observation o_t = [I_t¹ , …, It^n , ℓ_t, q_t] and the noisy actions A_t^τ are used in the vector field to be integrated to yield the next flow matching step A_t^{τ+ δ}. Each time a new action chunk A_t is predicted, each of the images I_t¹ , …, I_t^n are encoded and run a forward pass on the tokens corresponding to o_t. Then it runs 10 steps of flow matching, where each step requires running a forward pass on the tokens corresponding to A_t^τ (the keys and values corresponding to o_t are cached). So in summary</p><ol><li>Encode all camera images I_t^{1}, …, I_t^{n}.</li><li>Run forward pass on observation tokens o_t (cache keys and values)</li><li>Execute 10 flow matching integration steps, recomputing only action tokens per step</li><li>Output action chunk; execute 16–25 actions before re-inference</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qoqACW9ga5u9BqmOvictpQ.png" /><figcaption>Architecture of Pi0 as illustrated in <a href="https://www.physicalintelligence.company/download/pi0.pdf">https://www.physicalintelligence.company/download/pi0.pdf</a></figcaption></figure><h3>Upgraded Model Pi0.5</h3><p>The improved model is called π0.5 which is based on π0 but with some subtle architecture changes: For train, it is using a hierarchical architecture where during pre-training the discrete action tokens are predicted via FAST tokenizer. Post training using flow matching is used in the next step for continuous actions. For inference, it is using a two stage inference procedure where first a high level textual subtask is predicted first and then followed by a low-level action prediction. In this upgrade, the state input is discretized and included in language tokens.</p><p>The VLM backbone is a Paligemma that takes in a sequence of images [I_t¹ , . . . , I_t^N ] and a language prompt ℓ as in π0, but also the robot’s proprioceptive state q_t in tokenized form, and finally the tokenized actions, which will be auto-regressively predicted. The action expert is a smaller transformer that takes in a sequence of noisy action tokens a_{t:t+H}^{τ} for an action horizon of 50, i.e. H = 49, and is trained with the flow matching approach. Unlike π0 that fuses the flow-matching time-step τ with the noisy action before feeding into the transformer, π0.5 uses a separate MLP for projecting τ only and then applies adaptive RMSNorm to inject the timestep information to each layer of the action expert.</p><p>It sounds a bit complicated but apparently this architecture can boost accuracy!</p><h4>PI0 Training Characteristics</h4><ul><li><strong>State Processing</strong>: Separate continuous state embedding</li><li><strong>Time Conditioning</strong>: Concatenation + MLP processing</li><li><strong>Architecture</strong>: More parameters due to additional projections</li><li><strong>Training</strong>: Standard layer normalization</li></ul><h4>PI05 Training Improvements</h4><ul><li><strong>State Processing</strong>: Discretized state tokens in language sequence</li><li><strong>Time Conditioning</strong>: AdaRMSNorm for parameter modulation</li><li><strong>Architecture</strong>: More efficient parameter usage</li><li><strong>Training</strong>: Adaptive normalization improves flow matching convergence</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hw7qQyeGsBMGLBjnGwlqXA.png" /><figcaption>The architecture of pi05</figcaption></figure><p>Below you can see the source code of the Pytorch model used for both models:</p><pre><br>class PI0Pytorch(nn.Module):<br>    def __init__(self, config):<br>        super().__init__()<br>        self.config = config<br>        self.pi05 = config.pi05<br><br>        paligemma_config = _gemma.get_config(config.paligemma_variant)<br>        action_expert_config = _gemma.get_config(config.action_expert_variant)<br><br>        self.paligemma_with_expert = PaliGemmaWithExpertModel(<br>            paligemma_config,<br>            action_expert_config,<br>            use_adarms=[False, True] if self.pi05 else [False, False],<br>            precision=config.dtype,<br>        )<br><br>        self.action_in_proj = nn.Linear(32, action_expert_config.width)<br>        self.action_out_proj = nn.Linear(action_expert_config.width, 32)<br><br>        if self.pi05:<br>            self.time_mlp_in = nn.Linear(action_expert_config.width, action_expert_config.width)<br>            self.time_mlp_out = nn.Linear(action_expert_config.width, action_expert_config.width)<br>        else:<br>            self.state_proj = nn.Linear(32, action_expert_config.width)<br>            self.action_time_mlp_in = nn.Linear(2 * action_expert_config.width, action_expert_config.width)<br>            self.action_time_mlp_out = nn.Linear(action_expert_config.width, action_expert_config.width)<br><br>        torch.set_float32_matmul_precision(&quot;high&quot;)<br>        self.sample_actions = torch.compile(self.sample_actions, mode=&quot;max-autotune&quot;)<br><br>        # Initialize gradient checkpointing flag<br>        self.gradient_checkpointing_enabled = False<br><br>        msg = &quot;transformers_replace is not installed correctly. Please install it with `uv pip install transformers==4.53.2` and `cp -r ./src/openpi/models_pytorch/transformers_replace/* .venv/lib/python3.11/site-packages/transformers/`.&quot;<br>        try:<br>            from transformers.models.siglip import check<br><br>            if not check.check_whether_transformers_replace_is_installed_correctly():<br>                raise ValueError(msg)<br>        except ImportError:<br>            raise ValueError(msg) from None<br><br>    def gradient_checkpointing_enable(self):<br>        &quot;&quot;&quot;Enable gradient checkpointing for memory optimization.&quot;&quot;&quot;<br>        self.gradient_checkpointing_enabled = True<br>        self.paligemma_with_expert.paligemma.language_model.gradient_checkpointing = True<br>        self.paligemma_with_expert.paligemma.vision_tower.gradient_checkpointing = True<br>        self.paligemma_with_expert.gemma_expert.model.gradient_checkpointing = True<br><br>        logging.info(&quot;Enabled gradient checkpointing for PI0Pytorch model&quot;)<br><br>    def gradient_checkpointing_disable(self):<br>        &quot;&quot;&quot;Disable gradient checkpointing.&quot;&quot;&quot;<br>        self.gradient_checkpointing_enabled = False<br>        self.paligemma_with_expert.paligemma.language_model.gradient_checkpointing = False<br>        self.paligemma_with_expert.paligemma.vision_tower.gradient_checkpointing = False<br>        self.paligemma_with_expert.gemma_expert.model.gradient_checkpointing = False<br><br>        logging.info(&quot;Disabled gradient checkpointing for PI0Pytorch model&quot;)<br><br>    def is_gradient_checkpointing_enabled(self):<br>        &quot;&quot;&quot;Check if gradient checkpointing is enabled.&quot;&quot;&quot;<br>        return self.gradient_checkpointing_enabled<br><br>    def _apply_checkpoint(self, func, *args, **kwargs):<br>        &quot;&quot;&quot;Helper method to apply gradient checkpointing if enabled.&quot;&quot;&quot;<br>        if self.gradient_checkpointing_enabled and self.training:<br>            return torch.utils.checkpoint.checkpoint(<br>                func, *args, use_reentrant=False, preserve_rng_state=False, **kwargs<br>            )<br>        return func(*args, **kwargs)<br><br>    def _prepare_attention_masks_4d(self, att_2d_masks):<br>        &quot;&quot;&quot;Helper method to prepare 4D attention masks for transformer.&quot;&quot;&quot;<br>        att_2d_masks_4d = att_2d_masks[:, None, :, :]<br>        return torch.where(att_2d_masks_4d, 0.0, -2.3819763e38)<br><br>    def _preprocess_observation(self, observation, *, train=True):<br>        &quot;&quot;&quot;Helper method to preprocess observation.&quot;&quot;&quot;<br>        observation = _preprocessing.preprocess_observation_pytorch(observation, train=train)<br>        return (<br>            list(observation.images.values()),<br>            list(observation.image_masks.values()),<br>            observation.tokenized_prompt,<br>            observation.tokenized_prompt_mask,<br>            observation.state,<br>        )<br><br>    def sample_noise(self, shape, device):<br>        return torch.normal(<br>            mean=0.0,<br>            std=1.0,<br>            size=shape,<br>            dtype=torch.float32,<br>            device=device,<br>        )<br><br>    def sample_time(self, bsize, device):<br>        time_beta = sample_beta(1.5, 1.0, bsize, device)<br>        time = time_beta * 0.999 + 0.001<br>        return time.to(dtype=torch.float32, device=device)<br><br>    def embed_prefix(<br>        self, images, img_masks, lang_tokens, lang_masks<br>    ) -&gt; tuple[torch.Tensor, torch.Tensor, torch.Tensor]:<br>        &quot;&quot;&quot;Embed images with SigLIP and language tokens with embedding layer to prepare<br>        for PaliGemma transformer processing.<br>        &quot;&quot;&quot;<br>        embs = []<br>        pad_masks = []<br>        att_masks = []<br><br>        # Process images<br>        for img, img_mask in zip(images, img_masks, strict=True):<br><br>            def image_embed_func(img):<br>                return self.paligemma_with_expert.embed_image(img)<br><br>            img_emb = self._apply_checkpoint(image_embed_func, img)<br><br>            bsize, num_img_embs = img_emb.shape[:2]<br><br>            embs.append(img_emb)<br>            pad_masks.append(img_mask[:, None].expand(bsize, num_img_embs))<br><br>            # Create attention masks so that image tokens attend to each other<br>            att_masks += [0] * num_img_embs<br><br>        # Process language tokens<br>        def lang_embed_func(lang_tokens):<br>            lang_emb = self.paligemma_with_expert.embed_language_tokens(lang_tokens)<br>            lang_emb_dim = lang_emb.shape[-1]<br>            return lang_emb * math.sqrt(lang_emb_dim)<br><br>        lang_emb = self._apply_checkpoint(lang_embed_func, lang_tokens)<br><br>        embs.append(lang_emb)<br>        pad_masks.append(lang_masks)<br><br>        # full attention between image and language inputs<br>        num_lang_embs = lang_emb.shape[1]<br>        att_masks += [0] * num_lang_embs<br><br>        embs = torch.cat(embs, dim=1)<br>        pad_masks = torch.cat(pad_masks, dim=1)<br>        att_masks = torch.tensor(att_masks, dtype=torch.bool, device=pad_masks.device)<br><br>        # Get batch size from the first dimension of the concatenated tensors<br>        bsize = pad_masks.shape[0]<br>        att_masks = att_masks[None, :].expand(bsize, len(att_masks))<br><br>        return embs, pad_masks, att_masks<br><br>    def embed_suffix(self, state, noisy_actions, timestep):<br>        &quot;&quot;&quot;Embed state, noisy_actions, timestep to prepare for Expert Gemma processing.&quot;&quot;&quot;<br>        embs = []<br>        pad_masks = []<br>        att_masks = []<br><br>        if not self.pi05:<br>            if self.state_proj.weight.dtype == torch.float32:<br>                state = state.to(torch.float32)<br><br>            # Embed state<br>            def state_proj_func(state):<br>                return self.state_proj(state)<br><br>            state_emb = self._apply_checkpoint(state_proj_func, state)<br><br>            embs.append(state_emb[:, None, :])<br>            bsize = state_emb.shape[0]<br>            device = state_emb.device<br><br>            state_mask = torch.ones(bsize, 1, dtype=torch.bool, device=device)<br>            pad_masks.append(state_mask)<br><br>            # Set attention masks so that image and language inputs do not attend to state or actions<br>            att_masks += [1]<br><br>        # Embed timestep using sine-cosine positional encoding with sensitivity in the range [0, 1]<br>        time_emb = create_sinusoidal_pos_embedding(<br>            timestep, self.action_in_proj.out_features, min_period=4e-3, max_period=4.0, device=timestep.device<br>        )<br>        time_emb = time_emb.type(dtype=timestep.dtype)<br><br>        # Fuse timestep + action information using an MLP<br>        def action_proj_func(noisy_actions):<br>            return self.action_in_proj(noisy_actions)<br><br>        action_emb = self._apply_checkpoint(action_proj_func, noisy_actions)<br><br>        if not self.pi05:<br>            time_emb = time_emb[:, None, :].expand_as(action_emb)<br>            action_time_emb = torch.cat([action_emb, time_emb], dim=2)<br><br>            # Apply MLP layers<br>            def mlp_func(action_time_emb):<br>                x = self.action_time_mlp_in(action_time_emb)<br>                x = F.silu(x)  # swish == silu<br>                return self.action_time_mlp_out(x)<br><br>            action_time_emb = self._apply_checkpoint(mlp_func, action_time_emb)<br>            adarms_cond = None<br>        else:<br>            # time MLP (for adaRMS)<br>            def time_mlp_func(time_emb):<br>                x = self.time_mlp_in(time_emb)<br>                x = F.silu(x)  # swish == silu<br>                x = self.time_mlp_out(x)<br>                return F.silu(x)<br><br>            time_emb = self._apply_checkpoint(time_mlp_func, time_emb)<br>            action_time_emb = action_emb<br>            adarms_cond = time_emb<br><br>        # Add to input tokens<br>        embs.append(action_time_emb)<br><br>        bsize, action_time_dim = action_time_emb.shape[:2]<br>        action_time_mask = torch.ones(bsize, action_time_dim, dtype=torch.bool, device=timestep.device)<br>        pad_masks.append(action_time_mask)<br><br>        # Set attention masks so that image, language and state inputs do not attend to action tokens<br>        att_masks += [1] + ([0] * (self.config.action_horizon - 1))<br><br>        embs = torch.cat(embs, dim=1)<br>        pad_masks = torch.cat(pad_masks, dim=1)<br>        att_masks = torch.tensor(att_masks, dtype=embs.dtype, device=embs.device)<br>        att_masks = att_masks[None, :].expand(bsize, len(att_masks))<br><br>        return embs, pad_masks, att_masks, adarms_cond<br><br>    def forward(self, observation, actions, noise=None, time=None) -&gt; Tensor:<br>        &quot;&quot;&quot;Do a full training forward pass and compute the loss (batch_size x num_steps x num_motors)&quot;&quot;&quot;<br>        images, img_masks, lang_tokens, lang_masks, state = self._preprocess_observation(observation, train=True)<br><br>        if noise is None:<br>            noise = self.sample_noise(actions.shape, actions.device)<br><br>        if time is None:<br>            time = self.sample_time(actions.shape[0], actions.device)<br><br>        time_expanded = time[:, None, None]<br>        x_t = time_expanded * noise + (1 - time_expanded) * actions<br>        u_t = noise - actions<br><br>        prefix_embs, prefix_pad_masks, prefix_att_masks = self.embed_prefix(images, img_masks, lang_tokens, lang_masks)<br>        suffix_embs, suffix_pad_masks, suffix_att_masks, adarms_cond = self.embed_suffix(state, x_t, time)<br>        if (<br>            self.paligemma_with_expert.paligemma.language_model.layers[0].self_attn.q_proj.weight.dtype<br>            == torch.bfloat16<br>        ):<br>            suffix_embs = suffix_embs.to(dtype=torch.bfloat16)<br>            prefix_embs = prefix_embs.to(dtype=torch.bfloat16)<br><br>        pad_masks = torch.cat([prefix_pad_masks, suffix_pad_masks], dim=1)<br>        att_masks = torch.cat([prefix_att_masks, suffix_att_masks], dim=1)<br><br>        att_2d_masks = make_att_2d_masks(pad_masks, att_masks)<br>        position_ids = torch.cumsum(pad_masks, dim=1) - 1<br><br>        # Prepare attention masks<br>        att_2d_masks_4d = self._prepare_attention_masks_4d(att_2d_masks)<br><br>        # Apply gradient checkpointing if enabled<br>        def forward_func(prefix_embs, suffix_embs, att_2d_masks_4d, position_ids, adarms_cond):<br>            (_, suffix_out), _ = self.paligemma_with_expert.forward(<br>                attention_mask=att_2d_masks_4d,<br>                position_ids=position_ids,<br>                past_key_values=None,<br>                inputs_embeds=[prefix_embs, suffix_embs],<br>                use_cache=False,<br>                adarms_cond=[None, adarms_cond],<br>            )<br>            return suffix_out<br><br>        suffix_out = self._apply_checkpoint(<br>            forward_func, prefix_embs, suffix_embs, att_2d_masks_4d, position_ids, adarms_cond<br>        )<br><br>        suffix_out = suffix_out[:, -self.config.action_horizon :]<br>        suffix_out = suffix_out.to(dtype=torch.float32)<br><br>        # Apply gradient checkpointing to final action projection if enabled<br>        def action_out_proj_func(suffix_out):<br>            return self.action_out_proj(suffix_out)<br><br>        v_t = self._apply_checkpoint(action_out_proj_func, suffix_out)<br><br>        return F.mse_loss(u_t, v_t, reduction=&quot;none&quot;)<br><br>    @torch.no_grad()<br>    def sample_actions(self, device, observation, noise=None, num_steps=10) -&gt; Tensor:<br>        &quot;&quot;&quot;Do a full inference forward and compute the action (batch_size x num_steps x num_motors)&quot;&quot;&quot;<br>        bsize = observation.state.shape[0]<br>        if noise is None:<br>            actions_shape = (bsize, self.config.action_horizon, self.config.action_dim)<br>            noise = self.sample_noise(actions_shape, device)<br><br>        images, img_masks, lang_tokens, lang_masks, state = self._preprocess_observation(observation, train=False)<br><br>        prefix_embs, prefix_pad_masks, prefix_att_masks = self.embed_prefix(images, img_masks, lang_tokens, lang_masks)<br>        prefix_att_2d_masks = make_att_2d_masks(prefix_pad_masks, prefix_att_masks)<br>        prefix_position_ids = torch.cumsum(prefix_pad_masks, dim=1) - 1<br><br>        # Compute image and language key value cache<br>        prefix_att_2d_masks_4d = self._prepare_attention_masks_4d(prefix_att_2d_masks)<br>        self.paligemma_with_expert.paligemma.language_model.config._attn_implementation = &quot;eager&quot;  # noqa: SLF001<br><br>        _, past_key_values = self.paligemma_with_expert.forward(<br>            attention_mask=prefix_att_2d_masks_4d,<br>            position_ids=prefix_position_ids,<br>            past_key_values=None,<br>            inputs_embeds=[prefix_embs, None],<br>            use_cache=True,<br>        )<br><br>        dt = -1.0 / num_steps<br>        dt = torch.tensor(dt, dtype=torch.float32, device=device)<br><br>        x_t = noise<br>        time = torch.tensor(1.0, dtype=torch.float32, device=device)<br>        while time &gt;= -dt / 2:<br>            expanded_time = time.expand(bsize)<br>            v_t = self.denoise_step(<br>                state,<br>                prefix_pad_masks,<br>                past_key_values,<br>                x_t,<br>                expanded_time,<br>            )<br><br>            # Euler step - use new tensor assignment instead of in-place operation<br>            x_t = x_t + dt * v_t<br>            time += dt<br>        return x_t<br><br>    def denoise_step(<br>        self,<br>        state,<br>        prefix_pad_masks,<br>        past_key_values,<br>        x_t,<br>        timestep,<br>    ):<br>        &quot;&quot;&quot;Apply one denoising step of the noise `x_t` at a given timestep.&quot;&quot;&quot;<br>        suffix_embs, suffix_pad_masks, suffix_att_masks, adarms_cond = self.embed_suffix(state, x_t, timestep)<br><br>        suffix_len = suffix_pad_masks.shape[1]<br>        batch_size = prefix_pad_masks.shape[0]<br>        prefix_len = prefix_pad_masks.shape[1]<br><br>        prefix_pad_2d_masks = prefix_pad_masks[:, None, :].expand(batch_size, suffix_len, prefix_len)<br><br>        suffix_att_2d_masks = make_att_2d_masks(suffix_pad_masks, suffix_att_masks)<br><br>        full_att_2d_masks = torch.cat([prefix_pad_2d_masks, suffix_att_2d_masks], dim=2)<br><br>        prefix_offsets = torch.sum(prefix_pad_masks, dim=-1)[:, None]<br>        position_ids = prefix_offsets + torch.cumsum(suffix_pad_masks, dim=1) - 1<br><br>        # Prepare attention masks<br>        full_att_2d_masks_4d = self._prepare_attention_masks_4d(full_att_2d_masks)<br>        self.paligemma_with_expert.gemma_expert.model.config._attn_implementation = &quot;eager&quot;  # noqa: SLF001<br><br>        outputs_embeds, _ = self.paligemma_with_expert.forward(<br>            attention_mask=full_att_2d_masks_4d,<br>            position_ids=position_ids,<br>            past_key_values=past_key_values,<br>            inputs_embeds=[None, suffix_embs],<br>            use_cache=False,<br>            adarms_cond=[None, adarms_cond],<br>        )<br><br>        suffix_out = outputs_embeds[1]<br>        suffix_out = suffix_out[:, -self.config.action_horizon :]<br>        suffix_out = suffix_out.to(dtype=torch.float32)<br>        return self.action_out_proj(suffix_out)</pre><h3>Test results:</h3><p>Different datasets from simulation benchmarks such as Libero, and Aloha-sim as well as real robot dataset such as Droid, Aloha, and OXE were used for evaluation and you are referred to the papers for more detail. In general π0.5 outperforms π0 across Different tasks as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3eSoyA8XUx4ZMB4kKjjNbg.png" /></figure><h3>Conclusion &amp; Future Work</h3><p>The foundational VLA models are an important paradigm that can be used by any as the base and then fine-tuned for any robot in any environment. These models have opened a great potential for models that generalize to different robots in various environments. Physical Intelligence has also recently released a new model π0.6* that uses reinforcement learning for fine-tuning that I will talk about in the next blogs. Till then, keep learning before robots learn everything!</p><h3>References</h3><p>1- <a href="https://arxiv.org/abs/2406.09246">https://arxiv.org/abs/2406.09246</a></p><p><a href="https://www.pi.website/blog/pi0">2-https://www.pi.website/blog/pi0</a></p><p>3-<a href="https://www.pi.website/blog/pi05">https://www.pi.website/blog/pi05</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=663dc55aa9d3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Attention is NOT all you need! Will Diffusion LLMs replace transformers?]]></title>
            <link>https://medium.com/@siamak.yousefi.1984_58741/attention-is-not-all-you-need-will-diffusion-llms-replace-transformers-4211e49760c6?source=rss-024a4ac34f32------2</link>
            <guid isPermaLink="false">https://medium.com/p/4211e49760c6</guid>
            <dc:creator><![CDATA[Siamak Yousefi ]]></dc:creator>
            <pubDate>Sat, 24 May 2025 18:45:28 GMT</pubDate>
            <atom:updated>2025-05-24T18:45:28.539Z</atom:updated>
            <content:encoded><![CDATA[<p>During the past couple of years the transformers have undeniably reshaped the Natural Language Processing (NLP) landscape, powered by training on massive amount of text datasets. However, just as Transformers seemed to be the ultimate architecture for Large Language Models (LLMs), this spring a pivotal announcement from the AI community was made: the advent of diffusion-based LLMs. At the forefront is inceptionlabs.ai, which recently unveiled Mercury, its commercial chatbot built on this idea that in the similar manner that image is generated using diffusion models from noise one might be able to generate a big chunk of text iteratively from scratch. The company, which is founded by a Stanford professor and his lab, claims that Mercury is almost 10X faster than commercial chatbots used these days. The query responses are blazingly fast and you would see a big chunk of text suddenly appear on your screen. It can generate 1000 token in a second on off the shelf NVIDIA GPUs. Give it a try and you will love it. It has some free trial!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gs0RN8-pLQlzqsMLdiSeUg.png" /><figcaption>Number of tokens per second by different popular models. Figure from ArtificialAnalysis.ai</figcaption></figure><p><a href="https://www.inceptionlabs.ai/introducing-mercury">https://www.inceptionlabs.ai/introducing-mercur</a></p><h4>Diffusion Models</h4><p>But how is it done? Probably you have already observed the images and musics generated by AI that really look and sound real. Behind most of these AI generated content are the so-called diffusion models that are trained by iteratively adding small amount of noise to an image or audio waveform such that noise dominates. The literature has defined two terms for train and inference, i.e., forward process and reverse process, respectively that are explained below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BVYeK6IOOrDwmrjatYmNLg.png" /><figcaption>Figure from paper: <a href="https://arxiv.org/abs/2006.11239">https://arxiv.org/abs/2006.11239</a>; Iterative steps of the diffusion model. During training at step 0, image is perturbed with a small amount of noise using a distribution q(.|.) and after T steps the signal is completely dominated by noise. During the inference the noise is gradually removed till the new data is generated.</figcaption></figure><p><strong>Forward process (q-step):</strong></p><p>The forward process, in which the noise is gradually added to the signal is learned using U-Nets which are some kind of neural networks that have a U-shaped connection from input to output including encoder and decoder and direct connections from input layers to output layer. Alternatively, to learn the noise distribution it has been recently shown that instead of U-Net you could use transformers for training a diffusion model and it is actually better than U-Nets. However, in order not to confuse transformer diffusion model with transformer LLM I will stick to U-Nets in this blog! You may check the Unet paper below:</p><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/1505.04597&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/53fbfc4ea2e57e6577788c5289b9fb26/href">https://medium.com/media/53fbfc4ea2e57e6577788c5289b9fb26/href</a></iframe><p><strong>Reverse process (p-step):</strong></p><p>During the inference, a noisy image or audio is sampled from a Gaussian distribution and iteratively noise is removed from them based on the weight and biases learned during the forward process by U-Nets. After enough iterations the image or audio generated become very natural like the ones used during the training phase. Here is the code of the diffusion model and Unet classes:</p><pre>class DiffusionProcess:<br>    &quot;&quot;&quot;<br>    A class representing the diffusion process for a generative model.<br><br>    Attributes:<br>        num_steps (int): The number of diffusion steps.<br>        beta (numpy.ndarray): A linearly spaced array of beta values for each step.<br>        alpha (numpy.ndarray): The complement of beta values (1 - beta).<br>        alpha_bar (numpy.ndarray): The cumulative product of alpha values.<br><br>    Methods:<br>        q_sample(x_0, t):<br>            Samples a noisy version of the input data `x_0` at a given timestep `t` <br>            using the forward diffusion process.<br><br>        p_sample(model, x_t, t):<br>            Samples a denoised version of the input data `x_t` at a given timestep `t` <br>            using the reverse diffusion process and a provided model.<br>    &quot;&quot;&quot;<br>    def __init__(self, num_steps, beta_start, beta_end):<br>        self.num_steps = num_steps<br>        self.beta = np.linspace(beta_start, beta_end, num_steps)<br>        self.alpha = 1 - self.beta<br>        self.alpha_bar = np.cumprod(self.alpha)<br><br>    def q_sample(self, x_0, t):<br>        noise = torch.randn_like(x_0)<br>        alpha_bar_t = torch.gather(torch.tensor(self.alpha_bar, dtype=torch.float32), 0, t)<br>        return torch.sqrt(alpha_bar_t)[:, None] * x_0 + torch.sqrt(1 - alpha_bar_t)[:, None] * noise<br><br>    def p_sample(self, model, x_t, t):<br>        with torch.no_grad():<br>            epsilon_theta = model(x_t, t)<br>            alpha_t = torch.gather(torch.tensor(self.alpha, dtype=torch.float32), 0, t)<br>            alpha_bar_t = torch.gather(torch.tensor(self.alpha_bar, dtype=torch.float32), 0, t)<br>            alpha_bar_t_minus_1 = torch.gather(torch.tensor(self.alpha_bar[:-1], dtype=torch.float32), 0, t - 1)<br>            mean = (torch.sqrt(alpha_t) / torch.sqrt(1 - alpha_bar_t))[:, None] * x_t - \<br>                   (torch.sqrt(1 - alpha_t) * torch.sqrt(alpha_bar_t_minus_1) / (1 - alpha_bar_t))[:, None] * epsilon_theta<br>            noise = torch.randn_like(x_t)<br>            return mean + torch.sqrt(1 - alpha_bar_t_minus_1)[:, None] * noise<br><br>class UNet(nn.Module):<br>    &quot;&quot;&quot;<br>    A 1D U-Net model for sequence-to-sequence tasks, designed with an encoder-decoder architecture.<br><br>    Args:<br>        in_channels (int): Number of input channels.<br>        out_channels (int): Number of output channels.<br>        num_steps (int): Number of time steps or iterations (used for additional input concatenation).<br><br>    Attributes:<br>        encoder (nn.Sequential): The encoder part of the U-Net, consisting of convolutional layers and ReLU activations.<br>        bottleneck (nn.Sequential): The bottleneck part of the U-Net, consisting of a convolutional layer and ReLU activation.<br>        decoder (nn.Sequential): The decoder part of the U-Net, consisting of transposed convolutional layers and ReLU activations.<br>        output_layer (nn.Conv1d): The final output layer that maps the decoder output to the desired number of output channels.<br><br>    Methods:<br>        forward(x, t):<br>            Forward pass of the U-Net model.<br><br>            Args:<br>                x (torch.Tensor): Input tensor of shape (batch_size, in_channels, sequence_length).<br>                t (torch.Tensor): Time step tensor of shape (batch_size,) to be concatenated as an additional channel.<br><br>            Returns:<br>                torch.Tensor: Output tensor of shape (batch_size, out_channels, sequence_length).<br>    &quot;&quot;&quot;<br>    def __init__(self, in_channels, out_channels, num_steps):<br>        super(UNet, self).__init__()<br>        self.in_channels = in_channels<br>        self.out_channels = out_channels<br>        self.num_steps = num_steps<br><br>        self.encoder = nn.Sequential(<br>            nn.Conv1d(in_channels + 1, 64, kernel_size=3, padding=1),  # +1 for t<br>            nn.ReLU(),<br>            nn.Conv1d(64, 128, kernel_size=3, padding=1),<br>            nn.ReLU(),<br>            nn.Conv1d(128, 256, kernel_size=3, padding=1),<br>            nn.ReLU()<br>        )<br><br>        self.bottleneck = nn.Sequential(<br>            nn.Conv1d(256, 512, kernel_size=3, padding=1),<br>            nn.ReLU()<br>        )<br><br>        self.decoder = nn.Sequential(<br>            nn.ConvTranspose1d(512, 256, kernel_size=3, stride=1, padding=1),<br>            nn.ReLU(),<br>            nn.ConvTranspose1d(256, 128, kernel_size=3, stride=1, padding=1),<br>            nn.ReLU(),<br>            nn.ConvTranspose1d(128, 64, kernel_size=3, stride=1, padding=1),<br>            nn.ReLU()<br>        )<br><br>        self.output_layer = nn.Conv1d(64, out_channels, kernel_size=1)<br><br>    def forward(self, x, t):<br>        t = t.view(-1, 1, 1).float()  # Reshape t to match x&#39;s dimensions<br>        x = torch.cat([x, t], dim=1)  # Concatenate along the channel dimension<br><br>        x = self.encoder(x)<br>        x = self.bottleneck(x)<br>        x = self.decoder(x)<br>        x = self.output_layer(x)<br><br>        return x</pre><h4>Autoregressive vs Diffusion</h4><p>In transformers the tokens are being passed to the model in an autoregressive way, meaning that for every sequence of tokens in the input the next token is used for the output during training. This makes the training phase very time consuming and limited by the sequence length as the computation cost grows quadratically with the length of the sequence. In addition, the inference will also be one token at a time and for some real time applications it can be annoying.</p><p>In contrast, in the diffusion LLM, the same idea that is applied on the image is applied to a sequence of text tokens. A small amount of noise is added to tokens iteratively which will make the tokens change to a different token corresponding to another word. After a large amount of iterations the U-Net learns the process. During the denoising process, the model starts with some random noisy sequence and starts gradually removing noise from the tokens. After enough iterations, the generated text will be a reasonable response to the query.</p><pre>import torch<br>import torch.nn as nn<br>import torch.optim as optim<br>from torch.utils.data import DataLoader, Dataset<br>import numpy as np<br><br><br>class TextDataset(Dataset):<br>    &quot;&quot;&quot;<br>    A PyTorch Dataset class for handling sequences of text data.<br><br>    Args:<br>        data (list or array-like): The dataset containing sequences of text data.<br>        seq_length (int): The fixed length of sequences to be returned. <br>                          Sequences shorter than this length will be padded, <br>                          and sequences longer than this length will be truncated.<br><br>    Methods:<br>        __len__():<br>            Returns the total number of sequences in the dataset.<br><br>        __getitem__(idx):<br>            Retrieves the sequence at the specified index, adjusts its length <br>            to `seq_length` by padding or truncating, and returns it as a <br>            PyTorch tensor with an added channel dimension.<br><br>    Returns:<br>        torch.Tensor: A tensor representation of the sequence at the specified <br>                      index, with shape (1, seq_length).<br>    &quot;&quot;&quot;<br>    def __init__(self, data, seq_length):<br>        self.data = data<br>        self.seq_length = seq_length<br><br>    def __len__(self):<br>        return len(self.data)<br><br>    def __getitem__(self, idx):<br>        sequence = self.data[idx]<br>        if len(sequence) &lt; self.seq_length:<br>            sequence = np.pad(sequence, (0, self.seq_length - len(sequence)), &#39;constant&#39;)<br>        else:<br>            sequence = sequence[:self.seq_length]<br>        return torch.tensor(sequence, dtype=torch.float32).unsqueeze(0)  # Add channel dimension<br><br>def train(model, dataloader, diffusion_process, optimizer, num_epochs):<br>    &quot;&quot;&quot;<br>    Trains a model using a diffusion process and mean squared error loss.<br><br>    Args:<br>        model (torch.nn.Module): The neural network model to be trained.<br>        dataloader (torch.utils.data.DataLoader): DataLoader providing the training data.<br>        diffusion_process (object): An object representing the diffusion process, <br>            which must implement a `q_sample` method to add noise to the data.<br>        optimizer (torch.optim.Optimizer): Optimizer for updating the model&#39;s parameters.<br>        num_epochs (int): The number of epochs to train the model.<br><br>    Returns:<br>        None<br><br>    Notes:<br>        - The training process involves adding noise to the input data using the diffusion process,<br>          predicting the noise using the model, and computing the loss between the predicted noise<br>          and randomly generated noise.<br>        - The loss is computed using mean squared error (MSE).<br>        - The model&#39;s parameters are updated using backpropagation and the provided optimizer.<br>    &quot;&quot;&quot;<br>    criterion = nn.MSELoss()<br><br>    for epoch in range(num_epochs):<br>        for batch in dataloader:<br>            batch = batch.unsqueeze(1)  # Add channel dimension<br><br>            # Forward pass: add noise<br>            t = torch.randint(0, diffusion_process.num_steps, (batch.size(0),))<br>            noisy_batch = diffusion_process.q_sample(batch, t)<br><br>            # Forward pass: predict noise<br>            predicted_noise = model(noisy_batch, t)<br><br>            # Compute loss<br>            loss = criterion(predicted_noise, torch.randn_like(predicted_noise))<br><br>            # Backward pass and optimization<br>            optimizer.zero_grad()<br>            loss.backward()<br>            optimizer.step()<br><br>        print(f&#39;Epoch {epoch+1}/{num_epochs}, Loss: {loss.item()}&#39;)</pre><p>The code to do inference is also as follows:</p><pre>def sample(model, diffusion_process, seq_length, num_samples=1):<br>    &quot;&quot;&quot;<br>    Generates samples using a diffusion process and a given model.<br><br>    Args:<br>        model (torch.nn.Module): The model used to guide the diffusion process.<br>        diffusion_process (object): An object representing the diffusion process, <br>            which must have a `num_steps` attribute and a `p_sample` method.<br>        seq_length (int): The length of the sequence to generate.<br>        num_samples (int, optional): The number of samples to generate. Defaults to 1.<br><br>    Returns:<br>        numpy.ndarray: A NumPy array containing the generated samples with shape <br>            (num_samples, seq_length).<br>    &quot;&quot;&quot;<br>    x_T = torch.randn((num_samples, in_channels, seq_length))<br>    t = torch.arange(diffusion_process.num_steps, device=x_T.device)<br><br>    for step in reversed(range(diffusion_process.num_steps)):<br>        t_step = torch.full((num_samples,), step, dtype=torch.long, device=x_T.device)<br>        x_T = diffusion_process.p_sample(model, x_T, t_step)<br><br>    return x_T.squeeze(1).cpu().numpy()</pre><p>To train with some datasets available online you may check the following repository with a link to several free datasets suitable for LLM training.</p><p><a href="https://github.com/Zjh-819/LLMDataHub">GitHub - Zjh-819/LLMDataHub: A quick guide (especially) for trending instruction finetuning datasets</a></p><p>Below is a simple call to the above model to train with a sample dataset and then test it. I didn’t want to dig into training the model with a large dataset due to lack of a strong GPU and the cost associated with it but if you have time and resources, you may do so using Google Colab or other cloud-based services.</p><pre>if __name__ == &#39;__main__&#39;:<br>    # Example data: list of token sequences<br>    data = [<br>        [1, 2, 3, 4, 5],<br>        [6, 7, 8, 9, 10],<br>        [11, 12, 13, 14, 15],<br>        [16, 17, 18, 19, 20],<br>        [21, 22, 23, 24, 25],<br>        [26, 27, 28, 29, 30],<br>        [31, 32, 33, 34, 35],<br>        [36, 37, 38, 39, 40],<br>        [41, 42, 43, 44, 45],<br>        [46, 47, 48, 49, 50]<br>        # Add more sequences<br>    ]<br><br>    dataset = TextDataset(data, seq_length=5)<br>    dataloader = DataLoader(dataset, batch_size=2, shuffle=True)<br><br>    # Initialize model, optimizer, and diffusion process<br>    in_channels = 2  # Number of input channels (e.g., token embeddings)<br>    out_channels = 1  # Number of output channels (e.g., token embeddings)<br>    num_steps = 1000  # Number of diffusion steps<br>    beta_start = 1e-4<br>    beta_end = 0.02<br><br>    model = UNet(in_channels, out_channels, num_steps)<br>    diffusion_process = DiffusionProcess(num_steps, beta_start, beta_end)<br>    optimizer = optim.Adam(model.parameters(), lr=1e-4)<br><br>    # Train the model<br>    train(model, dataloader, diffusion_process, optimizer, num_epochs=10)<br><br>    # Generate text<br>    generated_text = sample(model, diffusion_process, seq_length=5)<br>    print(generated_text)</pre><h4>Use Mercury for VSCode</h4><p>If you are into coding or vibe coding and are using VSCode, the good news is that the Mercury models are now available on Continue which is a copilot extension for VSCode. The model is so fast that it suggests a chunk of code as soon as you type a character. It might be a bit annoying if you are not into vibe coding but otherwise it is much faster than Copilot. It’s accuracy in generating code is yet to be evaluated.</p><h4>Concluding Remarks</h4><p>So in conclusion, Mercury is a big news in the LLM community that can reduce the training as well as inference time for LLM by better use of GPU. Like in image generation that sometimes the images have some strange item, there might be a bit of imperfection in some of the generated tokens but due to large amount of text data online, probably that is less of an issue for text compared to images. Therefore, Mercury could become one of the most favorite chatbots soon so don’t miss out now as an early adopter!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4211e49760c6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Music Generation with AI: Discrete Variational Auto Encoders with Diffusion Transformers]]></title>
            <link>https://medium.com/@siamak.yousefi.1984_58741/music-generation-with-ai-discrete-variational-auto-encoders-with-diffusion-transformers-197b2115399a?source=rss-024a4ac34f32------2</link>
            <guid isPermaLink="false">https://medium.com/p/197b2115399a</guid>
            <dc:creator><![CDATA[Siamak Yousefi ]]></dc:creator>
            <pubDate>Fri, 14 Feb 2025 02:09:21 GMT</pubDate>
            <atom:updated>2025-02-14T02:27:35.780Z</atom:updated>
            <content:encoded><![CDATA[<p>In this blog, I am going to show you how to create music with AI from scratch for those familiar with AI development in Pytorch. This is an advanced topic covering some of the most popular music, audio, and image generation algorithms that are used by leading tech companies. While the major concept of latent diffusion models was introduced by researchers at Google, I take a different approach by implementing vector quantized variational auto encoders (VQ-VAE) instead of traditional variational auto encoders (VAEs). The use of discrete VAE is preferred for MIDI music format which is operating in discrete domain unlike music in audio format which is in continuous domain. While previous approaches used hierarchical VAEs to enforce music representation into a continuous domain for diffusion model, I leverage the discrete characteristics of MIDI data directly. I implement the architecture in Pytorch which is both easier to understand and more popular In the AI community than Tensorflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ez8Of853FgFogyYDrImyaQ.png" /><figcaption>The proposed architecture of latent diffusion transformer: <a href="https://archives.ismir.net/ismir2021/paper/000058.pdf">https://archives.ismir.net/ismir2021/paper/000058.pdf</a></figcaption></figure><p>If you don’t know what MIDI format is, it is a digital representation of the notes, their frequency, and length. In music generation usin AI, this domain is known as ‘symbolic music generation’ since it works with digital representations of music pieces rather than raw audio. While there are many impressive and mind blowing products for AI music generation using raw audio like Suno, Adobe, StableDiffusion, and Riffusion, there are several compelling reasons to work with symbolic music generation (MIDI):</p><ul><li>First, the simplicity of working with MIDI as it is already in digital format.</li><li>Second, MIDI data size is much smaller than raw audio and makes it a more suitable choice for loading and training.</li><li>Third, unlike audio signal, the MIDI, after being generated can be modified and adjusted by a musician, for instance by changing certain notes and instruments. This is not possible with audio format except for experienced sound engineers!</li></ul><p>To load the midi files, I use the pretty_midi python package. The loaded midi file will have instrument attribute and each instrument will have the following attributes:</p><ul><li>note.start: the time-stamp of when the note started playing.</li><li>note.end: the time-stamp of when it ended.</li><li>note.pitch: the music pitch, which is the center frequency of the tone.</li><li>note.velocity: how strong is the note in terms of loudness.</li></ul><p>To encode the midi file, we can use the following function to put them inside a list, called events. It also includes an id for the track as well as info about when the note started and when it ended, i.e., ‘note_on’ and ‘note_off’.</p><pre>def encode_midi(midi_data, time_step=0.025):<br>    events = []<br>    for track_idx, instrument in enumerate(midi_data.instruments):<br>        for note in instrument.notes:<br>            start_time = note.start<br>            end_time = note.end<br>            pitch = note.pitch<br>            velocity = note.velocity<br>            <br>            # Encode note-on event with absolute time<br>            events.append((start_time, &#39;note_on&#39;, track_idx, pitch, velocity))<br>            # Encode note-off event with absolute time<br>            events.append((end_time, &#39;note_off&#39;, track_idx, pitch, velocity))<br>    <br>    # Sort events by time<br>    events.sort(key=lambda x: x[0])<br>    <br>    return events</pre><p>Let’s say for instance we have this line of music:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FpC-Kky3M5hzK-l_xynf5g.png" /><figcaption>A sample notes of a piece called “Appenzel” in G Major in 2/4 beat and tempo 120 beats/seconds.</figcaption></figure><p>The first bar after encoding looks like this:</p><p>[(0.0, ‘note_on’, 0, 62, 90), (0.125, ‘note_off’, 0, 62, 90), (0.125, ‘note_on’, 0, 62, 90), (0.25, ‘note_off’, 0, 62, 90), (0.25, ‘note_on’, 0, 67, 90), (0.5, ‘note_off’, 0, 67, 90), (0.5, ‘note_on’, 0, 67, 90), (0.75, ‘note_off’, 0, 67, 90)]</p><p>The current list of events may not be a good choice for sequence models, since we want to have a sequence of notes where the notes of multiple instruments are sequenced after each other based on their appearance in the music sheet. So, we convert the events list to sequence list by using a small time step (time_step) and calculating the time difference between the start of new note and the previous note.</p><pre>def events_to_sequence(events, time_step=0.025):<br>    sequence = []<br>    current_time = 0<br>    <br>    for event in events:<br>        event_time, event_type, track_idx, pitch, velocity = event<br>        <br>        # # Add time shift if necessary<br>        time_diff = int((event_time - current_time) / time_step)<br>        if time_diff &gt; 0:<br>            sequence.append((&#39;time_shift&#39;, time_diff))<br>            current_time += time_diff * time_step<br>        <br>        # Add the event<br>        sequence.append((event_type, track_idx, pitch, velocity))<br>    <br>    return sequence</pre><p>The sequence of the previous events looks like this:</p><p>[(‘note_on’, 0, 62, 90), (‘time_shift’, 2), (‘note_off’, 0, 62, 90), (‘note_on’, 0, 62, 90), (‘time_shift’, 2), (‘note_off’, 0, 62, 90), (‘note_on’, 0, 67, 90), (‘time_shift’, 5), (‘note_off’, 0, 67, 90), (‘note_on’, 0, 67, 90), (‘time_shift’, 5), (‘note_off’, 0, 67, 90)]</p><p>where the time_shift is to denote how many time_steps exists between a note-on and note-off tuples.</p><p>Now we have to tokenize the sequence because like in text we cannot feed strings to the model. First, let us create vocabulary of all events that exists in our train dataset. So, if the event is a time shift depending on the length it is assigned a new vocab item.</p><pre>def create_vocabulary(sequence):<br>    vocab = set()<br>    print(len(sequence))<br>    for event in sequence:<br>        if event[0] == &#39;time_shift&#39;:<br>            vocab.add(f&quot;time_shift_{event[1]}&quot;)<br>        else:<br>            # print(&#39;event&#39;, len(event))<br>            event_type, track_idx, pitch, velocity = event<br>            vocab.add(f&quot;{event_type}_{track_idx}_{pitch}_{velocity}&quot;)<br>    return {token: idx for idx, token in enumerate(sorted(vocab))}<br></pre><p>The tokenize_sequence function then takes the sequence and vocabulary and returns tokens of the sequence.</p><pre>def tokenize_sequence(sequence, vocab):<br>    tokens = []<br>    for event in sequence:<br>        # print(&#39;len event&#39;, len(event))<br>        if event[0] == &#39;time_shift&#39;:<br>            token = f&quot;time_shift_{event[1]}&quot;<br>        else:<br>            event_type, track_idx, pitch, velocity = event<br>            token = f&quot;{event_type}_{track_idx}_{pitch}_{velocity}&quot;<br>        tokens.append(vocab[token])<br>    return tokens</pre><p><strong>Dataset</strong></p><p>For training I used the Piano Maestro dataset which is around several GBs of MIDI piano music. But I have also applied data augmentation on the midi to transform the scale, or increase/decrease the tempo. This way the robustness of the model is increased. For the sake of simplicity I only consider the right hand and the melody to be used during train in the model which will follow.</p><p><a href="https://magenta.tensorflow.org/datasets/maestro">The MAESTRO Dataset</a></p><h3><strong>Latent Representation of Sequence (VQ-VAE):</strong></h3><p>A piece of music is composed of a sequence of notes, a number of which form a phrase, which is similar to a sentence in a text. Then multiple of these phrases when played in a sequence, form a musical piece. To be able to generate new notes, a sequence to sequence model like transformer can be used. However, the computational cost of training transformer with attention grows quadratically with the sequence length. Since music is made up of several notes at each time and several different instruments, the token sequences are very long and due to correlation with the past, training the sequences using transformers will be very expensive. In order to avoid this long computational burden it has been proposed in the literature to use a latent representation of two or more bars using some sort of variational auto-encoders, called MusicVAE:</p><p><a href="https://arxiv.org/pdf/1803.05428">A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music</a></p><p>The VAEs are a very popular type of autoencoder that instead of just compressing an input tensor into a smaller one, in the hidden layer, it encodes both the mean and covariance parameter tensors, representing a Gaussian distribution. The latent space is designed to approximate a zero-mean Gaussian distribution, and this is achieved through a specialized loss function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jZXewEpUNMbNbwQQTtdPqw.png" /><figcaption><a href="https://medium.com/@rushikesh.shende/autoencoders-variational-autoencoders-vae-and-%CE%B2-vae-ceba9998773d">https://medium.com/@rushikesh.shende/autoencoders-variational-autoencoders-vae-and-%CE%B2-vae-ceba9998773d</a></figcaption></figure><p>The loss function has two components:</p><ul><li>reconstruction loss: the usual loss in the autoencoders, which maximizes the conditional probability of output given latent tensor.</li><li>Kullback-Leibler (KL) divergence loss: loss between standard normal distribution <em>N(0,I) </em>and Gaussian distribution with encoded mean and covariance tensors in the latent space.</li></ul><p>In order to learn in more detail how VAE works you are referred to this blog:</p><p><a href="https://towardsdatascience.com/difference-between-autoencoder-ae-and-variational-autoencoder-vae-ed7be1c038f2">Difference between AutoEncoder (AE) and Variational AutoEncoder (VAE)</a></p><p>The base form of VAE has not shown good performance according to the authors’ claims in the MusicVAE paper. The authors proposed a hierarchical form of VAE that reportedly performs better. In this blog, I propose using a discrete version of VAE since MIDI music is also a discrete representation of music and may be better suited for this problem. One popular version of discrete VAE is the Vector Quantized VAE (VQ-VAE), where the compressed vector is compared to vectors from a codebook, and the closest vector is selected. An advantage of VQ-VAE over traditional VAE is that the model doesn’t suffer from posterior collapse, unlike VAEs, and is easier to train. The loss function optimized in VQ-VAE differs from that of VAE and is defined as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BtdrR5DtMmh2h9asOelJvA.png" /><figcaption><a href="https://arxiv.org/pdf/1711.00937v2">https://arxiv.org/pdf/1711.00937v2</a></figcaption></figure><p>where sg[.] is the stop-gradient operator which means that during training and back propagation, that tensor is frozen and not updated. Below is the architecture diagram from the original paper for image generation application:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kfbLtBmq91EE-gvQOlDVPw.png" /><figcaption>Architecture of VQ-VAE for the image generation application. <a href="https://arxiv.org/pdf/1711.00937">1711.00937</a></figcaption></figure><p>For a good high level explanation of VQVAE, I suggest this blog post:</p><p><a href="https://shashank7-iitd.medium.com/understanding-vector-quantized-variational-autoencoders-vq-vae-323d710a888a">Understanding Vector Quantized Variational Autoencoders (VQ-VAE)</a></p><p>The code of VQVAE class as well as encode/decoder and vector quantizer is shown below:</p><pre><br>class VectorQuantizer(nn.Module):<br>    def __init__(self, num_embeddings, embedding_dim, commitment_cost=0.25):<br>        super().__init__()<br>        self.num_embeddings = num_embeddings<br>        self.embedding_dim = embedding_dim<br>        self.commitment_cost = commitment_cost<br>        <br>        self.embedding = nn.Embedding(num_embeddings, embedding_dim)<br>        self.embedding.weight.data.uniform_(-1.0 / num_embeddings, 1.0 / num_embeddings)<br><br>    def forward(self, inputs):<br>        # inputs shape: [batch_size, embedding_dim]<br>        flat_input = inputs.view(-1, self.embedding_dim)<br>        <br>        # Calculate distances<br>        distances = torch.cdist(flat_input, self.embedding.weight)<br>        <br>        # Encoding<br>        encoding_indices = torch.argmin(distances, dim=1)<br>        encodings = F.one_hot(encoding_indices, self.num_embeddings).float()<br>        <br>        # Quantize<br>        quantized = self.embedding(encoding_indices)<br>        <br>        # Loss<br>        e_latent_loss = F.mse_loss(quantized.detach(), inputs)<br>        q_latent_loss = F.mse_loss(quantized, inputs.detach())<br>        loss = q_latent_loss + self.commitment_cost * e_latent_loss<br>        <br>        quantized = inputs + (quantized - inputs).detach()<br>        <br>        return quantized, loss, encoding_indices<br><br>class BarEncoder(nn.Module):<br>    def __init__(self, input_dim, hidden_dim, latent_dim):<br>        super().__init__()<br>        self.encoder = nn.Sequential(<br>            nn.Linear(input_dim, hidden_dim),<br>            nn.LayerNorm(hidden_dim),<br>            nn.ReLU(),<br>            nn.Linear(hidden_dim, hidden_dim),<br>            nn.LayerNorm(hidden_dim),<br>            nn.ReLU(),<br>            nn.Linear(hidden_dim, latent_dim) <br>            )<br><br>    def forward(self, x):<br>        return self.encoder(x)<br><br>class BarDecoder(nn.Module):<br>    def __init__(self, latent_dim, hidden_dim, output_dim):<br>        super().__init__()<br>        self.decoder = nn.Sequential(<br>            nn.Linear(latent_dim, hidden_dim),<br>            nn.LayerNorm(hidden_dim),<br>            nn.ReLU(),<br>            nn.Linear(hidden_dim, hidden_dim),<br>            nn.LayerNorm(hidden_dim),<br>            nn.ReLU(),<br>            nn.Linear(hidden_dim, output_dim)<br>            )<br><br>    def forward(self, x):<br>        return self.decoder(x)<br><br>class BarVQVAE(nn.Module):<br>    def __init__(self, input_dim, hidden_dim=512, latent_dim=256, num_embeddings=512):<br>        super().__init__()<br>        self.input_dim = input_dim<br>        self.latent_dim = latent_dim<br>        self.hidden_dim = hidden_dim<br>        self.num_embeddings = num_embeddings<br>        <br>        self.encoder = BarEncoder(input_dim, hidden_dim, latent_dim)<br>        self.vq = VectorQuantizer(num_embeddings, latent_dim)<br>        self.decoder = BarDecoder(latent_dim, hidden_dim, input_dim)<br><br>    def forward(self, x):<br>        # Flatten the bar sequence<br>        batch_size, seq_len, features = x.shape<br>        x_flat = x.reshape(batch_size, seq_len * features)<br>        <br>        # Encode<br>        z = self.encoder(x_flat)<br>        z_q, vq_loss, indices = self.vq(z)<br>        <br>        # Decode<br>        x_recon = self.decoder(z_q)<br>        x_recon = x_recon.reshape(batch_size, seq_len, features)<br>        <br>        return x_recon, vq_loss, indices<br><br>    def encode(self, x):<br>        batch_size, seq_len, features = x.shape<br>        x_flat = x.reshape(batch_size, seq_len * features)<br>        z = self.encoder(x_flat)<br>        z_q, _, indices = self.vq(z)<br>        return z_q<br><br>    def decode(self, z_q):<br>        x_recon = self.decoder(z_q)<br>        return x_recon.reshape(-1, int(self.input_dim/2), 2)<br>        # return x_recon.reshape(-1, self.bar_length, 2)<br><br>    def save_model(self, path):<br>        torch.save({<br>            &#39;model_state_dict&#39;: self.state_dict(),<br>            &#39;model_config&#39;: {<br>                &#39;input_dim&#39;: self.input_dim,<br>                &#39;hidden_dim&#39;:self.hidden_dim,<br>                &#39;latent_dim&#39;: self.latent_dim,<br>                &#39;num_embeddings&#39;: self.num_embeddings<br>            }<br>        }, path)<br><br>    @classmethod<br>    def load_model(cls, path, device=&#39;cuda&#39;, strict=True):<br>        &quot;&quot;&quot;<br>        Load model state and configuration from a saved file<br>        &quot;&quot;&quot;<br>        checkpoint = torch.load(path, map_location=device)<br><br>        model = cls(<br>            input_dim=checkpoint[&#39;model_config&#39;][&#39;input_dim&#39;],<br>            hidden_dim=checkpoint[&#39;model_config&#39;][&#39;hidden_dim&#39;],<br>            latent_dim=checkpoint[&#39;model_config&#39;][&#39;latent_dim&#39;],<br>            num_embeddings=checkpoint[&#39;model_config&#39;][&#39;num_embeddings&#39;]<br>        )<br><br>        # Load state dict<br>        model.load_state_dict(checkpoint[&#39;model_state_dict&#39;], strict=strict)<br>        model = model.to(device)<br>        model.eval()<br>        <br>        return model, checkpoint<br><br><br></pre><p>The training script for representing each bar with a latent vector is as follows:</p><pre><br>def train_vqvae_on_bars(model, dataloader, num_epochs=100, device=&quot;cuda&quot;, save_path=None):<br>    optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)<br>    best_loss = float(&#39;inf&#39;)<br><br>    scheduler = torch.optim.lr_scheduler.OneCycleLR(<br>        optimizer,<br>        max_lr=1e-3,<br>        epochs=num_epochs,<br>        steps_per_epoch=len(dataloader),<br>        pct_start=0.3,<br>    )<br>    <br>    for epoch in range(num_epochs):<br>        total_loss = 0<br>        total_recon_loss = 0<br>        total_vq_loss = 0<br>        total_perplexity = 0<br>        <br>        model.train()<br>        for batch in dataloader:<br>            optimizer.zero_grad()<br><br>            # batch shape: [batch_size, bars_per_sequence, bar_length, 2]<br>            # Flatten the sequence dimension to process each bar independently<br>            bars = batch.view(-1, batch.size(2), batch.size(3))<br>            x= bars.to(device)<br><br>            x_recon, vq_loss, indices = model(x)<br>            <br>            # Reconstruction loss<br>            recon_loss = F.mse_loss(x_recon, x)<br>            <br>            # Total loss<br>            loss = recon_loss + vq_loss<br>            <br>            loss.backward()<br>            optimizer.step()<br>            <br>            total_loss += loss.item()<br>            total_recon_loss += recon_loss.item()<br>            total_vq_loss += vq_loss.item()<br>            # total_perplexity += perplexity.item()<br>        <br>        scheduler.step()<br>        <br>        avg_loss = total_loss / len(dataloader)<br>        avg_recon_loss = total_recon_loss / len(dataloader)<br>        avg_vq_loss = total_vq_loss / len(dataloader)<br>        # avg_perplexity = total_perplexity / len(dataloader)<br>        <br>        print(f&quot;Epoch {epoch+1}/{num_epochs}:&quot;)<br>        print(f&quot;  Loss: {avg_loss:.4f}&quot;)<br>        print(f&quot;  Reconstruction Loss: {avg_recon_loss:.4f}&quot;)<br>        print(f&quot;  VQ Loss: {avg_vq_loss:.4f}&quot;)<br>        # print(f&quot;  Codebook Perplexity: {avg_perplexity:.2f}&quot;)<br><br><br>        if save_path and avg_loss &lt; best_loss:<br>            best_loss = avg_loss<br>            model.save_model(save_path)<br>            print(f&quot;Saved best model with loss: {best_loss:.4f}&quot;)<br></pre><h3>Transformers!</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/430/1*O56R3-QnyEZbu8PPAGTbVg.png" /><figcaption>Encoder-decoder architecture of attention mechanism from the paper “attention is all you need”</figcaption></figure><p>The transformers are the foundations of Large Language Models (LLMs) such as ChatGPT, Claude Sonnet, LLAMA, Gemini, and DeepSeek, to name a few. At their core, transformers rely on attention mechanism, which allows the model to focus on different parts of the input sequence when generating an output. The attention mechanism works by performing a set of matrix multiplications on a sequence of token embeddings. These operations compute attention weights that quantify the relationship between each token in the sequence. During token prediction, this mechanism determines which part of the sequence is more relevant to the token to be predicted by assigning more weight to those elements, allowing the model to capture both long-term and short-term dependencies.</p><h4>Input Sequence and Positional Encoding</h4><p>Suppose we have a sequence of latent midi tokens of size embedding_dim and length sequence_len, that we try to learn in order to predict a new sequence. During the train phase, we should start feeding the sequence of tokens to the transformer where the output is the next token. So for example, we select a fixed sequence of length of say N and then we shift it by one and we get the output sequence. Now in the above diagram you can see where the input and the output (shifted input) are being fed. Unlike RNN and LSTM where the input sequence is processed one by one, in transformer, a fixed sequence length is fed through the network at once. To preserve the information about the order of the tokens in the sequence, another embedding is used known as positional encoding which is meant to indicate the position of a token in a sequence. The positional encoding is then added with the input embedding to have an embedding that has information about the order of the token in the sequence.</p><h4>Attention, Attention!</h4><p>The way the attention module works is to use three learnable matrices W_q, W_k, W_v to be multiplied with the embedding and get a new set of tensors query (Q), key (K), and values (V). This can be done using nn.Linear(hidden_dim, hidden_dim). The dimension of these tensors are (batch_size, sequence_len, embedding_dim). In transformer architecture, the embedding dimension is divided into num_heads chunks of size</p><ul><li>head_dim = embedding_dim/num_heads.</li></ul><p>This is the so-called multi-head attention mechanism where multiple parallel modules are used to enhance the diversity.</p><p>After reshaping and transposing, the dimensions of the Q, K, and V tensors become:</p><pre>(batch_size, num_heads, sequence_len, head_dim)</pre><p>The next important step is to multiply Q, K tensors using a scaled dot product to compute the attention scores. Attention score becomes a tensor of shape (batch_size, num_heads, sequence_len, sequence_len), where each value on the sequence_len*sequence_len tensor shows how each token attends to other tokens in the sequence. The larger the score, the more the two tokens are related to each other. One important consequence of this computation is that the computational cost of transformers grows quadratically with length of sequence it consumes. For midi sequence, it makes more sense to also put a causal mask on the attention scores as the future notes do not have impact on the past notes.</p><p>Attention scores are then passed through a Softmax layer to get the probabilities of each token and then go through another multiplication with tensor V. The dimension of output tensor after multiplication becomes (batch_size, num_heads, sequence_len, head_dim). The attention architecture is shown in this diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/1*NcxRR9qz_YGuRlJ_D_98yQ.png" /></figure><p>Then a concatenation is applied accross all heads to get the dimension: (batch_size, sequence_len, embeddin_dim), which is the original dimension of the input sequence. In Pytorch there is a package that you can use for transformers and makes your life much easier.</p><h3><strong>Diffusion Models</strong></h3><p>In the past few years, generative adversarial networks (GANs) have been the trend for image or audio generation. But in the past year or so the quality of these pictures have changed a lot. The algorithm behind most of these AI-generate photos you see online is actually not GANs but a model called diffusion model. The training and inference of diffusion model is as follows.</p><h4><strong>Forward Process</strong></h4><p>The idea of diffusion model is that during the train phase it uses the input signal and starts adding noise to it sequentially until it follows a zero mean Gaussian distribution with identity covariance matrix. The amount of noise at each iteration is a function of iteration step <em>t</em>. The process where the noise is added gradually at each step is called the forward process and the distribution of the noise is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/270/0*CTHZtsAIYa5QqcKb" /></figure><p>Where <strong>q </strong>is our forward process, <strong><em>x_t</em></strong>​ is the output of the forward process at step <strong>t</strong>. <strong>N</strong> is the Gaussian distribution with mean being a scaled version of <em>x_t</em> and standard deviation being <em>b_t</em>.</p><p>After doing some calculation iteratively it can be shown that the distribution at step <em>t</em> based on step 0 is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/260/0*Yn7DTIqQtojoILCj" /></figure><p>where</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/84/0*fz6zuZqxfvz6Vkp4" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/82/0*J8jzMymwmQrHt9Xm" /></figure><p>So one can calculate the amount of noise to be added to the signal x_t at each step <em>t</em>. Note that since <em>alpha_{t} &lt;1</em>, the mean of the distribution converges to zero and the variance converges to 1, i.e., the distribution of the signal after enough steps converges to a standard normal distribution.</p><h4><strong>Reverse Process</strong></h4><p>During the inference the idea is to start from a zero mean Gaussian distribution at time <em>t</em> and start predicting the amount of noise at each time step to get something that is close enough to the signal after a certain number of iterations. The model should have learned the mean and variance of the noise distribution and how much it should remove from the previous noisy data.</p><p>The diffusion model class is defined below. In the train phase we call the “add_noise” method iteratively to make a noisier version of the signal. During the inference we call the “sample_sequence” method to predict the noise at each time step <em>t </em>and then reconstruct signal from the noise based on the trained model. For the model to learn the weights we use a transformer.</p><pre>class SequenceDiffusion:<br>    def __init__(self, model, n_steps=1000, beta_start=1e-4, beta_end=0.02):<br>        self.model = model<br>        self.device = next(model.parameters()).device<br><br>        self.n_steps = n_steps<br>        self.beta = torch.linspace(beta_start, beta_end, n_steps).to(self.device)<br>        self.alpha = 1 - self.beta<br>        self.alpha_bar = torch.cumprod(self.alpha, dim=0)<br>        self.sigma = torch.sqrt(self.beta)<br><br><br>    def diffuse_sequence(self, x, t):<br>        sqrt_alpha_bar = torch.sqrt(self.alpha_bar[t])<br>        sqrt_one_minus_alpha_bar = torch.sqrt(1 - self.alpha_bar[t])<br>        epsilon = torch.randn_like(x)<br>        <br>        # Expand dimensions to match the input shape<br>        sqrt_alpha_bar = sqrt_alpha_bar.view(-1, 1, 1,1)<br>        sqrt_one_minus_alpha_bar = sqrt_one_minus_alpha_bar.view(-1, 1, 1,1)<br>        <br>        return sqrt_alpha_bar * x + sqrt_one_minus_alpha_bar * epsilon, epsilon<br><br>    @torch.no_grad()<br>    def sample_sequence(self, shape, device, conditioning=None):<br>        x = torch.randn(shape, device=device)<br>        <br>        for t in reversed(range(self.n_steps)):<br>            timesteps = torch.full((shape[0],), t, device=device, dtype=torch.long)<br>            predicted_noise = self.model(x, timesteps)<br>            <br>            alpha = self.alpha[t]<br>            alpha_bar = self.alpha_bar[t]<br>            beta = self.beta[t]<br>            <br>            if t &gt; 0:<br>                noise = torch.randn_like(x)<br>            else:<br>                noise = 0<br><br>            x = 1 / torch.sqrt(alpha) * (x - (1 - alpha) / torch.sqrt(1 - alpha_bar) * predicted_noise)<br>            x = x + torch.sqrt(beta) * noise<br>            <br>        return x</pre><h3>Diffusion Transformer for MIDI sequence generation</h3><p>Now that you’ve learned about diffusion model and transformers, the core idea of the diffusion transformer is to use a transformer to learn the denoising process. It is worth mentioning that when diffusion models were first proposed, an architecture called U-Net, which looks like a U shape from the encoder to decoder, was initially proposed as the network learning the denoising step in diffusion model. However, it was later found out that a transformer is actually more efficient than U-Net architecture and has since been the preferred architecture.</p><p>The input to the diffusion transformer is a sequence of latent vectors from the VQVAE output, where each vector has encoded one or more bars of MIDI notes from different instruments. These vectors undergo the diffusion process, iteratively transforming the discrete tokens that represent musical elements into noisy versions of the original sequence.</p><p>The noisy sequence is passed to the transformer, which is trained to predict the original latent embedding from the noisy one. The transformer takes the noisy sequence as input and returns the last latent embedding in the sequence as output. The transformer is an autoregressive model, where it analyzes the contextual relationships within the sequence to predict the next embedding.</p><p>Once the diffusion transformer is trained, during the inference the de-noised sequence of latent vectors is passed to the VQVAE decoder to return music tokens. The tokens are then converted to MIDI and returned as the generated piece of music.</p><p>The code of diffusion transformer is shown below.</p><pre><br><br>class PositionalEncoding(nn.Module):<br>    def __init__(self, d_model, dropout=0.1, max_len=5000):<br>        super().__init__()<br>        self.dropout = nn.Dropout(p=dropout)<br><br>        position = torch.arange(max_len).unsqueeze(1)<br>        div_term = torch.exp(torch.arange(0, d_model, 2) * (-math.log(10000.0) / d_model))<br>        pe = torch.zeros(1, max_len, d_model)<br>        pe[0, :, 0::2] = torch.sin(position * div_term)<br>        pe[0, :, 1::2] = torch.cos(position * div_term)<br>        self.register_buffer(&#39;pe&#39;, pe)<br><br>    def forward(self, x):<br>        x = x + self.pe[:, :x.size(1)]<br>        return self.dropout(x)<br><br>class DiffusionSequenceTransformer(nn.Module):<br>    def __init__(self, latent_dim, d_model=512, nhead=8, num_layers=6, dropout=0.1):<br>        super().__init__()<br>        self.latent_dim = latent_dim<br>        self.d_model = d_model<br><br>        self.num_layers = num_layers<br>        self.nhead = nhead<br>        <br>        # Input embedding<br>        # self.input_embed = nn.Linear(latent_dim, d_model)<br>        # Improved input embedding<br>        self.input_embed = nn.Sequential(<br>            nn.Linear(latent_dim, d_model),<br>            nn.LayerNorm(d_model),<br>            nn.GELU()<br>        )<br><br><br>        # More sophisticated time embedding<br>        self.time_embed = nn.Sequential(<br>            nn.Linear(d_model, d_model * 4),<br>            nn.LayerNorm(d_model * 4),<br>            nn.SiLU(),<br>            nn.Linear(d_model * 4, d_model),<br>            nn.LayerNorm(d_model),<br>            nn.SiLU()<br>        )<br>        <br>        # Positional encoding<br>        self.pos_encoder = PositionalEncoding(d_model, dropout)<br>        <br><br>        # Encoder<br>        encoder_layer = nn.TransformerEncoderLayer(<br>            d_model=d_model,<br>            nhead=nhead,<br>            dim_feedforward=d_model * 4,<br>            dropout=dropout,<br>            batch_first=True<br>        )<br>        self.encoder = nn.TransformerEncoder(encoder_layer, num_layers)<br>        <br>        # Decoder<br>        decoder_layer = nn.TransformerDecoderLayer(<br>            d_model=d_model,<br>            nhead=nhead,<br>            dim_feedforward=d_model * 4,<br>            dropout=dropout,<br>            batch_first=True<br>        )<br>        self.decoder = nn.TransformerDecoder(decoder_layer, num_layers)<br>        <br><br>        # Deeper noise projection<br>        self.noise_proj = nn.Sequential(<br>            nn.Linear(d_model, d_model * 2),<br>            nn.LayerNorm(d_model * 2),<br>            nn.GELU(),<br>            nn.Dropout(0.1),<br>            nn.Linear(d_model * 2, d_model),<br>            nn.LayerNorm(d_model),<br>            nn.GELU(),<br>            nn.Linear(d_model, latent_dim)<br>        )<br><br><br>    def forward(self, src, tgt, timesteps):<br>        # src, tgt shape: [batch_size, seq_len, latent_dim]<br>        batch_size, seq_len, _ = src.shape<br>        <br>        # Embeddings<br>        src_emb = self.input_embed(src)<br>        tgt_emb = self.input_embed(tgt)<br>        <br>        # Time embeddings<br>        time_emb = self.get_timestep_embedding(timesteps, self.d_model)<br>        time_emb = self.time_embed(time_emb)<br>        <br>        # Add positional encoding and time embeddings<br>        src_emb = self.pos_encoder(src_emb) + time_emb.unsqueeze(1)<br>        tgt_emb = self.pos_encoder(tgt_emb) + time_emb.unsqueeze(1)<br>        <br>        # Create masks for decoder<br>        tgt_mask = self.generate_square_subsequent_mask(seq_len).to(src.device)<br>        <br>        # Encode source sequence<br>        memory = self.encoder(src_emb)<br>        <br>        # Decode target sequence<br>        output = self.decoder(tgt_emb, memory, tgt_mask=tgt_mask)<br><br>        # Project to noise prediction<br>        predicted_noise = self.noise_proj(output)<br>        <br>        # Project back to latent space<br>        return predicted_noise  # self.output_proj(output)<br><br>    def generate_square_subsequent_mask(self, sz):<br>        mask = (torch.triu(torch.ones(sz, sz)) == 1).transpose(0, 1)<br>        mask = mask.float().masked_fill(mask == 0, float(&#39;-inf&#39;)).masked_fill(mask == 1, float(0.0))<br>        return mask<br><br><br>    def get_timestep_embedding(self, timesteps, dim):<br>        half = dim // 2<br>        freqs = torch.exp(-math.log(10000) * torch.arange(0, half, device=timesteps.device) / half)<br>        args = timesteps[:, None].float() * freqs[None]<br>        embedding = torch.cat([torch.cos(args), torch.sin(args)], dim=-1)<br>        if dim % 2:<br>            embedding = torch.cat([embedding, torch.zeros_like(embedding[:, :1])], dim=-1)<br>        return embedding<br><br>    def save_model(self, path):<br>        torch.save({<br>            &#39;model_state_dict&#39;: self.state_dict(),<br>            &#39;latent_dim&#39;: self.latent_dim,<br>            &#39;d_model&#39;: self.d_model,<br>            &#39;nhead&#39;: self.nhead,<br>            &#39;num_layers&#39;: self.num_layers<br>        }, path)<br>        print(f&quot;Diffusion model saved to {path}&quot;)</pre><pre><br>def train_diffusion_sequence(model, diffusion, dataloader, num_epochs, device, save_path=None):<br>    optimizer = torch.optim.AdamW(model.parameters(), lr=2e-4, betas=(0.9, 0.999), weight_decay=0.01)<br>    best_loss = float(&#39;inf&#39;)<br><br>    for epoch in range(num_epochs):<br>        model.train()<br>        total_loss = 0<br>        <br>        for batch_idx,(src_seq, target_seq) in enumerate(dataloader):<br>            src_seq = src_seq.to(device)<br>            tgt_seq = target_seq.to(device)<br><br>            batch_size = src_seq.shape[0]<br>            <br>            optimizer.zero_grad()<br>            <br>            # Sample timesteps<br>            t = torch.randint(0, diffusion.n_steps, (batch_size,), device=device)<br>            <br>            # Add noise to sequence<br>            noised_src, src_noise = diffusion.diffuse_sequence(src_seq, t)<br>            noised_tgt, tgt_noise = diffusion.diffuse_sequence(tgt_seq, t)<br>    <br>            # Teacher forcing: use target sequence shifted right<br>            tgt_input = noised_tgt # [:, :-1]  # Remove last token<br>            tgt_output = tgt_seq # [:, 1:]     # Remove first token<br><br>            # Predict<br>            predicted_noise = model(noised_src.squeeze(2), tgt_input.squeeze(2), t) ### noise predicted<br><br>            # # Predict noise<br>            # predicted_noise = model(noised_seq, t)<br><br>            tgt_noise = tgt_noise.squeeze(2)<br>            tgt_output = tgt_output.squeeze(2)<br><br>            noised_tgt = noised_tgt.squeeze(2)<br>            <br>            # Calculate diffusion losses<br>            diff_loss = F.mse_loss(predicted_noise, tgt_noise) # [:, 1:])  # Diffusion loss<br>                       <br>            # Get denoised prediction using predicted noise<br>            alpha = diffusion.alpha[t]<br>            alpha_bar = diffusion.alpha_bar[t]<br>            # Denoise using predicted noise<br>            denoised = (1 / torch.sqrt(alpha))[:, None, None] * (<br>                noised_tgt - <br>                (1 - alpha)[:, None, None] / torch.sqrt(1 - alpha_bar)[:, None, None] * predicted_noise<br>            )<br>            <br>            recon_loss = F.mse_loss(denoised, tgt_output)        # Reconstruction loss<br>            <br>            # L1 regularization<br>            reg_loss = sum(p.abs().sum() for p in model.parameters())<br><br>            loss = diff_loss + recon_loss + 1e-5 * reg_loss<br><br>            # Gradient clipping<br>            torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=0.5)<br>            <br>            loss.backward()<br>            torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)<br>            optimizer.step()<br>            <br>            total_loss += loss.item()<br>            <br>            if batch_idx % 100 == 0:<br>                print(f&quot;Epoch {epoch+1}, Batch {batch_idx}, Loss: {loss.item():.6f}&quot;)<br>        <br>        avg_loss = total_loss / len(dataloader)<br>        print(f&quot;Epoch {epoch+1} Average Loss: {avg_loss:.6f}&quot;)<br><br>        if save_path and avg_loss &lt; best_loss:<br>            best_loss = avg_loss<br>            model.save_model(save_path)<br>            print(f&quot;Saved best model with loss: {best_loss:.4f}&quot;)</pre><p>The full implementation is in the following colab notebook:</p><p><a href="https://colab.research.google.com/drive/1oDHjFk7m6WvoHcSCvXzDaR4mP6HBBLoA">Google Colab</a></p><p>Training diffusion transformers is very costly but if you have a good GPU, you may run it for a few days to get a good piece of MIDI generated, although using advanced GPUs is preferred considering the training time.</p><p>All right AI music enthusiasts! This blog illustrated VQVAE and diffusion transformers applied to the field of symbolic music generation. If you are interested in similar topic on MUSIC AI please follow my page and check my future blogs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=197b2115399a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Run Open-Sourced LLM Models Like DeepSeek-R1 on Your Local Machine for Free Using Ollama]]></title>
            <link>https://medium.com/@siamak.yousefi.1984_58741/run-light-llms-on-your-local-gpu-for-free-using-ollama-26a804908dd6?source=rss-024a4ac34f32------2</link>
            <guid isPermaLink="false">https://medium.com/p/26a804908dd6</guid>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ollama]]></category>
            <category><![CDATA[gen-ai-tools]]></category>
            <dc:creator><![CDATA[Siamak Yousefi ]]></dc:creator>
            <pubDate>Thu, 06 Feb 2025 05:29:25 GMT</pubDate>
            <atom:updated>2025-02-06T17:46:46.618Z</atom:updated>
            <content:encoded><![CDATA[<p>In this blog, I will talk about, OLLAMA which enables you to run open-source models like Llama, DeepSeek-R1, Mistral, Qwen, and other models on your local GPU or even CPU for free. You may use your MAC as well with no issues.</p><p>In today’s era, the AI and Large Language Models (LLMs) are an essential part of many people’s life. While cloud services like ChatGPT, Gemini, and Claude offer very good options, there are still good reasons to run models locally than using these closed-source web-based models.</p><h4>Why Run LLMs Locally?</h4><p>The reason that you don’t always want to use the web-based LLMs like ChatGPT, Gemini, or Claude can be summarized below:</p><ol><li><strong>Offline Accessibility:</strong></li></ol><p>You need internet connection which may not be guaranteed for every workplace or situation. Maybe you even don’t have permission to access the web in certain environments.</p><p>2. <strong>Data Privacy:</strong></p><p>You may not like to share certain information with these AI corporations due to privacy. This way your information is not exposed to third parties. This is very crucial for healthcare and mental health related applications where patients care about sharing their personal information online.</p><p>3. <strong>Corporate Compliance:</strong></p><p>Third, when you work for some companies, they may not allow using their machines to chat with platforms like OpenAI, Google, and DeepSeek and other sites for security and compliance.</p><p>However, by downloading these models and running them on your own machine locally you and your organization can actually avoid the above limitations.</p><p>The good news for those not too familiar with coding is that you can do it now easily using OLLAMA!</p><p>The first step is to download it from the Ollama website:</p><p><a href="http://www.ollama.com/">www.ollama.com</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*uxXpUUOsjQIWiiE3ZJZo9A.png" /></figure><h4>Model Selection and Hardware Requirement</h4><p>After downloading OLLAMA for your platform you may go to the model section and chose the model you want to download. Say for instance you want to run DeepSeek R1 with 7B parameter. On MAC, Linux, or WSL go to a terminal and type:</p><pre>ollama run deepseek-r1:7b</pre><p>In order to make good use of your GPU make sure how much virtual RAM (VRAM) your GPU has. For example, for running a 7B parameter like Mistral 7B, each parameter is in INT16 format and needs 2 bytes. So We need 14 GB of VRAM. Therefore, a GPU with at least 16GB is preferred. Here you can see the comparison of some GPUs for running Mistral 7B model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nY_zr1gKxXvdlXOG_BATrg.png" /><figcaption><a href="https://www.hardware-corner.net/guides/hardware-for-mistral-llm/#:~:text=Mistral%2C%20being%20a%207B%20model%2C%20requires%20a%20minimum,the%20RTX%203060%20with%20its%2012GB%20VRAM%20variant.">Choosing Hardware for Running Mistral LLM (7B) Locally</a></figcaption></figure><p>But with further quantization the model size can be reduced and the model can run on your machine efficiently. For instance if instead of INT16, INT8 is used, then we only need around 7GB of VRAM, so any 8GB GPU can be used with no issues. Many gaming laptops now have 8GB of VRAM. If you use an NVIDIA GPU, make sure CUDA is configured such that the models can make benefit from optimized calculations. On MAC M-series, the GPU doesn’t have a dedicated virtual RAM and the system RAM is used for loading the model. So, in this case, make sure there is some RAM space left for other tasks.</p><h4>Quantization: Reducing Model Size Further</h4><p>If the GPU VRAM you have is too small for the model, the good news is that OLLAMA has different quantized models that you can download and run, and things will be taken care of from there. For instance, to run Llama2 with 7b there are a few options:</p><pre>ollama run llama2:7b-q4_k_m  # 4-bit quantized<br>ollama run llama2:7b-q8_0    # 8-bit quantized<br>ollama run llama2:7b-f16     # 16-bit precision</pre><ul><li>q4_k_m format: 4bit quantized, ~4–5GB VRAM for 7B models</li><li>q8_0 format: 8-bit quantized ~8GB VRAM for 7B models</li><li>f16 format: 16 bit quantized ~14GB VRAM for 7B models</li></ul><p>The model will be downloaded and saved on your machine. Then a chat cursor will be shown where you can type your questions. Here is a sample question about the size of GPU for running a 7B parameter DeepSeek-R1 model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6nvjtxu-7Ui6S7YccnrMTg.png" /></figure><h4>Summary of Performance</h4><p>So, if you have a GPU like RTX4060 with 8GB of VRAM you should be able to run a 7B parameter LLM on your machine with good precision. The 7B models may not be as good as OpenAI-O1 or Claude-Sonnet 3.5 but they offer a lot of advantages like:</p><ul><li>Local processing without internet</li><li>No subscription fees</li><li>low cost (a laptop and electricity)</li><li>Complete privacy</li></ul><p>The trade-off between the above advantages and a small degradation in performance still makes these models desired for many personal and professional applications where privacy matters like AI health chatbots.</p><p>Thanks for tuning in!</p><p>Follow me for more blogs on AI!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=26a804908dd6" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>