<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Davide Ghilardi on Medium]]></title>
        <description><![CDATA[Stories by Davide Ghilardi on Medium]]></description>
        <link>https://medium.com/@ddxzzx?source=rss-2907a3374fa5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*6MZTEWMYH-iFLdd_PE3LFg.jpeg</url>
            <title>Stories by Davide Ghilardi on Medium</title>
            <link>https://medium.com/@ddxzzx?source=rss-2907a3374fa5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 14:50:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ddxzzx/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why and How to Achieve Longer Context Windows for LLMs]]></title>
            <link>https://medium.com/data-science/why-and-how-to-achieve-longer-context-windows-for-llms-5f76f8656ea9?source=rss-2907a3374fa5------2</link>
            <guid isPermaLink="false">https://medium.com/p/5f76f8656ea9</guid>
            <category><![CDATA[interpretability]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[positional-encoding]]></category>
            <dc:creator><![CDATA[Davide Ghilardi]]></dc:creator>
            <pubDate>Fri, 13 Oct 2023 10:41:05 GMT</pubDate>
            <atom:updated>2024-03-07T15:18:55.984Z</atom:updated>
            <content:encoded><![CDATA[<p>Language models (LLMs) have revolutionized the field of natural language processing (NLP) over the last few years, achieving state-of-the-art results on a wide range of tasks. However, a key challenge in developing and improving these models lies in extending the length of their context. This is very important since it determines how much information is available to the model when generating an output.</p><p>However, increasing the context window of a LLM isn’t so simple. In fact, it comes at the cost of increased computational complexity, since the attention matrix grows quadratically.</p><p>One solution could be training the model on a large amount of data on a relatively small window (e.g. 4K tokens) and then fine-tuning it on a bigger one (e.g. 64K tokens). This operation isn’t straightforward, because even though the context length doesn&#39;t impact the number of model’s weights, it does affect how <strong>positional information</strong> of tokens is encoded by those weights in the tokens’ dense representation.</p><p>This reduces the model’s capacity to adapt to longer context windows even after fine-tuning, resulting in poor performance and thus requiring new techniques to encode positional information correctly and dynamically between training and fine-tuning.</p><h3>Absolute positional encoding</h3><p>In transformers, the information about the position of each token is encoded before the token is fed to the attention heads. It’s a crucial step since transformers, differently from RNNs, don’t keep track of the tokens’ position.</p><p>The original transformer architecture [<a href="#d8de">1</a>] encodes positions as vectors of the same shape of tokens’ embeddings so that they can be <strong>added</strong> together. In particular, they used a combination of cos/sin waves with lengths increasing from low to higher order dimensions of the embedding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wkNxcQ-Pg2MWCkxdwQJ5RQ.png" /><figcaption>Image by author</figcaption></figure><p>This method allows an efficient unique dense representation of all the tokens’ positions. However, it doesn’t solve the challenge of extending context length since these representations can&#39;t efficiently encode relative positions of tokens.</p><p>To see why this happens, let&#39;s focus on a single couple of two consecutive dimensions. If we plot them on a chart, we can represent token and position embeddings as 2D vectors, as shown in the following figure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MGzUUs0zTf7Y3XzGPjsKtA.png" /><figcaption><em>Image by author</em></figcaption></figure><p>The figure above includes two charts: on the left, we have the embedding space, and on the right, there’s the key/query space. Embedding vectors are the black ones, positional vectors are the blue ones, and their sum is coloured in green.</p><p>To shift from embeddings to keys (K) and queries (Q), the transformer applies some linear transformations defined by the two matrices <strong>W</strong>q and <strong>W</strong>k. Thanks to the linearity property, the transformation can be applied separately between embeddings and positional vectors. In the KQ-space the attention is computed as the dot product between keys and queries, and in the figure, it’s represented by the yellow area between the two green vectors.</p><p>The relative distance between tokens is not directly accessible to the model since rotations mix up differently based on the original orientations of the tokens’ and positions’ embeddings, and on how they’re scaled in the transformations.</p><p>It’s also important to note that, in this case, the addition is applied before the transformation and not vice-versa since position embeddings have to be scaled by the linear layer.</p><h3>Relative positional encoding</h3><p>To efficiently encode relative position information among tokens other methods have been proposed. We will focus on <strong>RoPE </strong>[<a href="#d8de">2</a>], which stands for <strong>Ro</strong>tary <strong>P</strong>osition <strong>E</strong>mbedding, and for which extensions for longer context windows have been proposed.</p><p>The two main innovations of RoPE are:</p><ul><li>With RoPE, we can make the dot product between keys and queries embeddings sensitive only to the <strong>relative distance</strong> between them.</li><li>Positional embeddings are <strong>multiplied</strong> by the tokens’ embeddings without the need for a fixed-size look-up table.</li></ul><p>To achieve this goal, first, we map tokens’ embeddings from the real D-dimensional space to a complex D/2-dimensional one and apply the rotation in that space.</p><p>Again, let’s consider a single couple of two consecutive dimensions. If we plot them on the complex plane (setting the first on the real axis and the second on the imaginary one), we can represent token embeddings as complex vectors, as shown in the following figure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dHQ2OnMDuJO7oDUJfnAknw.jpeg" /><figcaption><em>Image by author</em></figcaption></figure><p>Here positional encodings acquire a new meaning, in fact, they become rotations directly applied to the vectors. This change allows them to both uniquely encode the position of the token, and be preserved in the linear transformation.</p><p>This last property is fundamental to incorporate the relative positions in the computation of attention. In fact, if we consider the attentions between the unrotated (in black) and rotated (in green) versions of <strong>k</strong>ₙ and <strong>q</strong>ₙ₊ₖ, represented respectively by the orange and yellow angles in the right chart, we can note something interesting:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/517/1*YIUq11zLVHA2OKPovlZKTQ.png" /><figcaption><em>Image by author</em></figcaption></figure><p>The attention between the rotated keys and queries differs from the unrotated version only by a factor proportional to the difference between their positions!</p><p>This new property allows models trained with RoPE to show <strong>rapid convergence</strong> and <strong>lower losses</strong> and lies at the heart of some popular techniques to extend the context window beyond the training set.</p><p>Until now we have always kept embedding dimensions fixed, but to see the whole framework, it’s also important to observe what happens along that direction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z9nBNSPssTeWCtfF7lxSdg.png" /><figcaption><em>Image by author</em></figcaption></figure><p>The RoPE formula defines the angle of rotation for the <em>d</em>-th dimension proportional to <em>exp(-d)</em> so, as we can see in the figure above, as <em>d </em>grows, the rotation applied to the embedding vector decreases exponentially (for fixed values of n and position). This feature of RoPE lets the model shift the type of information encoded in the embeddings from low-frequency (close tokens associations) to high-frequency (far tokens associations) by going from low to higher dimensions.</p><h3>RoPE extensions</h3><p>Once we have efficiently incorporated relative position information inside our model, the most straightforward way to increase the context window L of our LLM is by fine-tuning with <strong>position interpolation</strong> (<strong>PI</strong>) [<a href="#d8de">3</a>].</p><p>It is a simple technique that scales tokens&#39; position to fit the new context length. So, for example, if we decide to double it, all positions will be divided in half. In particular, for any context length L* &gt; L we want to achieve, we can define a scale factor <em>s</em> = L/ L* &lt; 1.</p><p>Although this technique has shown promising results by successfully extending the context of LLM with fine-tuning on a relatively small amount of tokens, it has its own drawbacks.</p><p>One of them is that it slightly decreases performance (e.g. perplexity increases) for short context sizes after fine-tuning on larger ones. This issue happens because by scaling by <em>s &lt; </em>1 the position of tokens (and so also their relative distances), we reduce the rotation applied to the vectors, causing the loss of high-frequency information. Thus, the model is less able to recognize small rotations and so to figure out the positional order of close-by tokens.</p><p>To solve this problem, we can apply a clever mechanism called <strong>NTK-aware </strong>[<a href="#d8de">4</a>] positional interpolation that instead of scaling every dimension of RoPE equally by <em>s</em>, spreads out the interpolation pressure across multiple dimensions by scaling high frequencies less and low frequencies more.</p><p>Other PI extensions exist such as the <strong>NTK-by-parts </strong>[<a href="#d8de">5</a>] and <strong>dynamic NTK </strong>[<a href="#d8de">6</a>]<strong> </strong>methods. The first imposes two thresholds to limit the scaling above and below certain dimensions; the second dynamically adjusts <em>s</em> during inference.</p><p>Finally, since it was observed that as the number of tokens increases, the attention softmax distribution becomes more and more “spikier” (the average entropy of attention softmax decreases), <strong>YaRN</strong> [<a href="#d8de">7</a>] (<strong>Y</strong>et <strong>a</strong>nother <strong>R</strong>oPE extensio<strong>N</strong> method) is a method that inverts this process by multiplying the attention matrix by a temperature factor <em>t </em>before the application of the softmax.</p><p>Here’s a look at what these methods do from both position (numbers of rotations) and dimension (degree per single rotation) perspectives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5Y70irOVILHKy_q80pjQLQ.png" /><figcaption><em>Image by author</em></figcaption></figure><h4>Other methods</h4><p>Finally, as we said before, other context extension methods exist, here’s a brief description of the most popular ones and how they operate:</p><ul><li>Alibi [<a href="#d8de">8</a>]: It’s another method for positional encoding which penalizes the attention value that that query can assign to the key depending on how far away the key and query are.</li><li>XPos [<a href="#d8de">9</a>]: Again another positional encoding method that generalizes RoPE to include a scaling factor.</li></ul><h3>References</h3><ol><li>Vaswani et al, 2017. Attention Is All You Need. <a href="https://arxiv.org/abs/1706.03762">link</a></li><li>Su et al, 2022. RoFormer: Enhanced transformer with rotary position embedding. <a href="https://arxiv.org/abs/2104.09864">link</a></li><li>Chen et al, 2023. Extending context window of large language models via positional interpolation. <a href="https://arxiv.org/abs/2306.15595">link</a></li><li>bloc97, 2023. NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation. <a href="https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have/">link</a></li><li>bloc97, 2023. Add NTK-Aware interpolation “by parts” correction. <a href="https://github.com/jquesnelle/yarn/pull/1">link</a></li><li>emozilla, 2023. Dynamically Scaled RoPE further increases performance of long context LLaMA with zero fine-tuning. <a href="https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamically_scaled_rope_further_increases/">link</a></li><li>Peng et al, 2023. YaRN: Efficient Context Window Extension of Large Language Models. <a href="https://arxiv.org/abs/2309.00071">link</a></li><li>Press et al, 2022. Train Short, Test Long: Attention with linear biases enables input length extrapolation. <a href="https://openreview.net/forum?id=R8sQPpGCv0">link</a></li><li>Sun et al, 2022. A Length-Extrapolatable Transformer. <a href="https://arxiv.org/abs/2212.10554">link</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5f76f8656ea9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/why-and-how-to-achieve-longer-context-windows-for-llms-5f76f8656ea9">Why and How to Achieve Longer Context Windows for LLMs</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Image classification with Bayesian Flow Networks in Python]]></title>
            <link>https://medium.com/@ddxzzx/image-classification-with-bayesian-flow-networks-in-python-d83342e1374?source=rss-2907a3374fa5------2</link>
            <guid isPermaLink="false">https://medium.com/p/d83342e1374</guid>
            <category><![CDATA[bayesian-statistics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[diffusion-models]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Davide Ghilardi]]></dc:creator>
            <pubDate>Fri, 08 Sep 2023 09:01:10 GMT</pubDate>
            <atom:updated>2023-09-08T15:39:41.733Z</atom:updated>
            <content:encoded><![CDATA[<p>Generative models are a powerful class of machine learning models that can learn to generate new, realistic samples from a given distribution. Unlike discriminative models, designed to recognize and classify examples, generative ones can learn to create new data that fits the patterns and structures of the training data. Generative models, especially autoregressive and diffusion ones, have been gaining popularity in the last period also thanks to the rapid growth in computing power, which allows training bigger networks with hundreds of billions of parameters.</p><p>Even though the spotlight is directed to these kinds of models, other generative architectures exist, such as generative adversarial networks (GANs) and variational autoencoders (VAEs). Recently, Alex Graves and his team at NNAISENSE introduced a new class of generative models called <strong>Bayesian Flow Networks </strong>(<strong>BFNs</strong>)<strong> </strong>that with a simple yet powerful structure, can take an important role in the scene.</p><h3>1. Introduction</h3><p>The expressive power of the most popular generative networks, including autoregressive models, deep VAEs, and diffusion models is that they encode the joint distribution by brokening it down into a series of steps, thereby saving any effort to explicitly define all the interactions among so many variables (avoiding the “curse of dimensionality”).</p><p>In the paper, the authors describe this process as an exchange of messages between a sender, <strong>Alice</strong>, and a receiver, <strong>Bob</strong>, who wants to get the message in as few bits as possible.</p><p>For example, in autoregressive models, Alice sends one token at each step, and Bob uses that information to predict the next one and adjust his prior. The transmission cost (the loss function), which has to be minimized, is the negative log-probability under the prior (updated at each step) of the whole sentence.</p><p>On the other hand, in diffusion models, Alice sends a noisy version of the data and Bob tries to reconstruct the original message. The loss is calculated as the distance between Alice’s and Bob’s distributions (measured in terms of KL divergence).</p><p>While autoregressive networks are the SOTA in language modelling, they are less effective in other domains such as image generation. On the other hand, diffusion models prevail in the latter domain but struggle in processing text.</p><p>The main reason for that can be traced back to the nature of the data. In fact, autoregressive works better with textual data because of its discrete form and because it has an implicit natural order. Diffusion instead, is better suited for continuous data since noise can’t be discrete. Returning to the transmission metaphor, if the message is textual, then Bob will receive a totally garbled text, with every symbol being randomly altered or left unchanged by each of Alice’s messages.</p><p>Bayesian Flow Networks tries exactly to fill this gap by operating on the parameters of a data distribution, rather than on the data itself. This ensures continuity and differentiability, even when the data is discrete.</p><h3>2. Ok, but how do they work?</h3><p>BFNs can also be represented as a transmission task, as shown in the following figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xNNRNxVWBSgckXm-N7oTFg.png" /><figcaption>The modelling process of a Bayesian Flow Network.</figcaption></figure><p>Here, Bob has an <strong><em>input distribution</em></strong> which is his prior, and at each transmission step, he sends the parameters of that distribution to a neural network which returns the parameters of a second distribution called <strong><em>output distribution</em></strong>.</p><p>Then, Alice creates a <strong><em>sender distribution</em></strong> by adding noise to the data according to a predefined schedule (that Bob knows), and Bob creates a <strong><em>receiver distribution</em></strong><em> </em>as the expected value of the sender distribution on the values that the data can take based on the output distribution.</p><p>Alice then picks a sample from the sender distribution and sends it to Bob at a cost equal to the KL divergence from receiver to sender. Bob then uses the sample to update his input distribution, following the <strong>Bayes’ Rule</strong>. This happens <em>n </em>times for each variable until Bob can predict the data accurately enough that Alice can send it to him without any noise.</p><p>It’s important here to highlight the difference between input and output distributions. While the first contains information about each variable independently, and thus can’t exploit contextual information, the output distribution, on the other hand, is produced by a neural network that has access to the parameters of all the variables’ distributions.</p><p>For example, if we want to process an image, the input distribution will have access only to each pixel independently, while the output one, which will be probably produced by a ConvNet, will leverage information coming from other parts of the image.</p><h3>3. Python implementation for MNIST</h3><p>The following example, uploaded on <a href="https://github.com/Algomancer/Bayesian-Flow-Networks/blob/main/BFN_MNIST.ipynb">GitHub</a>, shows how BFNs can be implemented in Python for image generation based on the discretized MNIST dataset.</p><h4>3.1 Data loading</h4><p>The binarized MNIST dataset can be created from the MNIST dataset of handwritten images by treating the grayscale pixel intensities as Bernoulli probabilities and sampling a particular binarization which is held fixed during training.</p><p>In this example, a variant of the same benchmark, called dynamically binarized MNIST was adopted, with a new binarization sampled from the probabilities for every training batch.</p><h4>3.2 Building the neural network</h4><p>The backbone of this BFN is a simplified version of the <a href="https://medium.com/analytics-vidhya/what-is-unet-157314c87634">UNet</a> architecture that is shown in the following figure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LIi08LmK22TXJ2OlBns85Q.png" /><figcaption>The UNet architecture used for the BFN.</figcaption></figure><p>The model has two residual blocks going from 64 to 256 feature maps in the inner layer for a total of ~2.8M parameters.</p><p>Once the UNet has been defined, we can start implementing the BFN by writing the two main functions: <strong>process()</strong> and <strong>sample()</strong>.</p><p>The first, which is used for training, implies five steps:</p><ol><li>Sample the timestep <em>t </em>from a uniform distribution in the unit range;</li><li>Compute β, the accuracy schedule, based on <em>t;</em></li><li>Sample <strong>y</strong><em> </em>from a Gaussian distribution centred on the image and with standard deviation inverse proportional to <em>t;</em></li><li>Compute <strong>θ</strong>, the input distribution’s parameters, by taking the softmax of <strong>y</strong><em> </em>(rescaling it in the unit interval);</li><li>Send <strong>θ </strong>and <em>t </em>to the UNet to obtain the output distribution’s parameters.</li></ol><p>Once you have the output parameters you can compute the loss by exploiting the following closed form:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*KqNO0GznkslgMYYMiP74Bw.png" /></figure><p>Below you can find the code for the implementation of the process function. In our example K=2 since the digits are binarized.</p><pre>def process(self, x, t=None, training=True):<br><br>        # Step 1: Sample t from U(0, 1)<br>        if t is None:<br>          t = torch.rand((x.size(0),), device=x.device, dtype=torch.float32)<br>        else:<br>          t = torch.tensor(t, device=x.device, dtype=torch.float32)[None]<br><br>        # Step 2: Calculate Beta<br>        beta = self.beta * (t ** 2)  # (B,)<br><br>        # Step 3: Sample y from N(beta * (K * one_hot(X))<br>        one_hot_x = F.one_hot(x.permute(0, 2, 3, 1).to(torch.int64), num_classes=self.K).float().squeeze()  # (B, D, D, K)<br>        mean = beta[:, None, None, None] * (self.K * one_hot_x - 1)<br>        std = (beta * self.K)[:, None, None, None].sqrt()<br>        eps = torch.randn_like(mean)<br>        y = mean + std * eps<br><br>        # Step 4: Compute the Theta<br>        theta = F.softmax(y, dim=-1)<br><br>        # Step 5: Calculate the output distribution<br>        p_0 = self.discrete_output_distribution(theta, t)  # (B, D, D, K)<br><br>        e_x = one_hot_x<br>        e_hat = p_0  # (B, D, D, K)<br>        L_infinity = self.K * self.beta * t[:, None, None, None] * ((e_x - e_hat) ** 2)<br>        if training:<br>          return L_infinity.mean()<br>        else:<br>           k = torch.distributions.Categorical(probs=p_0).sample()<br>           return L_infinity.mean(), y, k, t</pre><p>The second main function we have to define is sampling; it takes as input the number of sampling steps and the size of the batch of images we want to generate. It can divided into the following steps:</p><ol><li>Initialize <strong>θ </strong>as 1/K, so as the parameter of a uniform prior;</li><li>Start iterating through all sampling steps by defining <em>t</em> = (step-1)/n_step;</li><li>Sample <strong>k</strong>, the generated batch, from the output distribution obtained by passing <strong>θ </strong>and <em>t</em> to the network</li><li>Sample <strong>y</strong> from a Gaussian distribution centred on <strong>k</strong> and with standard deviation inverse proportional to the sampling step;</li><li>Update <strong>θ </strong>by taking the softmax of <strong>y </strong>• log(<strong>θ</strong>)</li><li>When the loop is finished, sample and return <strong>k</strong> from the output distribution obtained by passing <strong>θ </strong>and <em>t</em> to the network.</li></ol><p>Below you can find the code for the implementation of the sample function:</p><pre>@torch.inference_mode()<br>    def sample(self, batch_size=128, nb_steps=10, ema=None, device=&#39;cpu&#39;):<br>        self.eval()<br><br>        # Step 1: Initializing theta<br>        theta = torch.ones((batch_size, self.D, self.D, self.K), device=device) / self.K<br>        <br>        # Step 2: Iterate through the sampling steps<br>        for i in tqdm(range(1, nb_steps+1)):<br>            t = (i-1) / nb_steps<br>            t = t * torch.ones((theta.shape[0]), device=theta.device, dtype=theta.dtype)<br>            <br>            # Step 3: Sample k<br>            k_probs = self.discrete_output_distribution(theta, t, ema=ema)  # (B, D, D, K)<br>            k = torch.distributions.Categorical(probs=k_probs).sample()  # (B, D, D)<br>            <br>            # Step 4: Sample y<br>            alpha = self.beta * (2 * i - 1) / (nb_steps ** 2)<br><br>            e_k = F.one_hot(k, num_classes=self.K).float()  # (B, D, D, K)<br>            mean = alpha * (self.K * e_k - 1)<br>            var = (alpha * self.K)<br>            std = torch.full_like(mean, fill_value=var).sqrt()<br>            eps = torch.randn_like(e_k)<br><br>            y = mean + std * eps  # (B, D, D, K)<br><br>            # Step 5: Theta update<br>            theta_prime = torch.exp(y) * theta<br>            theta = theta_prime / theta_prime.sum(-1, keepdim=True)<br><br>        # Step 6: Final sampling<br>        k_probs_final = self.discrete_output_distribution(theta, torch.ones_like(t), ema=ema)<br>        k_final = torch.distributions.Categorical(probs=k_probs_final).sample()<br><br>        return k_final</pre><h4>3.3 Training the BFN</h4><p>To train the BFN we used <a href="https://cy96.medium.com/adamw-and-adabelief-optimizers-based-on-and-better-than-adam-3682c12bffc7">AdamW</a> optimizer with the hyper-parameters shown in the paper. We trained the model for 100 epochs and applied an exponential moving average (EMA) of the parameters for the sample generation.</p><pre>from torch.optim import AdamW<br>from tqdm.auto import tqdm<br><br># Model definition (with EMA) <br>bfn = BayesianFlowNetwork2D(UNet(2, 1, dropout=0.5), 28, 2)<br>ema = torch_ema.ExponentialMovingAverage(bfn.net.parameters(), decay=0.9999)<br><br>bfn.cuda()<br>ema.to(device)<br><br># Optimizer<br>optim = AdamW(bfn.parameters(), lr=1e-4, betas=(0.9, 0.98), weight_decay=0.01)<br><br># TRAINING LOOP<br><br>losses = []<br>epochs = 100<br><br>for i in range(epochs):<br>  for X, _ in tqdm(train_loader, desc=f&quot;Epoch {i+1}/{epochs}&quot;):<br>      optim.zero_grad()<br><br>      loss = bfn.process(X.to(device))<br>      loss.backward()<br><br>      optim.step() <br>      ema.update()<br><br>      losses.append(loss.item())</pre><h4>3.4 Reconstruction and sampling</h4><p>Once the model has been trained, we can use it to generate and reconstruct images of digits.</p><p>For sampling, we can simply call the function we defined before, as shown in the following code:</p><pre>x_hat = bfn.sample(device=&#39;cuda&#39;, nb_steps=1000, batch_size=16, ema=ema)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*6yAX_WPApOh6EFRaA4Wl7A.png" /><figcaption>Images sampled from the output distribution at 1000 steps.</figcaption></figure><p>We can also show the reconstruction process that takes place at each timestep. To do that, we simply call the process function and pass the timestep to it.</p><p>In the example below, we show the noisy sample from the sender distribution (above) and the sample extracted from the output distribution (below) for each timestep from 0 to 2<em>t</em>/3<em>.</em></p><pre>x_0 = get_sample()[0][0]<br><br>Ys = []<br>Ks = []<br>Ts = []<br><br># We set t going from 0 to 2/3<br>for t in np.linspace(0.001, 0.667, 20):<br>  l, y, k, t_ = bfn.process(x_0[None, :, :, None].to(torch.int64).to(device), training=False, t=t)<br><br>  Ys.append(y.permute(0, 2, 1, 3).cpu().numpy()[0, :, :, 1])<br>  Ks.append(k.permute(0, 2, 1).cpu().numpy()[0])<br>  Ts.append(t)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0FBjltOQCRi18VojCpULtw.png" /><figcaption>The reconstruction process at different timesteps.</figcaption></figure><h4>5. References</h4><ol><li><a href="https://arxiv.org/abs/2308.07037">Bayesian Flow Networks</a>, Alex Graves and Rupesh Kumar Srivastava and Timothy Atkinson and Faustino Gomez, 2023;</li><li><a href="https://arxiv.org/abs/1505.04597v1">U-Net: Convolutional Networks for Biomedical Image Segmentation</a>, Olaf Ronneberger and Philipp Fischer and Thomas Brox, 2015;</li><li><a href="https://arxiv.org/abs/1711.05101v3">Decoupled Weight Decay Regularization</a>, Ilya Loshchilov and Frank Hutter, 2019.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d83342e1374" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>