update week 15

Author: mhjensen

doc/pub/week15/html/week15-bs.html

@@ -42,13 +42,23 @@
                'plans-for-the-week-of-may-5-9-2025'),
               ('Readings', 2, None, 'readings'),
               ('Diffusion models, basics', 2, None, 'diffusion-models-basics'),
+              ('Why diffusion models?', 2, None, 'why-diffusion-models'),
+              ('What are diffusion models?',
+               2,
+               None,
+               'what-are-diffusion-models'),
               ('Problems with probabilistic models',
                2,
                None,
                'problems-with-probabilistic-models'),
               ('Diffusion models', 2, None, 'diffusion-models'),
               ('Original idea', 2, None, 'original-idea'),
               ('Diffusion learning', 2, None, 'diffusion-learning'),
+              ('How diffusion models work',
+               2,
+               None,
+               'how-diffusion-models-work'),
+              ('Data preprocessing', 2, None, 'data-preprocessing'),
               ('Mathematics of diffusion models',
                2,
                None,
@@ -101,6 +111,21 @@
                None,
                'diffusion-models-part-2-from-url-https-arxiv-org-abs-2208-11970'),
               ('Optimization cost', 2, None, 'optimization-cost'),
+              ('Image quality', 2, None, 'image-quality'),
+              ('Training stability', 2, None, 'training-stability'),
+              ('Input types', 2, None, 'input-types'),
+              ('Denoising diffusion probabilistic models (DDPMs)',
+               2,
+               None,
+               'denoising-diffusion-probabilistic-models-ddpms'),
+              ('Techniques for speeding up diffusion models',
+               2,
+               None,
+               'techniques-for-speeding-up-diffusion-models'),
+              ('Applications of diffusion models',
+               2,
+               None,
+               'applications-of-diffusion-models'),
               ('PyTorch implementation of a Denoising Diffusion Probabilistic '
                'Model (DDPM) trained on the MNIST dataset',
                2,
@@ -165,10 +190,14 @@
      <!-- navigation toc: --> <li><a href="#plans-for-the-week-of-may-5-9-2025" style="font-size: 80%;">Plans for the week of May 5-9, 2025</a></li>
      <!-- navigation toc: --> <li><a href="#readings" style="font-size: 80%;">Readings</a></li>
      <!-- navigation toc: --> <li><a href="#diffusion-models-basics" style="font-size: 80%;">Diffusion models, basics</a></li>
+     <!-- navigation toc: --> <li><a href="#why-diffusion-models" style="font-size: 80%;">Why diffusion models?</a></li>
+     <!-- navigation toc: --> <li><a href="#what-are-diffusion-models" style="font-size: 80%;">What are diffusion models?</a></li>
      <!-- navigation toc: --> <li><a href="#problems-with-probabilistic-models" style="font-size: 80%;">Problems with probabilistic models</a></li>
      <!-- navigation toc: --> <li><a href="#diffusion-models" style="font-size: 80%;">Diffusion models</a></li>
      <!-- navigation toc: --> <li><a href="#original-idea" style="font-size: 80%;">Original idea</a></li>
      <!-- navigation toc: --> <li><a href="#diffusion-learning" style="font-size: 80%;">Diffusion learning</a></li>
+     <!-- navigation toc: --> <li><a href="#how-diffusion-models-work" style="font-size: 80%;">How diffusion models work</a></li>
+     <!-- navigation toc: --> <li><a href="#data-preprocessing" style="font-size: 80%;">Data preprocessing</a></li>
      <!-- navigation toc: --> <li><a href="#mathematics-of-diffusion-models" style="font-size: 80%;">Mathematics of diffusion models</a></li>
      <!-- navigation toc: --> <li><a href="#chains-of-vaes" style="font-size: 80%;">Chains of VAEs</a></li>
      <!-- navigation toc: --> <li><a href="#mathematical-representation" style="font-size: 80%;">Mathematical representation</a></li>
@@ -189,6 +218,12 @@
      <!-- navigation toc: --> <li><a href="#the-last-term" style="font-size: 80%;">The last term</a></li>
      <!-- navigation toc: --> <li><a href="#diffusion-models-part-2-from-url-https-arxiv-org-abs-2208-11970" style="font-size: 80%;">Diffusion models, part 2, from URL:"https://arxiv.org/abs/2208.11970"</a></li>
      <!-- navigation toc: --> <li><a href="#optimization-cost" style="font-size: 80%;">Optimization cost</a></li>
+     <!-- navigation toc: --> <li><a href="#image-quality" style="font-size: 80%;">Image quality</a></li>
+     <!-- navigation toc: --> <li><a href="#training-stability" style="font-size: 80%;">Training stability</a></li>
+     <!-- navigation toc: --> <li><a href="#input-types" style="font-size: 80%;">Input types</a></li>
+     <!-- navigation toc: --> <li><a href="#denoising-diffusion-probabilistic-models-ddpms" style="font-size: 80%;">Denoising diffusion probabilistic models (DDPMs)</a></li>
+     <!-- navigation toc: --> <li><a href="#techniques-for-speeding-up-diffusion-models" style="font-size: 80%;">Techniques for speeding up diffusion models</a></li>
+     <!-- navigation toc: --> <li><a href="#applications-of-diffusion-models" style="font-size: 80%;">Applications of diffusion models</a></li>
      <!-- navigation toc: --> <li><a href="#pytorch-implementation-of-a-denoising-diffusion-probabilistic-model-ddpm-trained-on-the-mnist-dataset" style="font-size: 80%;">PyTorch implementation of a Denoising Diffusion Probabilistic Model (DDPM) trained on the MNIST dataset</a></li>
      <!-- navigation toc: --> <li><a href="#problem-with-diffusion-models" style="font-size: 80%;">Problem with diffusion models</a></li>
      <!-- navigation toc: --> <li><a href="#imports-and-utilities" style="font-size: 80%;">Imports and Utilities</a></li>
@@ -255,6 +290,7 @@ <h2 id="readings" class="anchor">Readings </h2>
 <li> A central paper is the one by Sohl-Dickstein et al, Deep Unsupervised Learning using Nonequilibrium Thermodynamics, <a href="https://arxiv.org/abs/1503.03585" target="_self"><tt>https://arxiv.org/abs/1503.03585</tt></a></li>
 <li> Calvin Luo at <a href="https://arxiv.org/abs/2208.11970" target="_self"><tt>https://arxiv.org/abs/2208.11970</tt></a></li>
 <li> See also Diederik P. Kingma, Tim Salimans, Ben Poole, Jonathan Ho, Variational Diffusion Models, <a href="https://arxiv.org/abs/2107.00630" target="_self"><tt>https://arxiv.org/abs/2107.00630</tt></a></li>
+<li> See also David Foster <em>Generative Deep Learning</em>, chapter 8 on diffusion models.</li>
 </ol>
 </div>
 </div>
@@ -271,6 +307,40 @@ <h2 id="diffusion-models-basics" class="anchor">Diffusion models, basics </h2>
 variable has high dimensionality (same as the original data).
 </p>
 
+<!-- !split -->
+<h2 id="why-diffusion-models" class="anchor">Why diffusion models? </h2>
+
+<p>Diffusion models are prominent in generating high-quality images,
+video, sound, etc. They are named for their similarity to the natural
+diffusion process in physics, which describes how molecules move from
+high-concentration to low-concentration areas. In the context of
+machine learning, diffusion models generate new data by reversing a
+diffusion process, that is information loss due to noise
+intervention. The main idea here is to add random noise to data and
+then undo the process to get the original data distribution from the
+noisy data.
+</p>
+
+<p>The famous DALL-E 2, Midjourney, and open-source Stable Diffusion that
+create realistic images based on the user's text input are all
+examples of diffusion models.
+</p>
+
+<!-- !split -->
+<h2 id="what-are-diffusion-models" class="anchor">What are diffusion models? </h2>
+
+<p>Diffusion models are advanced machine learning algorithms that
+uniquely generate high-quality data by progressively adding noise to a
+dataset and then learning to reverse this process. This innovative
+approach enables them to create remarkably accurate and detailed
+outputs, from lifelike images to coherent text sequences. Central to
+their function is the concept of gradually degrading data quality,
+only to reconstruct it to its original form or transform it into
+something new. This technique enhances the fidelity of generated data
+and offers new possibilities in areas like medical imaging, autonomous
+vehicles, and personalized AI assistants.
+</p>
+
 <!-- !split -->
 <h2 id="problems-with-probabilistic-models" class="anchor">Problems with probabilistic models </h2>
 
@@ -326,6 +396,32 @@ <h2 id="diffusion-learning" class="anchor">Diffusion learning </h2>
 of arbitrary form.
 </p>
 
+<!-- !split -->
+<h2 id="how-diffusion-models-work" class="anchor">How diffusion models work </h2>
+
+<p>Diffusion models work in a dual-phase mechanism: They first train a
+neural network to introduce noise into the dataset(a staple in the
+forward diffusion process) and then methodically reverse this
+process. 
+</p>
+
+<!-- !split -->
+<h2 id="data-preprocessing" class="anchor">Data preprocessing </h2>
+
+<p>Before the diffusion process begins, data needs to be appropriately
+formatted for model training. This process involves data cleaning to
+remove outliers, data normalization to scale features consistently,
+and data augmentation to increase dataset diversity, especially in the
+case of image data. Standardization is also applied to achieve normal
+data distribution, which is important for handling noisy image
+data. Different data types, such as text or images, may require
+specific preprocessing steps, like addressing class-imbalance
+issues. Well-executed data processing ensures high-quality training
+data and contributes to the model's ability to learn meaningful
+patterns and generate high-quality images (or other data types) during
+inference.
+</p>
+
 <!-- !split -->
 <h2 id="mathematics-of-diffusion-models" class="anchor">Mathematics of diffusion models </h2>
 
@@ -606,6 +702,76 @@ <h2 id="optimization-cost" class="anchor">Optimization cost </h2>
 value may have high variance for large \( T \) values.
 </p>
 
+<!-- !split -->
+<h2 id="image-quality" class="anchor">Image quality </h2>
+
+<p>An advantage of diffusion models over for example VAEs (and also GANs
+to be discussed next time) is the ease of training with simple and
+efficient loss functions and their ability to generate highly
+realistic images. They excel at closely matching the distribution of
+real images, outperforming GANs in this aspect. This proficiency is
+due to the distinct mechanisms in diffusion models, allowing for more
+precise replication of real-world imagery.
+</p>
+
+<!-- !split -->
+<h2 id="training-stability" class="anchor">Training stability </h2>
+
+<p>Regarding training stability, generative diffusion models have an edge
+over GANs. GANs often struggle with <em>mode collapse</em>, which is  a limitation
+where they produce a limited output variety. Diffusion models
+effectively avoid this issue through their gradual data smoothing
+process, leading to a more diverse range of generated images.
+</p>
+
+<!-- !split -->
+<h2 id="input-types" class="anchor">Input types </h2>
+
+<p>It is also important to mention that diffusion models handle various
+input types. They perform diverse generative tasks like text-to-image
+synthesis, layout-to-image generation, inpainting, and
+super-resolution tasks.
+</p>
+
+<!-- !split -->
+<h2 id="denoising-diffusion-probabilistic-models-ddpms" class="anchor">Denoising diffusion probabilistic models (DDPMs) </h2>
+
+<p>Denoising diffusion probabilistic models (DDPMs) are a specific type
+of diffusion model that focuses on probabilistically removing noise
+from data. During training, they learn how noise is added to data over
+time and how to reverse this process to recover the original
+data. This involves using probabilities to make educated guesses about
+what the data looked like before noise was added. This approach is
+essential for the model's capability to accurately reconstruct data,
+ensuring the outputs aren&#8217;t just noise-free but also closely resemble
+the original data.
+</p>
+
+<!-- !split -->
+<h2 id="techniques-for-speeding-up-diffusion-models" class="anchor">Techniques for speeding up diffusion models </h2>
+
+<p>Generating a sample from DDPM using the reverse diffusion process is
+quite slow because it involves many steps, possibly up to a
+thousand. For instance, according to Song et al. (2020), it takes
+about 20 hours to generate 50,000 small images with a DDPM, while a
+GAN can create the same amount in less than a minute using an Nvidia
+2080 Ti GPU.
+</p>
+
+<p>There is an alternative method called Denoising Diffusion Implicit
+Model (DDIM) that stands out for its efficiency and quality. Unlike
+traditional models, DDIM needs fewer steps to create clear images from
+noisy data.
+</p>
+
+<!-- !split -->
+<h2 id="applications-of-diffusion-models" class="anchor">Applications of diffusion models </h2>
+
+<p>There are very diverse applications of diffusion models, one of the most exciting being digital art creation.
+The document at <a href="https://www.superannotate.com/blog/diffusion-models#:~:text=A%20primary%20advantage%20of%20diffusion,to%20generate%20highly%20realistic%20images" target="_self"><tt>https://www.superannotate.com/blog/diffusion-models#:~:text=A%20primary%20advantage%20of%20diffusion,to%20generate%20highly%20realistic%20images</tt></a> gives many nice examples of applications.
+.
+</p>
+
 <!-- !split -->
 <h2 id="pytorch-implementation-of-a-denoising-diffusion-probabilistic-model-ddpm-trained-on-the-mnist-dataset" class="anchor">PyTorch implementation of a Denoising Diffusion Probabilistic Model (DDPM) trained on the MNIST dataset </h2>