What is DDPM/DDIM?

In this post, I will unfold the DDPM and DDIM through the lens of the SDE-based score model that discussed previous post. To begin, I would like to clarify that diffusion based models are essentially approximating z ~ p(z)(where p(z) is Gaussian) to look like x ~ p(x)(where p(x) is unknow data distribution). You will account formula, I want to emphasize that mathematics underlying DDPM and DDIM is not a magic, and it is not that difficult as it looks like. It will be sufficient some reasonable mathematical tools to grasp the derivations below.

This post consists of two parts: an explanation of the model structure and code-based experiments.

Part 1: The structure of Models

DDPM: Discrete VP-SDE as a Markovian Path

The essence of DDPM lies in its mechanism for reconstructing a clean image x from a noisy latent z. It defines the forward process as a discrete-time Markov chain, which effectively serves as a collector which gathering the necessary ingredients at each step. By ensuring the reversibility of this Gaussian process, the reverse process utilizes these gathered ingredients to reshape z.

DDIM:Discrete PF-ODE

On the other hand, Although DDIM shares the same objective as DDPM, but it takes different trajectory from DDPM. DDIM is liberated from the Markovian constraint by redefining the inference process. Instead of soley depending on the immediate previous step as Markov, it uses model’s prediction of estimated the original x at every iteration. This allows the model to establish a deterministic trajectory guaranteed by the Probability Flow ODE. While DDPM follows a stochastic trail, DDIM constructs a refined highway that follows the same probability flow.

Now, I will dive into how DDPM constructs its on forward and reverese processes, followed by an exploration of how DDIM makes its own way while following the same probability flow as DDPM.

VP-SDE: The First Building Block

In the paper “SCORE-BASED GENERATIVE MODELING THROUGH STOCHASTIC DIFFERENTIAL EQUATIONS”, VP-SDE is identified as the continuous version of DDPM. Based on this connection, I’m going to derive DDPM’s forward process formula by discretizing the VP-SDE. In this section, It will show you how the VP-SDE looks like below.

And from this, how we get below.

Conditions of VP-SDE

1. Linearity of Drift Term: (f(x,t) = f(t)x)
2. Variance Preservation: Each step maintains the same variance
3. Zero-mean Drift

Before we start, keep thoese properties in mind. We are going to use these properties to derive the formula

VP-SDE(1)

Let’s start form Linear drift SDE.

f(x,t)=f(t)x (for linearity of Drift term)

To assess the evolution of the variance, we define H(x) = x² and apply ito’s Lemma.(Var(x) = E[x²]-E[x]². Here, E[x] = 0, so the Var(x)=E[x]²)

Then:

I applied f(t) = f, g(t) = g for length

So we get:

Next we are going to take Expatation in both side.

Before we move on, let’s just accept one mathematical shortcut: dE[x²]=E[dx²].

By the computational Formula for Variance and properties of VP-SDE, we can know Var(x) = E[x²] = 1

The computatioal Formula for Variance

So we get the relation between drift coefficenct and diffusion coefficent.

relation of drift and diffusion term.

We parametrize the diffusion coefficient using βt to control the noise schedule. Then we get:

VP-SDE(2)

Next, we have to discretize VP-SDE with Euler-Maruyama(Here, EM for short) to make it computable. You can think of this entire process as the Euler-Maruyama discretization. It’s simply the way we turn a continuous SDE into a discrete formula that a computer can actually calculate.

Let’s set initial configuration to discretize by EM.

Substitute these into our VP-SDE above:

Here we redefine:

It doesn’t mean ∆t is 1 or it can redefine like this anytime by mathmatical trick. It is just “redefine”

Then we get:

The variance of this discretized form is not 1. If we caculate Var(xt+∆t), we can find out it is 1+beta²/4(Didn’t preserved as 1) so here, we use Taylor Expaion:

And, let’s relabeling the indices. We align the equation with the standard discrete-time equation(xt+∆t,xt →xt, xt-1). Then we get:

You can use Var(ax + by) = a²Var(x)+b²Var(y) to compute variance.

Markov property: The Second Building Block.

Now, we need to know markov property to get things straight. “The future state depends on only on the present state, independent of the past. Once the present is known, the future and the past become independent.”. The formula which has markov property isn’t built that way because it has the Markov property. Rather, we can say it has the Markov property because the formual is constructed to depend only immediate previous step.

Transition Kernel

We formalize the state update rule as a transition kernel q(xt|xt-1) to describe the stochastic evoultion of x. Simply, once we discretized VP-SDE, the existance of probability distribution which describe the rule of transition from xt-1 to xt is reasonable.

∊～N(0,I)

As∊follows standard normal distribution where μ:0, σ:I. So, the transition kernel can be formulated as follows:

Then, the transition kernel is defined as a Gaussian distribution.

Markov Chain

The Markov chain is a sequence of transitions where the joint distribution devides into a product of kernels. (Like chain rule of differentiation).

Gaussianity Preserved: The third Building Block

The Gaussianity is preserved throughout the Markov chain because the underlying SDE is driven by Brownian motion within a linear framework.(No matter whay you multiply or add, the Gaussian form is never chaneged. And this property makes everything easier)

Thanks to this Gaussianity, we can construct entire process both forward and reverse without complex computations.

Accumulated form of forward process: The 4th Building Block

Now, we can consruct 4th building block by our previous building block.

Reparameterize

Let’s set new parameter αₜ, and its cumulative product form αₜ(bar)

Then, forward formula can be written:

Recursive Substitution

Let’s start from plug x1 into x2

Subsituting x1 into x2:

Gaussian Sum Property

For two independent Gaussian variables X~N(0,sigma1²), Y~N(0,sigma2²), their sum follows:

Applying this property to the noise terms in x2:

Final General Form

By using the cumulative product form, we can generalize this for any time step t:

This Accumulated Forward form is a direct consequence of the Markov property. It enables us to sample xt at any random time step t during training without sequential iterations.

Get BLED’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

This is everything for forward process. Let’s move on reverse process for now.

Reverse Process

To construct reverse process, we have to know q(xt-1|xt)(reverse transition kernel).

By Bayes’ Rule

In this form, we know q(xt|xt-1), but we don’t know q(xt-1),q(xt). To calculate this we have to know data distribution itself(q(x0)). Since we don’t know the exact data distributionq(x0)(We do all this approximate it becasue we don’t know about it) but only have access to data samples, we need a different approach.

Identity of Joint Distribution

Forward chain

Reverse chain

Those two chain above are same, so we can get relation:

So we get

Applying Markov Property

We know markov property, so we can rewritten q(xt|xt-1,x0) to q(xt|xt-1), then:

In this post,I just revealed the distribution what q(xt-1|xt,x0) follows. You can see the specific derivation from my notion link[https://www.notion.so/and-304daf4ec4318015aa1ffd0f694f386d?source=copy_link]

mean

variance

Substitution x0 with the expression for xt

When we do sampling, model don’t know exact x0. So, sustitute x0 with expression for xt

Plug in:

Reverse Equation

The σ blow is the variance that we explored above

And the noise term in the meas is the object that model have to predict.

Reverse Markov chain

Mathematically, the reverse process is formulated as a Markov Chain, but in practice, it is a step-by-step iterative calculation of what we derived above. Now that we have constructed what we need. Next step is to set the engine which can actually run those processes. In DDPM, the variances are fixed but in models like IDDPM, the model also learns the variance as well. There is no absolute rule. It’s all about the trade-off between simpicity and performace

How to Learn?

Loss function of DDPM is given as below.

∊θ(xt,t)

Here, I’m going to figure this loss function out from DSM, which I mentioned in previous post. We know q(xt|x0) so we can also calculate the score of q(xt|x0).

We know q(xt|x0) because, we have VP-SDE whose drift is linear and diffusion follows Gaussian, Markov chain, which can calculate jumped step by markov chain, and Preserved Gaussianity which make the caculation much simpler, and thanks to all of this, we can make Accumulated form.

log e^-(x+z) = -x-z

Then plug this relation into score:

Then we get:

Denosing Score Matching

Let’s back to the DSM how is it looks like

Here, we can rewrite sθ as:

Substitute our score and sθ into DSM, then

At a specific timestpe t, the term 1/2(1-αₜ(bar)) acts as a constant coeffictient. By ingnoring this scale and focusing only on the noise prediction error, we can simplify the objective in to:

This identity proves that minimizing noise prediction error is equivalent to learning the structure of data distribution.

DDIM

In this section, I’m not going to derive the formula which defines DDIM, and only explain the validity and Sampler. I will skip the derivation and jump straight to the results. For the full mathematical derivationm check here.[https://www.notion.so/DDIM-310daf4ec43180bdb188c187e171fc44?source=copy_link]

Limitations of DDPM

Reverse process of DDPM is constrained by the Markovian property. This means that once xt is given, x0 provides no additional information for determining xt-1. Consequently, we are forced to chain of dependencies xt to xt-1, and reverse x by this chain of dependencies. It requires many steps through the reverse process, leading to slow sampling process.

Non-Markovian: The First Building Block

Non-Markovian simply refers to property that doen not follow the Markovian property. We will explore how DDIM broke the chain of dependency that DDPM was bound to, and how it achieved a massive leap in sampling speed. But as I explained Markov Chain, being Non-Markovian isn’t some complex mystery. It simply means the process is no longer restricted by the Markovian rule. DDIM is Non-Markovian because it has a access to x0(predicted) during sampling.

Probability Flow ODE: The Second Building Block

Probability Flow ODE(PF-ODE for short) is a deterministic Ordinary Differential Equation that recovers the exact same marginal probability trajectories correspond with stochastic SDE . Theoretically,once the prior noise z is given, x0 can be calculated in a single step via PF-ODE. However, since the noise that predicted by model is not perfectly accurate, so we need more then one step(More opportunity to correct the error.) to recover x from z.

continuous-time PF-ODE

Reverse Equation(Mathematically)

By solving the ODE, we derive the form below. Which depends solely on a deterministic path. However, due to certain engineering issues, such as error accumuliatoin, we incorporate with a stochastic term to improve its quality of x.

Reverse Equation(Engineered)

Variance same as DDPM.

predicted x0

The reason the variance term is structured this way is to ensure that when η = 1(η is the scaler which controll how the path is stochastic), the sampling trajectory becomes indentical to that of the DDPM sampler.

Conclusion

DDPM and DDIM can be differed by how they make z to x. While DDPM take step by step, DDIM makes its own way by refined PF-ODE. The differences of model is how they will construct the bridge between z to x. As for me, this concept is really helpful to understand diffusion model. I was confused those mathematics, and I’ve overwhelmed by the dense of mathematics, and losing sight of big picture. However, once I realized that these complex formulas are simply tools to build that the bridge, everything started to make sense.

Part 2:Experiment-Different eta, step, corrector

In this section, I would like to share my experiment only. If you want, you can see my code from github[B-LED-LUE/DDPM_DDIM_MNIST-study: A comparative study of DDPM and DDIM on MNIST (digit 5), exploring variance schedules and stochastic dynamics.].

Experimental Setup

For the experinmental setup, the model was trained using a linear noise schedule with 1000 steps in the forward diffusion process. I utilized the MNIST datasetm intentionally resricting the training process to the 5 class only. This focused training approach allowed for a more controlled obsevation of the model’s generative quality and structural consistency.

DDPM: Corrector, variance, steps.

Figure A, B | DDPM 1000 steps, corrector:O, X, sigma: beta

Figure C, D | DDPM 500 steps, corrector:O, X, sigma: beta

Figure E, F | DDPM 1000 steps, corrector:O,X simga:beta_tilde

Figure G, H |DDPM 500 steps, corrector:O,X sigma:beta_tilde

Experimental Analysis

I observed a significant degradation in visual quality when force-reducing the sampling process to 500 steps, comaperd to the 1000-step reverse process the model was originally trained on. .Regarding the variance schedule, while the difference between the β approximation and the exact β_tilde was subtle. It is hard to generalization which is better, but in this case, the β-based approach felt more “five-like”. Futhermore, the presence of a corrector did not appear to yield a substantial impact on the final image clarity in this condition.

The degradation in image quality when forcing a 500-step sampling is likely due to the disruption of the reverse markov chain. Since the model was optimized for a 1000-step trajectory, reducing the steps creates significant discretization errors, breaking the mathematical continuity required for high-quality reconstruction.

DDIM: Different Steps, Different Eta

Figure 1,2,3 | 5step, 50step, 500step, eta=0

Figure 4,5,6 | 50 steps eta = 0, eta= 0.3, eta = 0.7

Experimental Analysis

While DDIM mathematically enables reaching x0 with fewer steps, I observed that excessively low step counts lead to unstable image quality, as the model has fewer opportunities to correct errors in the predicted noise. Futhermore, when testing η values of 0, 0.3 and 0.7 at 50 steps, the introduction of stochasticity significantly reduced the “fog-like” noise artifacts. This confirms that adding a probabilistic component effectively refines the output and enhances overall visual quality.

Conclusion

It would be premature to generalize the full characteristics of DDPM and DDIM based solely on my specific experimental setup. However, this study provided a meaningful platform to empirically observe the influence of stochastic dynamics and the direct impact of step counts on generative quality. Futhermore, it offered a valuable opportunity to evaluate the extent to which mathematical properties can be approximated while maintaining sructural integrity.