Stable Diffusion v1.5 for Retinal Image Generation: Architecture, Limitations, and Motivation for GANDiffusion

In the previous article, I designed a Structural Causal Model (SCM) to provide clinically meaningful conditioning for retinal image generation. Beginning with the motivation for structured conditioning in medical image synthesis, I developed a hierarchical causal model that captures the relationships between disease severity, lesion burden, vascular abnormalities, anatomical characteristics, and imaging conditions. I then demonstrated how this causal graph is probabilistically evaluated to generate structured metadata, how the metadata is transformed into semantic soft prompts and spatial clinical priors, and finally how these conditioning signals are prepared for training through the RetinaDataset pipeline.

While the previous article established what clinical information should guide retinal image generation, an equally important question still remains:

How is this structured clinical knowledge incorporated into the Stable Diffusion architecture during training and image generation?

Answering this question requires a detailed understanding of the internal design of Stable Diffusion. Although the original Stable Diffusion model was developed for text-to-image generation, my framework replaces natural-language conditioning with structured clinical knowledge generated by the SCM. Consequently, understanding how semantic conditioning, spatial conditioning, and latent image representations interact inside the diffusion model becomes essential before introducing the GANDiffusion architecture.

This article therefore explains the architectural foundation upon which my framework is built. I discuss the Variational Autoencoder (VAE), the UNet denoising network, Attention Blocks, ControlNet, and the diffusion scheduler, emphasizing how these components interact during the denoising process and how they are adapted to incorporate SCM-derived semantic and spatial conditioning. This architectural understanding forms the basis for the next article, where I present the complete low-level design and training methodology of GANDiffusion.

Index

1. Stable Diffusion Components
2. Variational Autoencoder (VAE)
3. CLIP Text Encoder
4. Understanding the Diffusion Process
5. UNet Denoising Network
6. Noise Scheduler
7. ControlNet and UNet Conditioning Using the Structural Causal Model
8. Conclusion

1.Stable Diffusion Components

Before discussing why Stable Diffusion alone was not sufficient for my retinal image generation pipeline, it is important to first understand how the model itself operates. Many of the architectural decisions that follow — including the introduction of adversarial learning, LoRA, the hierarchical critic, and eventually GANDiffusion — are built directly on top of Stable Diffusion’s internal components. Without understanding how information flows through the model, it becomes difficult to appreciate why certain modifications were necessary and why they were introduced at specific locations within the architecture. Therefore, before exploring the limitations of Stable Diffusion and the motivation behind GANDiffusion, I first describe the major components of Stable Diffusion and the role each component plays during image generation. Throughout this section, I will discuss each of these components individually before showing how I extended the original architecture to build GANDiffusion.

The major components of Stable Diffusion v1.5 are:

• Variational Autoencoder (VAE)
• CLIP Text Encoder
• UNet Denoising Network
• Noise Scheduler

After explaining the core components of Stable Diffusion, I will then discuss how I extended the original architecture by integrating ControlNet for spatial conditioning, LoRA for parameter-efficient fine-tuning, SCM-based semantic conditioning in place of the CLIP text encoder.

2. Variational Autoencoder (VAE)

The first major component of Stable Diffusion is the Variational Autoencoder (VAE). Although Stable Diffusion is commonly described as a diffusion model, the diffusion process itself does not operate directly on high-resolution RGB images. Instead, it operates within a continuous latent space that approximately follows a standard normal distribution. The responsibility of the VAE is to learn this latent space and provide a mapping between RGB images and their corresponding latent vectors.

A Variational Autoencoder consists of two components:

• VAE Encoder — Encodes an input image into a point in the latent space.
• VAE Decoder — Decodes a latent point back into an RGB image.

Unlike a conventional autoencoder, which simply compresses an image into an arbitrary feature vector, a VAE learns a structured latent space in which nearby points correspond to visually similar images and the latent variables approximately follow a standard normal distribution. This property allows diffusion models to begin image generation from random Gaussian noise and progressively transform it into meaningful images.

If diffusion were performed directly on a 512 × 512 × 3 RGB image, the model would need to predict and remove noise from more than 786,000 pixel values at every denoising step. Since diffusion requires dozens of denoising iterations, operating directly in pixel space would be computationally expensive and memory intensive.

Instead, the VAE projects every image into a much smaller latent space (typically represented by tensors of size 64 × 64 × 4 in Stable Diffusion v1.5). The diffusion model therefore performs denoising entirely within this compact latent space rather than on the original image, greatly reducing both computation and GPU memory requirements while preserving the semantic information necessary to reconstruct the image.

The two components of the VAE are used at different stages of the diffusion process.

During training, the VAE Encoder receives the real image and maps it into the latent space. Noise sampled from a Gaussian distribution is then added to this latent according to the diffusion schedule, and the resulting noisy latent is passed to the UNet, which learns to predict the injected noise.

The VAE Decoder is not involved in the training forward pass because the diffusion model never operates directly on RGB images.

During inference, the process begins by sampling a random point from the same Gaussian latent space learned by the VAE. The UNet progressively removes noise from this latent over multiple denoising steps until a clean latent is obtained. The VAE Decoder then converts this denoised latent back into the final RGB retinal image.

The VAE acts as the interface between the image domain and the latent space. Every real image first passes through the VAE Encoder before entering the diffusion model, and every generated latent is finally converted back into an image by the VAE Decoder. The remaining components of Stable Diffusion — including the UNet, attention blocks, noise scheduler, timestep embeddings, and text conditioning — operate entirely within this latent space.

3. CLIP Text Encoder

The CLIP Text Encoder provides the semantic conditioning required by Stable Diffusion. Given a textual prompt such as “A retinal fundus image with severe diabetic retinopathy”, it converts the prompt into a sequence of dense semantic embeddings that capture the meaning of the text. These embeddings are then supplied to the UNet through cross-attention layers, allowing the diffusion model to generate images that are consistent with the provided prompt.

Although the CLIP Text Encoder plays a central role in the original Stable Diffusion architecture, it is not used in my implementation. Since my objective is to generate retinal images conditioned on structured clinical variables rather than natural language descriptions, I replace the CLIP Text Encoder with an SCM-based semantic conditioning network, which produces learnable semantic embeddings directly from the Structural Causal Model.

4. Understanding the Diffusion Process

Before understanding the architecture of the UNet, it is important to first understand the diffusion process, since the UNet is specifically designed to learn one part of this process. The diffusion model generates images by learning to reverse a gradual corruption process. Instead of generating an image directly from random noise in a single step, it progressively transforms random Gaussian noise into a meaningful image through a sequence of denoising operations. This generation process consists of two complementary stages:

• Forward Diffusion Process
• Reverse Diffusion Process

The forward diffusion process is a predefined mathematical process that gradually corrupts a clean latent by injecting Gaussian noise over multiple timesteps. Let the clean latent produced by the VAE Encoder be denoted as z0.

At every timestep t, Gaussian noise

is injected according to a predefined noise schedule. One of the most important properties of diffusion models is that the noisy latent corresponding to any timestep can be generated directly without computing all intermediate noisy latents. Instead of sequentially computing

the scheduler directly constructs the noisy latent using

Consequently, during every training iteration, the training pipeline requires only:

1. the clean latent z0​,
2. a randomly sampled timestep t,
3. randomly sampled Gaussian noise ϵ.

Using these three quantities, the diffusion scheduler immediately constructs the noisy latent corresponding to the selected timestep.

This direct construction is one of the key reasons diffusion models are computationally efficient during training. Even if the diffusion process contains hundreds or thousands of timesteps, the training algorithm does not need to generate every intermediate noisy latent. Instead, it can directly sample any timestep and construct the corresponding noisy latent in a single operation.

The objective of the reverse diffusion process is exactly the opposite. Starting from a noisy latent, the model progressively estimates and removes the injected Gaussian noise until the original clean latent is recovered. Unlike the forward diffusion process, whose equations are predefined, the reverse diffusion process is unknown and must be learned from data. The component responsible for learning this reverse process is the UNet, which will be discussed in the next subsection.

5. UNet Denoising Network

Training the UNet to Learn Reverse Diffusion

Having understood the forward and reverse diffusion processes, the next question is how the UNet actually learns to reverse the corruption introduced during forward diffusion.

A natural intuition would be to train the model to directly predict the original clean latent z0 from the noisy latent zt​. However, diffusion models instead train the UNet to predict the Gaussian noise that was originally injected into the clean latent.

The UNet receives three inputs:

• the noisy latent zt,
• the timestep embedding t,
• the conditioning information c.

Its objective is to learn the mapping

where RHS represents the predicted Gaussian noise.

Since the training pipeline generated the Gaussian noise while constructing the noisy latent, the actual injected noise is already available as the ground-truth target. Therefore, the UNet is optimized using the Mean Squared Error (MSE) loss

The resulting loss is backpropagated through the entire UNet, allowing it to gradually learn how Gaussian noise appears at different diffusion timesteps.

The diffusion timestep specifies the current level of noise present in the latent, whereas the encoder, bottleneck, and decoder are simply layers within a single neural network. During training, the complete UNet is executed once for the selected timestep. The noisy latent sequentially passes through every encoder block, bottleneck block, and decoder block before the network predicts the injected Gaussian noise.

During inference, however, the same pretrained UNet is reused repeatedly. Suppose inference begins from timestep T=50. The scheduler first provides the noisy latent z50​ to the UNet, which predicts the injected noise. The scheduler then uses this prediction to compute z49. The newly computed latent is again passed through the same UNet, and this procedure continues until the final clean latent is obtained.

Consequently, if inference is performed using 50 denoising steps, the entire UNet architecture is executed 50 separate times. At every step, the encoder, bottleneck, decoder, residual blocks, and attention blocks participate in predicting the Gaussian noise corresponding to the current timestep, while the scheduler computes the next, less noisy latent. The UNet itself never performs the denoising — it only predicts the noise. The actual denoising is carried out by the diffusion scheduler using the UNet’s predictions.

UNet Architecture

Having understood how the UNet is trained to predict Gaussian noise, the next step is to understand how the network itself is organized. Although the UNet is invoked repeatedly throughout the reverse diffusion process, its internal architecture remains identical at every denoising step.

The UNet follows a symmetric encoder–decoder architecture with skip connections and consists of three major components:

1. Downsampling Blocks (Encoder)
2. Middle (Bottleneck) Block
3. Upsampling Blocks (Decoder)

Terminology

• Feature map refers to any intermediate tensor produced inside the UNet, at any stage of the encoder, bottleneck, or decoder.
• Spatial location refers to one (h,w) position within a feature map, i.e., one vector along the channel dimension

Resolution and Channel Progression

Rather than scattering resolution numbers throughout the text, the table below summarizes how spatial resolution and channel count change at each stage, for a UNet operating on a 64×64 input latent (corresponding to a 512×512 image compressed by the VAE):

As resolution decreases, channel count increases — spatial information is progressively traded for semantic richness. This single pattern explains most of the encoder’s and decoder’s design.

Common Building Blocks: ResNet Block and Attention Block

Residual (ResNet) Block

Residual Block is the primary feature-extraction unit of the UNet. Every ResNet Block is explicitly conditioned on the current diffusion timestep t, so the same anatomical structure is processed differently depending on how much noise is expected at that timestep. This same block is reused, unchanged in structure, in every encoder stage, in the bottleneck, and in every decoder stage — the only thing that varies between uses is the input feature map fed into it (e.g., a plain encoder feature map vs. a skip-concatenated decoder feature map).

Attention Block

Convolutions only see a small local neighborhood, so a vessel on one side of the image cannot directly influence a distant vessel or lesion without passing through many layers. The Attention Block removes this limitation by letting every spatial location directly exchange information with every other spatial location. Like the ResNet Block, this same block — same equations, same four learnable projection matrices — is reused at selected resolutions in the encoder, in the bottleneck, and in the decoder.

Attention trades a local view for a global one, at the cost of O(N²) computation — which is why it is used at reduced resolutions, and it is reused identically everywhere in the UNet.

Self-Attention, Cross-Attention and Feed-Forward Networks

The attention mechanism described above computes relationships between feature vectors using the Query, Key, and Value projections. In Stable Diffusion, however, an Attention Block consists of three consecutive operations rather than a single attention computation:

Input Features
│
▼
Self-Attention
│
▼
Cross-Attention
│
▼
Feed-Forward Network
│
▼
Output Features

Each component serves a different purpose during image generation.

Self-Attention

Self-attention models the relationships within the image itself. The Query, Key, and Value matrices are all computed from the same feature map produced by the UNet.

Because all three projections originate from the same input, every spatial location can exchange information with every other spatial location in the feature map. This enables the network to capture long-range dependencies that cannot be learned efficiently using convolutional layers alone.

For example, when generating retinal fundus images, a blood vessel appearing on one side of the retina can directly influence vessel continuity on the opposite side without requiring information to propagate through many convolutional layers.

Consequently, self-attention helps the diffusion model learn global consistency within the generated image.

Cross-Attention

While self-attention relates image features to other image features, cross-attention relates image features to external conditioning information.

Instead of computing the Query, Key, and Value matrices from the same feature map, cross-attention computes

from the UNet feature map,

are computed from the encoder hidden states E.

Get Ujjjwalx’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

During conventional Stable Diffusion training, these encoder hidden states are produced by the CLIP text encoder from a natural-language prompt.

In my framework, however, the CLIP text encoder is replaced by the SCM Primary Conditioning Network, which generates encoder hidden states directly from the 24-dimensional SCM parameter vector.

Consequently, during image generation, every spatial location within the retinal image continuously asks

“Which clinical information contained in the SCM is most relevant to me?”

The attention weights computed from the Query-Key similarity determine how strongly each retinal region attends to different semantic descriptors such as lesion burden, vascular abnormalities, or anatomical characteristics.

Cross-attention therefore provides the mechanism through which structured clinical knowledge influences the diffusion process.

Feed-Forward Network

After the attention operations, every feature vector has already exchanged information with both the image itself (through self-attention) and the conditioning information (through cross-attention).

To increase the representational capacity of the network, the attention block applies a Feed-Forward Network (FFN) independently to every spatial location.

A typical FFN consists of two fully connected layers separated by a nonlinear activation function:

Unlike attention, the Feed-Forward Network does not exchange information between different spatial locations. Instead, it performs nonlinear feature transformation independently for every location, allowing the network to learn more expressive representations after the attention mechanisms have aggregated the relevant information.

Downsampling Blocks (Encoder)

The encoder is the first half of the UNet. It takes the noisy latent ztz_t zt​ and repeatedly extracts more abstract, semantically rich features from it while shrinking its spatial size, producing a sequence of feature maps that the decoder will later reuse.

At first instance, further downsampling may look redundant, since the VAE encoder already compressed the 512×512 image into a 64×64 latent. But the VAE’s compression only produces a computationally efficient starting resolution — it says nothing about how information should be organized internally for denoising. That reorganization is the encoder’s job.

As the feature map propagates through the encoder, spatial resolution drops while channel count rises (see table above). Early on, each spatial location holds only a few channels, limiting how much it can represent. As channels increase, each remaining location can encode a much richer description — no longer just raw pixel intensities, but concepts such as vessel boundaries, vessel thickness, optic disc structures, hemorrhages, exudates, background texture, illumination, and disease severity, all discovered automatically during training.

This hierarchical transformation serves three purposes:

1. Larger receptive field. At high resolution, each convolution sees only a small neighborhood — good for local structures like vessel edges. At low resolution, each feature effectively summarizes a large image region, enabling reasoning about global anatomical organization.
2. Cheaper attention. Since attention cost grows as O(N²) in the number of spatial locations N, reducing resolution makes global attention computationally tractable.
3. Richer per-location representation. Instead of spreading information across many spatial locations, the network packs increasingly complex information into the channel dimension.

Each Downsampling Block contains:

• Multiple Residual (ResNet) Blocks (as defined above)
• Attention Blocks at selected resolutions (as defined above)
• A Downsampling Layer (encoder-specific, defined below)

Before each downsampling operation, the current feature map is stored via a skip connection — these stored feature maps are reused later by the decoder to recover fine detail that would otherwise be lost (see the Decoder’s Skip Connection section below).

Downsampling Layer

What it is: The one layer that is specific to the encoder — the operation that shrinks spatial resolution and grows channel count between encoder stages.

Mechanism: A strided convolution (stride 2) halves each spatial dimension while doubling (or otherwise increasing) the channel count:

Middle (Bottleneck) Block

What it is: After several rounds of downsampling, the feature map reaches its smallest spatial size but its largest channel count. Each remaining “pixel” no longer represents a small image patch — it summarizes a large region of the retina. The network has moved from raw image detail to an abstract, semantic understanding of the whole scene. Unlike the encoder and decoder, the bottleneck does not resize the feature map at all; it only refines it, using nothing but the two shared blocks defined above.

The bottleneck consists of three stages:

Formally, the whole bottleneck computes:

Upsampling Blocks (Decoder)

What it is: The decoder is the mirror image of the encoder. Starting from the bottleneck’s abstract representation, it progressively reconstructs a full-resolution feature map, from which the injected noise will eventually be predicted. Where the encoder compressed spatial detail into channels, the decoder expands channels back into spatial detail — but it does not have to reconstruct fine anatomy from scratch, because it reuses the encoder’s stored feature maps via skip connections.

Each Upsampling Block consists of:

• A Skip Connection from the corresponding encoder stage (decoder-specific, defined below)
• Multiple Residual (ResNet) Blocks (as defined above, applied to the skip-concatenated feature map)
• Attention Blocks at selected resolutions (as defined above)
• An Upsampling Layer (decoder-specific, defined below)

Skip Connection

Skip Connection is the mechanism that lets the decoder recover fine detail (vessel edges, small lesions) that downsampling would otherwise have destroyed, by directly reusing the matching-resolution feature map saved during the encoder pass.

The decoder does not need to reconstruct fine retinal structures purely from the bottleneck’s abstract representation — it directly reuses precise encoder features while refining them with the globally contextualized information carried down from the bottleneck.

Upsampling Layer

Upsample layer is the one layer that is specific to the decoder — the operation that grows spatial resolution and shrinks channel count between decoder stages

Although the encoder, bottleneck, and decoder each transform the feature map, none of them individually predicts the injected Gaussian noise — they progressively build richer representations that the final layer uses to estimate it.

This error is backpropagated through the entire UNet, jointly optimizing the encoder, bottleneck, decoder, and final prediction layer. The three components are not learning separate tasks — they cooperate to build progressively richer representations, using the same shared ResNet and Attention Blocks throughout, so that the final layer can accurately estimate the Gaussian noise added at the sampled timestep.

6. Noise Schedular

The noise scheduler is the mathematical component that governs both the forward and reverse diffusion processes introduced in the previous sections. Unlike the UNet, it contains no learnable parameters; instead, it implements the predefined diffusion equations that specify how Gaussian noise is added during training and removed during inference.

to step from zt to zt-1. It is worth restating the division of labor between the two components: the UNet does not denoise directly — it only predicts the noise component of the current latent. The scheduler is what converts that prediction into an actual state transition. In short, the UNet learns what noise is present, while the scheduler determines how that prediction is used to move from one timestep to the next.

While the forward process is fixed across diffusion models, the reverse process admits different algorithmic choices. The two most widely used are DDPM and DDIM.

DDPM Scheduler

The DDPM scheduler follows the original probabilistic reverse process. At each step, it computes the previous latent from the UNet’s predicted noise and injects a fresh sample of Gaussian noise:

Because new randomness enters at every step, the reverse trajectory is stochastic — even from an identical starting latent, different runs can yield different images.

DDPM’s main strength is sample diversity, since it closely follows the true probabilistic reverse process. Its drawback is cost: it typically needs several hundred to a thousand denoising steps, making inference expensive.

DDIM Scheduler

The DDIM scheduler modifies only the reverse diffusion process while leaving the training procedure completely unchanged. Instead of introducing new Gaussian noise during every denoising step, DDIM performs a deterministic update,

using only the current latent and the UNet’s predicted noise.

Since no additional randomness is injected after the initial Gaussian latent, the reverse trajectory becomes deterministic. Given the same initial latent, conditioning information, and pretrained UNet, DDIM always reconstructs the same image.

Because of this deterministic formulation, DDIM can safely skip many intermediate diffusion timesteps while still maintaining high image quality. As a result, high-quality images can often be generated using only 20–50 denoising steps, making DDIM substantially faster than DDPM.

The choice between these schedulers depends on the application. DDPM is generally preferred when maximizing sample diversity is more important than inference speed, whereas DDIM is preferred when fast and reproducible image generation is required. Since my proposed GANDiffusion framework focuses on improving the denoising model rather than modifying the diffusion equations, I employ the DDIM scheduler to achieve significantly faster inference while maintaining high-quality retinal image generation.

7. ControlNet and UNet Conditioning Using the Structural Causal Model

1. UNet Learns What, Not Where

In the standard Stable Diffusion UNet, the denoising process is conditioned via cross‑attention layers. These layers take a sequence of token embeddings (e.g., from a text prompt or a learned vector) and allow each spatial location of the latent feature map to attend to these tokens.

This mechanism is excellent for conveying semantic content — it tells the UNet what kind of objects or patterns should appear (e.g., “hemorrhages,” “microaneurysms”). However, cross‑attention is inherently position‑agnostic: the same token can be attended to by any pixel, and the network has no built‑in bias to place a specific pathology at a specific anatomical location.

As a result, even when the semantic conditioning is identical, the generated images may exhibit random spatial layouts — hemorrhages could appear near the optic disc, in the periphery, or anywhere in between. This is unacceptable for medical image generation, where anatomical consistency is crucial.

2. Semantic Conditioning via encoder_hidden_states

Our approach uses a Structural Causal Model (SCM) to generate a 24‑dimensional parameter vector that encodes hierarchical clinical factors (root burdens, lesion severities, vascular changes, etc.). This vector is fed into a small network called SCMPrimaryConditioning, which transforms it into a sequence of 77 tokens of 768 dimensions – exactly the format expected by the UNet’s cross‑attention layers.

class SCMPrimaryConditioning(nn.Module):
    def __init__(self, input_dim=len(PARAM_KEYS), seq_len=77, embed_dim=768):
        super().__init__()
        self.seq_len = seq_len
        self.embed_dim = embed_dim
        self.net = nn.Sequential(
            nn.Linear(input_dim, 256), nn.SiLU(),
            nn.Linear(256, 512), nn.SiLU(),
            nn.Linear(512, seq_len * embed_dim),
        )
        self.pos_embed = nn.Parameter(torch.randn(1, seq_len, embed_dim) * 0.02)
        self.layer_norm = nn.LayerNorm(embed_dim)

    def forward(self, x):
        out = self.net(x).view(-1, self.seq_len, self.embed_dim)
        out = out + self.pos_embed
        return self.layer_norm(out)

In the training loop, we obtain these tokens and pass them directly to the UNet as encoder_hidden_states:

encoder_hidden_states = scm_primary(params_tensor)   # (B, 77, 768)
...
noise_pred = unet(
    noisy_latents,
    timesteps,
    encoder_hidden_states=encoder_hidden_states,
    ...
)

This direct injection works because the UNet has built‑in cross‑attention layers that accept a context tensor of shape (batch, seq_len, embed_dim).

3. The Spatial Prior Cannot Be Directly Injected

While the semantic vector is a one‑dimensional sequence, the spatial prior generated from the same SCM parameters is a 2D heatmap that encodes where pathologies are expected to appear (e.g., hemorrhages near the optic disc, vessel changes in the central retina). This heatmap is in pixel space (e.g., 512×512) and has a completely different structure from the latent space (4×64×64) or the token sequence.

The UNet has no dedicated input for such a spatial map. We cannot simply concatenate it to the latent or pass it as an extra channel because:

• The UNet expects fixed‑size latent features and does not have a mechanism to fuse external spatial conditioning at multiple scales.
• The spatial prior is in image space, while the UNet operates in a compressed latent space; feeding it directly would require a learnable encoder to project it into the latent manifold.

Therefore, we need an auxiliary network that can encode the spatial prior and inject its information into the UNet’s internal feature hierarchy.

4. How ControlNet Encodes the Spatial Prior

Now that we understand why the spatial prior cannot be fed directly into UNet, let’s examine how ControlNet actually encodes this heatmap and injects the “where” information into the generation process.

4.1 ControlNet’s Architectural Design

In our implementation, ControlNet is initialized from the pretrained UNet:

controlnet = ControlNetModel.from_unet(unet)

This is a critical design choice. By copying the UNet’s encoder weights, ControlNet inherits the same powerful feature extraction capabilities that the base model learned on millions of images. However, it also adds zero-initialized trainable conditioning layers specifically designed to accept a spatial conditioning image.

┌─────────────────────────────────────────────────────────────────┐
│               CONTROLNET ARCHITECTURE (from code)               │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│   Input 1: noisy_latents (B, 4, 64, 64)                         │
│   Input 2: timestep (scalar)                                    │
│   Input 3: encoder_hidden_states (B, 77, 768) ← semantic        │
│   Input 4: controlnet_cond (B, 3, 512, 512) ← SPATIAL PRIOR     │
│                                                                 │
│   ┌──────────────────────────────────────────────────────┐      │
│   │  Copied UNet Encoder (frozen or trainable)           │      │
│   │                                                      │      │
│   │  Block 0: (B, 320, 64, 64)   ← spatial prior         │      │
│   │  Block 1: (B, 640, 32, 32)   ← processed through     │      │
│   │  Block 2: (B, 1280, 16, 16)  ← zero-conv layers      │      │
│   │  Mid:    (B, 1280, 8, 8)                             │      │
│   └──────────────────────────────────────────────────────┘      │
│                         │                                       │
│                         ▼                                       │
│   Output: Spatial Residuals (matching UNet skip connections)    │
│   • down_res[0]: (B, 320, 64, 64)                               │
│   • down_res[1]: (B, 640, 32, 32)                               │
│   • down_res[2]: (B, 1280, 16, 16)                              │
│   • mid_res:     (B, 1280, 8, 8)                                │
└─────────────────────────────────────────────────────────────────┘

ControlNet processes the spatial prior through its copied UNet encoder. At each down-sampling block, it performs the following operations:

Step 1: The spatial prior (controlnet_cond) is passed through a series of zero-convolution layers that project it into the appropriate feature space for each UNet block. These layers start with zero weights, ensuring that initially, ControlNet has no effect on the UNet (preserving the pretrained behavior).

Step 2: At each encoder block, ControlNet combines:

• The encoded features from the noisy latents
• The spatially-conditioned features from the prior
• The semantic tokens via cross-attention (same as UNet)

Step 3: The output of each block is collected as a residual that matches the exact shape of the corresponding UNet skip connection. The residual is the encoded spatial guidance that has been influenced by all three inputs.

In our training loop, this entire encoding process happens in one forward call:

down_res, mid_res = controlnet(
    noisy_latents,                      # (B, 4, 64, 64)
    timesteps,                          # (B,)
    encoder_hidden_states=encoder_hidden_states,  # (B, 77, 768) ← semantic
    controlnet_cond=cond_img,           # (B, 3, 512, 512) ← SPATIAL PRIOR
    conditioning_scale=1.0,
    return_dict=False,
)

4.4 Injection into the UNet: The Additive Residual Mechanism

When we call the UNet with these residuals:

noise_pred = unet(
    noisy_latents,
    timesteps,
    encoder_hidden_states=encoder_hidden_states,      # ← semantic
    down_block_additional_residuals=down_res,         # ← spatial guidance
    mid_block_additional_residual=mid_res,            # ← spatial guidance
).sample

The UNet’s internal forward pass performs the following at each encoder block:

# Inside UNet (conceptual)
for block_idx, block in enumerate(unet.encoder_blocks):
    # Normal UNet processing
    features = block(features, timestep, encoder_hidden_states)
    
    # ADD the ControlNet residual (spatial anchor!)
    features = features + down_res[block_idx]   # ← THE "WHERE"

This additive mechanism is the key. The spatial prior has been transformed into the same feature space as the UNet’s internal representations, so adding them directly anchors the UNet’s features to the spatial locations prescribed by the prior.

4.5 Summary of the Encoding Pipeline

# Step 1: Generate spatial prior (pixel-space)
prior = generate_spatial_prior(scm_samples, grid_size=512)   # (512, 512)
cond_img = prior.unsqueeze(0).repeat(3, 1, 1)               # (3, 512, 512)

# Step 2: ControlNet encodes it into multi-scale residuals (latent-space)
down_res, mid_res = controlnet(
    noisy_latents,
    timesteps,
    encoder_hidden_states=encoder_hidden_states,  # semantic context
    controlnet_cond=cond_img,                     # ← spatial prior
)

# Step 3: UNet receives the encoded spatial anchors as residuals
noise_pred = unet(
    noisy_latents,
    timesteps,
    encoder_hidden_states=encoder_hidden_states,  # WHAT
    down_block_additional_residuals=down_res,     # WHERE (encoded)
    mid_block_additional_residual=mid_res,        # WHERE (encoded)
).sample

In essence, ControlNet translates the pixel-space “where” information into multi-scale latent-space residuals that the UNet can natively incorporate through addition, completing the conditioning architecture. The UNet now receives both the semantic tokens (via cross-attention) and the spatial anchors (via residual addition), ensuring that the generated retinal pathologies are both clinically accurate and anatomically consistent.

The UNet_Features is the synthesis engine — it knows how to draw. The ControlNet_Residual is the steering wheel — it knows where to draw. You need the engine to move, and the steering wheel to go in the right direction. Remove the engine, and the steering wheel is useless. Remove the steering wheel, and you crash into the wrong anatomy

8. Conclusion

Building a Stable Diffusion v1.5 pipeline for severe diabetic retinopathy image generation demonstrated that diffusion models learn meaningful retinal representations far more effectively than my earlier GAN‑based approaches. By replacing text prompts with SCM‑based semantic conditioning and incorporating ControlNet to provide structural guidance, the model produced encouraging retinal fundus samples. However, even with these sophisticated conditioning mechanisms, the generated outputs remained highly inconsistent in terms of anatomical fidelity and overall realism during inference. Under identical conditioning settings, the model would occasionally produce reasonably plausible retinal images, but in many cases it generated structurally distorted fundus geometries, abnormal background coloration, or implausible tissue textures — ranging from poor to moderate quality, but never reliably realistic. This variability revealed that diffusion loss alone, while powerful for learning the global image distribution, is insufficient to consistently enforce fine‑grained anatomical constraints and photorealism. My initial attempt to remedy this with a separate GAN‑based adversarial refinement stage applied after image generation proved ineffective, as it failed to correct the structural flaws introduced during the diffusion process. This failure led to the critical realization that adversarial learning should not be treated as a post‑hoc patch, but rather become an integral part of the diffusion optimization workflow itself. This insight motivated the development of GANDiffusion, where adversarial supervision is woven directly into the training loop to guide the denoising network toward more realistic, anatomically stable, and consistently high‑quality retinal images. In the next article, I will present the mathematical formulation, architectural design, and complete low‑level implementation of GANDiffusion, detailing how this integrated adversarial framework overcomes the limitations observed in the purely diffusion‑based approach.