Stable Diffusion
Stable Diffusion is a latent text-to-image diffusion model ,it is a powerful, open-source text-to-image generation model. While there exist multiple open-source implementations that allow you to easily create images from textual prompts, KerasCV’s one of those open source Library .
Unlike what you might expect at this point, Stable Diffusion doesn’t actually run on magic. It’s a kind of “latent diffusion model”. Let’s dig into what that means.
You may be aware with the concept of super-resolution: by training a deep learning model to remove noise from an input image, a higher quality version of the image can be created. Instead of using its training data distribution to imagine the visual details that would be most likely given the input, the deep learning model achieves this by magically retrieving the information that is absent from the noisy, low-resolution input.
When you push this idea to the limit, you may start asking — what if we just run such a model on pure noise? The model would then “denoise the noise” and start hallucinating a brand new image. By repeating the process multiple times, you can get turn a small patch of noise into an increasingly clear and high-resolution artificial picture.
Now, to go from latent diffusion to a text-to-image system, you still need to add one key feature: the ability to control the generated visual contents via prompt keywords. This is done via “conditioning”, a classic deep learning technique which consists of concatenating to the noise patch a vector that represents a bit of text, then training the model on a dataset of {image: caption} pairs.
This gives rise to the Stable Diffusion architecture. Stable Diffusion consists of three parts:
- A text encoder, which turns your prompt into a latent vector.
- A diffusion model, which repeatedly “denoises” a 64x64 latent image patch.
- A decoder, which turns the final 64x64 latent patch into a higher-resolution 512x512 image.
First, your text prompt gets projected into a latent vector space by the text encoder, which is simply a pretrained, frozen language model. Then that prompt vector is concatenate to a randomly generated noise patch, which is repeatedly “denoised” by the decoder over a series of “steps” (the more steps you run the clearer and nicer your image will be — the default value is 50 steps).
Finally, the 64x64 latent image is sent through the decoder to properly render it in high resolution.
As mentioned one of the most powerful library in Diffusion models is KerasCV .. Let’s see the result .
First you would need to install KerasCV
!pip install tensorflow keras_cv --upgrade --quiet
!apt install --allow-change-held-packages libcudnn8=8.1.0.77-1+cuda11.2Next you would need to import your model
import keras_cv
from tensorflow import keras
import matplotlib.pyplot as plt
import tensorflow as tf
import numpy as np
model = keras_cv.models.StableDiffusion(img_width=512, img_height=512)Next you would just need to pass the text (Caption ) to the model for generation .
images = model.text_to_image("photograph of Man with a dog", batch_size=3)
def plot_images(images):
plt.figure(figsize=(20, 20))
for i in range(len(images)):
ax = plt.subplot(1, len(images), i + 1)
plt.imshow(images[i])
plt.axis("off")
plot_images(images)
images = model.text_to_image("photograph of an astronaut riding a horse", batch_size=3)
def plot_images(images):
plt.figure(figsize=(20, 20))
for i in range(len(images)):
ax = plt.subplot(1, len(images), i + 1)
plt.imshow(images[i])
plt.axis("off")
plot_images(images)
It is better to run the model using GPU and It would take quite bit time around 5 mins on Colab GPU.
Get Redamahmoud’s stories in your inbox
Join Medium for free to get updates from this writer.
You can add more one description .
images = model.text_to_image(
"cute magical flying dog, fantasy art, "
"golden color, high quality, highly detailed, elegant, sharp focus, "
"concept art, character concepts, digital painting, mystery, adventure",
batch_size=3,
)
plot_images(images)
In Stable Diffusion, a text prompt is first encoded into a vector, and that encoding is used to guide the diffusion process. The latent encoding vector has shape 77x768 (that’s huge!), and when we give Stable Diffusion a text prompt, we’re generating images from just one such point on the latent manifold.
To explore more of this manifold, we can interpolate between two text encodings and generate images at those interpolated points:
prompt_1 = "An Soldier painting the beach at home"
prompt_2 = "photograph of an astronaut riding a horse"
interpolation_steps = 5
encoding_1 = tf.squeeze(model.encode_text(prompt_1))
encoding_2 = tf.squeeze(model.encode_text(prompt_2))
interpolated_encodings = tf.linspace(encoding_1, encoding_2, interpolation_steps)Once we’ve interpolated the encodings, we can generate images from each point. Note that in order to maintain some stability between the resulting images we keep the diffusion noise constant between images.
seed = 12345
noise = tf.random.normal((512 // 8, 512 // 8, 4), seed=seed)
images = model.generate_image(
interpolated_encodings,
batch_size=interpolation_steps,
diffusion_noise=noise,
)We are going to display the result as gifs on Colab using the following code .
from IPython.display import Image as IImage
IImage("result.gif")
def export_as_gif(filename, images, frames_per_second=10, rubber_band=False):
if rubber_band:
images += images[2:-1][::-1]
images[0].save(
filename,
save_all=True,
append_images=images[1:],
duration=1000 // frames_per_second,
loop=0,
)
export_as_gif(
"result.gif",
[Image.fromarray(img) for img in images],
frames_per_second=2,
rubber_band=True,
)
you can make interpolating with more than 2 sentences and change the steps and noise for different results.
Conclusion
Stable Diffusion offers a lot more than just single text-to-image generation. Exploring the latent manifold of the text encoder and the noise space of the diffusion model are two fun ways to experience the power of this model, and KerasCV makes it easy!
Thanks for reading
