GANs Part11: Image-to-Image Translation.
Understand image-to-image translation framework, and implement a U-Net generator and Pix2Pix, a paired image-to-image translation to turn satellite images into map routes (and vice versa) with advanced U-Net generator and PatchGAN discriminator architectures.
Introduction.
Applying GANs isn’t just about using its generated data, we can also use GAN for image translation, which is a task of transferring styles from one image to another. I.e., you can construct against transfer a satellite image style to map routes and vice versa.
You can do all this by feeding an entire image into the conditional GAN. But, instead of a small one-hot class vector to generate something that looks like a satellite image but with a map style.
This new GAN architecture called Pix2Pix has a beefed-up generator and discriminator.
Image-to-Image Translation
Image-to-Image translation refers to the process of converting an image from one domain to another through applying a transformation to get it in a different style with maintaining its content, which means we’re preserving its essential characteristics (e.g. from a sketch to a realistic image).
This process is often done to generate new images that are visually similar to the original but belong to a different domain.
This is a type of conditional generation, it’s conditioning on the content of one image, e.g., say “conditioning on this black & white image, create a colored style”.
Another example, is going from a segmentation map. These are segmentations of this road, of a car or pedestrians, the sidewalk and trees.
Or, if we’ve frames of a video, we can go map onto the frames of another.
It’s essentially image-to-image translation, but for many frames.
You can take this black & white film and make it 4K with some realistic color. This’s actually not just adding realistic color but going from low resolution to a high resolution (super-resolution)
Paired Image-to-Image Translation.
Every single example in the training set has a corresponding target image that contains the contents of that input image with a different style.
imagine a lot of buildings that could be generated from these techniques. Here are other examples going from day to a nighttime photo. Also, we can draw sketches and have a GAN to generate photo-realistic images from those. you can actually get edges to create this paired training data by just using an edge detection algorithm.
you can easily create this paired dataset. Then you can actually go from any kind of edge that you do draw to a realistic photo after it’s been trained. Then image-to-image translation can actually get even more complex.
Also, instead of conditioning on a single input image, you can take as input models wearing different clothes and also a point-wise map of where they should be standing, their pose.
Those dots represent different poses. Then from these 2 pieces of information, you want your game to generate that person in a different pose. That’s what you see on the right.
Pix2Pix Overview.
Pix2Pix is type of conditional GAN that does paired-image translation. It has a unique generator and discriminator design.
First, how it was named. Image-to-Image translation goes from image-to-image, and then Pix-to-Pix, pix as an image and then Pix2Pix.
Remember the conditional GAN, where a generator takes in a class vector to generate an image of that class. Pix2Pix is similar, but instead of a class vector, you actually pass in an entire image as the input.
This image could be a segmentation map of a building (a real input), where different colors correspond to different classes within that image.
The authors found that the noise vector didn’t make a huge difference to the generator’s outputs (how they look like). As there’s a paired output image that this generator is trying to get. Instead of noise, they actually found that they could add some stochasticity into the network using dropout. So, different nodes can learn different things. As a result, this adds some randomness to the outputs, though it’s not as drastic with a variety of different dogs when you input a different noise vector as before.
Discriminator also gets this real input, this condition to segmentation map that was inputted into the generator. Then it also gets concatenated with it, the real target outputs and the generated output.
It gets one of those concatenated with it and the discriminator has to decide whether it was real or fake. It gets to see the condition which is very similar to the conditional GAN.
Both the generator and discriminator will get an upgrade.
Unit is typically used for segmentation. it encodes an image and then pass it to decoder after that, followed by some skip connections in between.
What that means is that it gives more feedback on different parts of the image. Instead of saying: “real or fake for an entire image”. It will be real or fake for different patches of that image. A lot of real fake determinations in a matrix as an output. This means that the discriminator will give more feedback back to the generator.
Pix2Pix discriminator: PatchGAN
PatchGAN architecture is largely about outputting a matrix of classification values instead of a single output.
Each value in this matrix, is between 0 and 1 where 0 is fake and 1 is real. And PatchGAN does this for all patches by sliding its field of view across all the patches in the image, then give feedback on each region of the image. And because it outputs the probability of each patch being real, it can be trained with BCE loss. So, for a fake image from the generator, the PatchGAN should try to output a matrix of 0.
And the same logic goes for a real image, patchGAN will try to output a matrix of all ones indicating that each patch of the image is real.
Pix2Pix: U-Net
It’s an encoder-decoder model that uses skip connections in between.
Segmentation is taking an image and getting a segmentation mask on every single pixel in terms of what object it is. I.e., Pixels on a car is going to be a car, and you can’t segment it any other way. the segmentation task is very much an image-to-image translation task.
But when it comes to generating things, it’s something without a correct answer. I.e., this image of this car, there’s no correct answer in terms of what that car could be.
I.e., it could be this car back in that image, but since it’s just a segmentation mask, realistically it could be a different car. In all of these, cars are technically correct if you were to look at it as a human, right?
Get AI & Data Science 604’s stories in your inbox
Join Medium for free to get updates from this writer.
Pix2Pix uses the U-Net architecture for the generator architecture. As it’s good at taking in an input image and mapping it to an output image.
The traditional generator architecture takes in a noise vector. The U-Net generator actually takes in an entire image (X). this means that the generator has to be a lot beefier with convolutions to handle that image input. So the architecture of the Pix2Pix generator is this encoder-decoder structure, where 1st you encode things.
you can also think of this middle layer as being a bottleneck where it is able to embed the information in this image. All that important information in that image is compressed into this little bit of space, just so you can get those high-level features, and to then decode it an output (y) (anther image).
And this might remind you of an auto-encoder except for an auto-encoder, you won’t want to be as close as possible to X. You want Y to be a different style conditioned on X. However, since it’s easy to overfit these networks to your training image pairs, U-Net also introduces skip connections from the encoder to decoder. And this is also really useful for getting information that might have been lost during the encoding stage.
So, what happens is that during the encoding stage, every single block that is the same resolution as its corresponding block in the decoding stage get this extra connection to go concatenate with that value. Such that information that might have been compressed too much can still trickle through and still get to some of these later layers.
These skip connections are concatenated from the encoder before going into each convolutional block in the decoder. Skip connections allow information to flow from earlier layers to later layers. In this information could be concatenated, but somehow included into the later layers. So this makes easy to get certain details that the encoder may have lost in the process of down-sampling to the decoder, and that means those finer grain details, this is in the forward pass.
skip connections were introduced to help with the vanishing gradient problem when you stack too many layers together. the gradient gets so tiny when it’s multiplied in back prop, limiting our networks from going deeper and having more layers.
What’s cool in U-Net that very much in the backward pass, this does improve gradient flow to the encoder so that those layers learn from information that might have been in the decoder.
The encoder takes in an image (X) of (256 by 256 by 3) and it goes through all of these blocks (8 encoder blocks) to compress that input. Each block down samples the spatial size by factor of 2 to be (1 by 1by 512) by the very end (8th block) to encode that information. Each of these encoder blocks contains a convolutional layer, a BN norm layer, and LeakyReLU activation.
Meanwhile, on the decoder side you have this input size of (1 by 1 by 512). And because you want to generate an output image with the same size as the encoder input, the decoder actually contains the same number of blocks as the encoder (8 blocks). And then you get (y) (generated image), which is the same size as the input (256 by 256 by 3).
Each decoder block is composed of a transposed convolution to make the output bigger, followed by a BN layer, and then a ReLU activation function. Dropout is added to this network, but it’s actually just added to the first 3 blocks of this decoder.
Note, this stochasticity only present during training (randomly dropout some neurons) and it is typically turned off during inference (all nodes are active). To maintain consistent output during inference, the active neurons are scaled by the inverse of the dropout probability. This ensures that the model’s predictions remain stable and reflect the expected distribution of outputs. The introduction of noise during training helps the model generalize better to new data by preventing it from relying too heavily on any single neuron or feature.
Putting the 2 halves Together.
the information about the input is passed through this small bottleneck of size (1 by 1 by 512) and this small spatial size can be understood as summarizing the encoding of that input image. Think of the decoder as performing the inverse operations as the encoder, which is why they contain the same number of blocks.
The encodings from every single encoder block, are past or forwarded to the corresponding blocker level in the decoder at the same resolution.
U-Net integrates this information from the encoder into the decoder by using concatenation. So concatenating the encoder outputs to each decoder input at each blocker level, where it’s the same resolution, so it’s easy to concatenate them.
Pixel Distance Loss Term
pixel distance loss is an additional loss for the Pix2Pix generator, used to encourages the generator to make the output be similar to the real.
Remember that you can formulate a loss as the adversarial loss e.g., BCE loss or W loss. And you can add an additional last term such as L1 regularization or gradient penalty and a Lambda term (less than one) is added to wait that loss function by some amount.
For Pix2Pixin, we want our output to look pretty, so we’ll add an additional pixel loss term to give the generator a little bit more information about the real image.
let’s take a look at what that means. The pixel distance loss looks at the generated output + it takes the pixel difference between the 2 building, trying to encourage the generated output to be as close as possible to the real output.
If pixel distance is small, then the images are almost the same, and if it’s really big, that means the generated image is far from the target. so, pixel distance laws come in a lot of handy for helping with this realism.
looking across multiple different samples. Some kind of waiting Lambda that you can learn or tune or tweak and looking at the difference between the generated and the real outputs and specifically the L1 or pixel distance between them.
Putting All Together
It’s time to put all the components of Pix2Pix together including: the U-Net generator, patchGAN discriminator, and pixel distance loss term.
We put our input into the U-Net generator, and it generates some output. Then gets concatenated along the channel dimension with the original real image, that is out real input image that was used for conditioning.
That goes into the patchGAN discriminator and that outputs a matrix of different values, a classification matrix (0,1) of how real are or how fake different parts of that image look.
Then for the discriminator’s loss, the output of concatenating the generated output with the real input will be compared to the fake label matrix, which is a matrix of all zeros because a discriminator will succeed if it classified every single patch of that image looked fake, was zero.
If it’s a real outputs, then the discriminator will want to compare its classification matrix with the real label because they want to get as close as possible to all 1s in this classification matrix in his predictions.
Again, as input, these were concatenated along the channel dimension.
For the generator’s loss, it’s still the discriminator looking at the generated output concatenated with that real input along the channel dimension. But instead of a matrix of all zeros assimilable is looking at the real labeled matrix of ones because the generator wants a discriminator to think that every single patch of it’s generated image looks real. But that’s not it for the generator’s loss.
That’s just using BCE loss between these two matrices. The generator also has this pixel distance loss term that’s looking at how different it’s generated output is to the real target output multiplied by some waiting Lambda there. That’s the full generator loss.
Pix2Pix Advancements
many different improvements and extaintions for paired-to-paired image translation (Pix2PixHD and GauGAN) have been raised up. And these includes high resolution images and also very explicitly as an image editing.
So first another model came out that works on higher resolution images, so it’s called Pix2PixHD, It operates on much higher resolution images and includes a lot of modifications that make it significantly better. I recommend checking out the Pix2PixHD paper and an optional notebook if you want to be doing image-to-image translation.
what you can do is you can have the segmentation mask of someone’s face and you can adapt that mask however you want and be able to generate a different type of face.
And actually I think what’s being done here is it’s not even changing the mask. You actually just have lots of different possible faces that you can generate from this one mask, so that’s pretty cool too. You can definitely do the version where you adopt the mask using Pix2Pix. another really really cool model that’s come out is called GauGAN.
And what’s cool about GauGAN is that it uses some of the advances in StyleGAN (i.e., it uses AdaIN to take in the segmentation map, and for informing styles).
