GANs Part12: CycleGAN
Understanding how unpaired image-to-image translation differs from paired translation, their key difference necessitates a different GAN architecture, how CycleGAN is implemented using 2 GANs to transform between horses and zebras and vice versa!
Introduction.
We’ll build on the previous blog on image translation, except you don’t actually have to know the exact correspondence between 2 paired images. In fact, all you need is 2 piles of images of 2 different styles and your GAN will figure out the mapping from there.
Actually, it’ll be 2 GANs figuring out that mapping. One will go one direction and the other will go the other direction.
The insight that makes this crazy task work is to have one GAN take in a horse image and make it into a zebra, and then have the other GAN take that fake zebra and generate a horse.
Since our GANs are only expected to generate different styles of the same image, that last generated horse from the 2nd GAN should look like the original real horse you put into the 1st GAN and the interaction between these 2 GANs form a cycle (cycleGAN).
Unpaired Image-to-Image Translation
We’ve seen a paired image to image translation, where we’ve input: output in pairs.
We can use edge detector and go from a realistic image to those edges to get that paired data set pretty easily, and that’s great when we’ve algorithms to get that paired training data.
Unpaired image to image translation, is not so straightforward. I.e., If you want to change a horse into a zebra or have a painting by Monet and you would like to turn it into a photo, this generation is much harder because it’s even harder to get such a training data that was paired. I.e., for every single photograph something that looked like it was painted by Monet or vice versa.
The paradigm between these 2 image-to-image translation tasks, is that for one of them, you’ve these paired images that match onto each other.
but you don’t necessarily have this correspondence all the time, and so an unpaired image to image translation, you actually just have 2 piles of 2 different image styles, x and y.
One pile which is maybe realistic photos, the other maybe it’s Monet or someone else. Or you can have a pile of winter looking images VS summer scenes or a pile of horses in one or zebras in another, where you don’t have a one-to-one correspondence at all. Using these 2 piles, from x and y, you want your model to learn general stylistic elements from one pile to the other and transform images from one to another and vice versa.
Here photograph to “Monet” and maybe you can have a paint to “Van Gogh” and vice versa. But what’s a key here, is that you’re making this photo of a poppy field look like it was drawn by “Monet”, but it’s still a field of poppies. Which means there’s still some type of content that is preserved. It’s just the stylistic elements that are changed. That’s pretty key in thinking about this translation task because there are commonalities, stylistic differences and unique things about each pile where you want to be able to tease out what’s common, and keep those common elements, and only transfer those unique elements that are to each pile.
Concretely, with zebras or horses, you’ve a pile of zebras, and the horse you want to generate from this zebra image should still be in the same orientation.
You just want those stripes to go away. The model’s goals are to learn that mapping between these 2 piles of horses and zebras into figure out those common elements and unique elements.
The common elements are often known as content, which is common to both of these piles and styles often referred to what is different between them. I.e., here, content is the general shape of the horse or zebra and the background, and style is obviously stripes in zebras, as opposed to a single color or less of a pattern on horses.
In summary unpaired image to image translation is really about finding the common content of these 2 different piles as well as their differences.
CycleGAN Overview.
At this point, unpaired image translation might sound impossible. You don’t have pairs of images, so how your model is really supposed to know what to generate? How is it supposed to take the zebra and turn it into a horse?
But, there’s a really cool insight that makes CycleGAN work. Which is if you transform an image of one pile into the style of the other pile, then transform it back, and it’s still fake It’s taking in this fake horse to generate another fake zebra.
As you’re only changing styles and not the content of the image, these 2 images should be the same (distance loss).
This content preservation while mapping from one pile to another and back again, is known as cycle consistency because the translation forms a cycle between these 2 piles. One simple way of creating this cycle is actually to use 2 different GANs. One is going from zebras to horses (ZH), and the other to go the opposite direction (HZ).
Together these 2 GANs have cycle consistency to make realistic unpaired translation possible. The adversarial part of the GANs just discriminates it for both of them, will ensure realism in the images, while the cycle consistency part is really in charge of getting the content to be preserved while only moving around the styles.
From the earlier example, you would use the GAN Z-H, or zebra to horse, to generate that fake horse, then you would use the 2nd GAN horse to zebra (GAN H-Z), to generate from that fake horse into a fake zebra.
The architecture for these GANs are things you’ve already learned about. So like in pix2pix, the discriminator is a PatchGAN.
But of course, there are 2 GANs, so there are 2 PatchGANs which will look at patches in the image and output and matrix of values, and the generator draws from prior work on image super-resolution in style transfer.
So basically from both U-net, which is from Pix2Pix as well as DCGAN. CycleGAN basically borrows concepts from U-Net, in that there’s down sampling and the encoding section, and then there’s some decoding section where there’s up sampling. These various encoding and decoding box are composed of CONV layers with batch norm and ReLu.
In addition to this U-Net framework, CycleGAN also expands up bottleneck section with more convolutional layers like from the DCGANs generator. Since the generator in CycleGAN takes an image as input (X), it’s quite possible a regular U-net would actually have done well. But it is probably not the limiting factor in terms of what makes this entire thing work. It’s just an improvement, but there are others found to be pretty useful.
The bottleneck section here also uses additional skip connections. They help with adding additional layers and image transformations by allowing the model to learn identity functions.
CycleGAN: 4 Components
As expected from 2 GANs, you have 4 components, 2 generators, and 2 discriminators.
Starting in one direction, taking in a zebra image, mapping that into a horse using the Z-H generator. Then the H discriminator is in charge of looking at real images of horses from that one pile of horses, as well as those fake horses coming into and determining how real or fake, they are. And it is a PatchGAN, so it is outputting a classification matrix of the patches it sees in the image.
In the opposite direction is similar, just going from horses to zebras, and the discriminator will be focused on the zebras and how realistic those are.
Cycle Consistency.
Cycle consistency is an extra loss term for CycleGAN (each of the 2 GANs).
For the Z-H direction, Z-H generator will map a real image of a zebra to a fake horse, then H-Z generator will map that fake horse back to a zebra.
Cycle consistency expects the generated fake zebra to look exactly like the real one, as only styles should have been changed.
What we can do is to take the pixel difference between these 2 images and add that to the loss function to encourage these 2 images to be as close as possible. This also applies in the opposite direction.
We can now construct the entire cycle consistency loss by summing the pixel differences from both directions.
One is going Z to H to Z again, where you look at the difference between the real zebra and that fake zebra after going through the cycle, and the same goes for H to Z to H again. This constitutes the entire cycle consistency loss term for the generator.
How to add this to the adversarial loss?!
Since each direction uses both generators, we’ve just one optimizer for both generators. There’s only one loss term that both of your generators are using. From this cycle consistency loss is calculated and shared between the 2 generators. We sum the cycle consistency over zebras and horses in your training dataset and weight this by some Lambda term to get full cycle consistency loss term then add to generators loss.
Cycle consistency is a broader concept than it’s used in CycleGAN. It’s actually used across DL quite a bit. It helped with taxtual data augmentation, i.e., it has been used in back translation, from going French to English to French. Where you expect to get the same phrase back.
Get AI & Data Science 604’s stories in your inbox
Join Medium for free to get updates from this writer.
The CycleGAN paper also showed some cool ablation studies, where you’re cutting out various components of what you’re introducing in CycleGAN and seeing how the model does without those various components.
What if it only had cycle consistency loss?
With just cycle consistency loss doesn’t do too well.
CycleGAN operates on unpaired and so it’s really just looking out 2 piles. It’s not taking into consideration the pairing at all. But the reason to use a pair dataset, is to get a sense of how far away the output is from the ground truth. That’s a pretty cool way to evaluate and look at the ablation study of your model, even if you’re looking at unpaired translation, is to use a paired translation dataset.
What’s going on here is that the ground truth is taking in this realistic image and producing this segmentation output or vice versa, taking a segmentation output and producing this realistic image. These are the same pair here in different directions and the 2nd set is also the same pair just in different directions as well. With cycle consistency alone, these outputs are just not realistic at all. You probably get completely black squares, or some mode collapse going on, and there’s not much realism.
Adversarial loss from GANs really makes things look realistic, and so as expected without it, and with just cycle consistency, you don’t get those realistic outputs that you would want.
What if there’s no cycle consistency?
Well, the outputs actually look fairly realistic, except for there might be a little bit of mode collapse going on.
These 2 (marketed above in red) are very different input images, but the segmentation masks that are produced from them are very similar (a little bit of mode collapse) and that’s obviously not ideal.
While these images are somewhat realistic much better than if it is only CC loss but still look a little bit mode collapsing they have all the same features, is not as diverse as the real images.
Interestingly, we could use half of CC loss, but that also isn’t enough for the GAN to learn diverse and quality mappings.
What CycleGAN is trying to say is that with both the adversarial loss and CC loss in both directions. Then it really helps with these outputs with diversity, less mode collapse going on, and it looks fairly realistic.
Least Square (LS) Loss
LS loss is used to help with training stability, namely vanishing gradient problem, mode collapse and other issues resulting (from BCE loss) in the end of learning.
LS is a pretty simple concept from statistics. It’s a method that minimizes the sum of squared residuals. It tries to find the best fit line that has the smallest sum of squared distances between that line and all the points.
I.e., there’re 3 different points and you want to find to the best fit a line through them. what we do is to draw a line then try to find each points distance from a line. Specifically, we want to take a look at their squared distances to make it more sensitive.
Now, get the sum of squares as the total cost for each line that we tried to do, you minimize that sum of squares to find that best fit line.
How to translates this into GAN context?!
The line is our label of real or fake. From the discriminators point of view, a label of 1 is real and the equivalent of a point relative to that line is the discriminators prediction on a real image.
We want to do this across many different points, so we can take the expected value across many real images, and for fake images it’s the same but with a label of zero.
This is a very general way of putting those, with cycle GAN, you’re expecting G is taking in that real image and discriminator is evaluating that fake image. Then you did the same, you do this on multiple fake examples, which is just multiple z values passed into G, where z is perhaps different z images. You can simplify this by removing that 0.
On the generator side.The generator is wanting its fake outputs to look as real as possible; it will instead want to see how far away it is from one.
This might look similar to previous BCE loss function. But importantly, it isn’t very flat like with those sigmoids in BCE, which cause all of those vanishing gradient problems because it’s only flat with the discriminators predictions are exactly 1 for real or exactly 0 for fake.
For those that might be familiar at, this is also known as the mean squared error (MSE). In the context of cycle GAN, what this looks like is you’ve the CC loss term with the Lambda, but your adversarial loss is the LS loss.
Identity Loss
Identity loss is an optional loss term introduced in CycleGAN to help with color preservation in the outputs. It is used alongside adversarial loss and CC loss to guide the generators in CycleGAN. Identity loss helps preserve color in images and make the mapping more sense.
It ensures that putting an image of a horse into the Z-H generator, which is the generator that goes from zebra to horse, should ideally output the exact same image.
Identity loss encourages the generator to avoid changing images that are already in the desired style. I.e., if a horse image is input into a Z-H generator, the generator should output the same horse image.
If the generator alters the image (e.g., changing colors), this indicates a failure in maintaining the identity mapping, and the identity loss will help discourage such behavior.
In the overall loss function for the generators, identity loss is added to the existing losses:
- Adversarial Loss: Ensures the generated images are realistic.
- CC Loss: Ensures that images can be transformed back to their original style.
- Identity Loss: Helps maintain the original style for images that are already in the target domain.
Each loss term can be weighted with lambda parameters (e.g., λ1 for CC loss and λ2 for identity loss) to balance their contributions to the overall loss.
Without identity loss, the generator might produce distorted colors when trying to map between different styles (e.g., summer to winter scenes). This can lead to unwanted color changes, such as making a summer image appear too blue for winter.
With identity loss, the generator is encouraged to maintain the original characteristics of the input images, leading to better preservation of color and style.
Put-It-All-Together.
I’ll show the process with an example going from a Z-H, and the opposite. first input a zebra image to get a fake horse using Z-H generator.
The H discriminator then looks at this fake image as well as real images (It doesn’t know which one’s which) and outputs a classification matrix of how real or how fake it thinks those patches of images are.
What’s used here is least squares loss, and specifically for reals, this classification matrix is full of 1s and for fakes it’s full of 0s.
For the generator, this classification matrix is all 1s. In addition to least squares adversarial loss, we take the fake horse and feed it through the H-Z generator then compute the cycle consistency loss in this direction. And you do that by taking the pixel difference between the real input and this fake generated zebra because they really should look the same as you’re only supposed to be transferring styles between these 2 generators.
The same is done in the opposite direction as well, and so this is going from horses and zebras. Then back to horses for the cycle consistency loss and then of course their zebra discriminator here will also apply. If you do choose to use identity loss for your task, you also want to put your zebra through the opposite generator from horses and zebras.
Take the pixel difference (the identity loss) and do the same with horses going through Z-H generator.
There are so many losses in CycleGAN (6 total). least squares adversarial loss for both of GANs, cycle consistency losses going in both directions, identity loss for each of your generators there. And that makes up the entire generator loss function, which is for both of generators.
