What do CycleGANs do?

Unpaired Image to Image Translation Using Cycle-Consistency Adversarial Networks.

What does that mean?

The goal of CycleGANs, is to learn to automatically translate an image from one collection to another given any two unordered image collections, X and Y.

How does this work?

Using:

• two Generative Adversarial Networks: model G and model F.
• two unordered image collections*: X and Y
• one Cycle- Consistency Loss with three parts: modelGLoss, modelFLoss, and cycleLoss function

Note: While we use unpaired image collections, we assume there is some relationship between the images in the collections. Ideally they are different renderings of the same structure.

The significant developments CycleGANs come from solving the previous GAN variations problems.

Origin of CycleGANs

Prior to CycleGANs, there was no way for the computer to learn to generate photos unsupervised. The first step to developing CycleGANs came with Generative Adversarial Networks in 2014.

2014 Generative Adversarial Networks by Ian Goodfellow et al., https://arxiv.org/abs/1406.2661

These results demonstrated promising results of using an Adversarial Network to generate images.

Three years after GANs were developed, a model, Image-to-Image Translation with Conditional Adversarial Networks, was released marking the second significant breakthrough in developing CycleGANs.

2017 Image-to-Image Translation with Conditional Adversarial Networks

by Philip Isola et al., https://arxiv.org/abs/1611.07004

Paired image-to-image translation, or PIX2PIX, allows images to be translated from one form of the image to another.

How? By mapping pixels from one image to match the pixels from the other image. However, it only works for paired image datasets which are expensive, time consuming, and even impossible to find.

CycleGANs solves the problem of paired data.

2017 Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks

by Jun-Yan Zhu et al., https://arxiv.org/abs/1703.10593

Unpaired Image-to-Image Translation Using Cycle Consistent Adversarial Networks is a mouthful, let’s call it CycleGANs.

CycleGANs takes two collections of images, X and Y, and learns to translate an image from collection X to collection Y. It does this by using two GANs that battle against each other using a cycle consistent loss function.

The CycleGANs model has been successful with a variety of applications as seen below.

While CycleGAN image quality is not state of the art, solving the problem of needing paired image data is.

Why Do We Care?

One of the founding fathers of Machine Learning, Yann LeCun, said, “There are many interesting recent developments in deep learning… The most important one, in my opinion, is adversarial training, also called GAN for Generative Adversarial Networks. This, and the variations that are now being proposed, is the most interesting idea in the last 10 years in Machine Leaning.” Considering what Yann LeCun said, the fact CycleGANs arise the one of the most recent variations, and a CycleGAN model uses two GANs simultaneously, learning to generate photos with unpaired image datasets.

CycleGAN architecture

The CycleGAN model consists of two generators, two GAN models, and one cycle consistent loss function. We will discuss all of the three models in more detail.

Generator Model

There are three main stages that happen in the generator model: encoding, transformation, and decoding.

Encoding — extracting features via a convolutional network, relu layer, batch normalization layer → gives us a feature vectorA

Transformation — layers of combining nearby features of an image, allow us to save layers and return to the original image, use resnet blocks → feature vectorB

Decoding — deconvolutional layer takes feature vectorA and feature vectorB and build back the low level features, then back to an image in the new domain.

Discriminator Model

The discriminator has two main stages.

Stage one - extract the features from the image using convolutional networks

Stage two - decide whether or not these features belong to images from the training dataset

It’s goal is to take an image as input and tries to predict if it is an original image or a synthetic image from the generator.

Model Build

This model needs to work in both directions A→B and from B→A.

To do that we have two Generator Models: GeneratorA→B and GeneratorB→A.

and two Discriminator Models: DiscriminatorA and DiscriminatorB.

CycleLoss

Goal of the cycle loss: Get the generated image back to original image, thus minimizing the loss.

Putting it all together we achieve our cycle consistent loss function.

Cycle Consistent Loss

The Cycle Consistent Loss Function is made up of three parts.

1. The loss from GAN model G
2. The loss from GAN model F
3. The Cycle Loss Function tying together model G and model F

It is important to note that the Cycle Loss Function is more important than the loss from the two GAN models because the Cycle Loss Function is what allows CycleGANs to work with unpaired data. To properly represent the importance of the Cycle Loss Function, we give it extra weight by multiplying it by Lambda.

Bad Results

While CycleGANs is an unsupervised method, in order to successfully carry out applications such as object transfiguration, semi-supervised data may be required.

Why would certain applications need partial supervision?

The original paper touched on this saying, “Some failure cases are caused by the distribution characteristics of the training datasets. For example, the horse → zebra example has got confused, because our model was trained on the wild horse and zebra synsets of ImageNet, which does not contain images of a person riding a horse or zebra.”

If you look at the four images above of “wild horses” you will notice there are no humans in the training set. So, when the model worked to translate the input image of a “wild horse” to a zebra, it recognized the entire focus of the image, man and horse, as the horse and the model ends up creating a joint man-zebra.

This happens because the model works to recognize the structure of the image from one collection and translates it to the structure of another image. The structure is defined by the curves and edges that constitute the image. When the collections obtain images with different structures, i.e one horse, horses and man, cougar and zebra, there will be limited success as they are clearly structurally different. These extra objects created a strong enough structural difference in the image collections which resulted in the discriminator accepting images that have poor similarity the photos in the zebra training set. This implementation uncovers one of the weakness of the CycleGANs model.

To improve the quality of these results, we could work to ensure that the two collections are more structurally similar by using for example a collection X of horse profiles, and a collection Y of zebra profiles.

Good Results

The implementation of CycleGANs for season transfer Summer Yosemite ←→ Winter Yosemite produced highly successful results. Ideally with CycleGANs, although we do not need to know the relationship between the image collections, we will have two unpaired image collections X and Y that are strongly structurally similar as seen in the Summer ← →Winter Implementation below.

Our Summer ← → Winter Implementation is working with landscape photos that have roughly the same structure, all being from Yosemite, the main work to be done is translating the pixels between these two image collections. When the main task at hand is translating pixels, versus translating structure and pixels, the results are more successful due to the fact we already have state of the art GAN models like pix2pix[1].

Conclusion

It is important to recognize that while models such as pix2pix produce images with state of the art quality, the requirement of paired image datasets is a massive handicap. Paired image datasets are time consuming to curate, they are expensive to curate, and it is likely impossible to find the exact dataset required to carry out of the desired implementation. CycleGANs solves that problem of paired data opening a brand new branch in Machine Learning: unsupervised image to image translation.

Resources

[1] Pix-to-Pix Supervised GANs

Isola, Phillip, et al. “Image-to-Image Translation with Conditional Adversarial Networks.” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, doi:10.1109/cvpr.2017.632.

[2] Pix-to-Pix Unsupervised with CycleGANs

Zhu, Jun-Yan, et al. “Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks.” 2017 IEEE International Conference on Computer Vision (ICCV), 2017, doi:10.1109/iccv.2017.244.

[3] Why Cycle GANs are Important and Useful

Likahovich, Olga, et al. “Learning Image to Image Translation with CycleGANs” Microsoft Developer Blog, 2017.

https://www.microsoft.com/developerblog/2017/06/12/learning-image-image-translation-cyclegans/

[4] Generative Adversarial Networks

Goodfellow, Ian, et al. “Generative Adversarial Networks.” Cornell University Library, 2014.

https://arxiv.org/abs/1406.2661

[5] Generative Models Lecture Stanford

Li, Fei-Fei, et al. “Generative Models.” Stanford University School of Engineering, 2017.

https://www.youtube.com/watch?v=5WoItGTWV54