GANs — Part4: Conditional and Controllable Generations
Control GANs by adding conditional inputs to modify the features of the generated images. Manipulate outputs by adjusting z-vectors and build conditional GANs capable of generating examples from specified categories. Additionally, explain the concept of disentanglement in GANs.
Introduction
Over the previous blogs, we’ve discussed how GANs work and how to build them to produce examples that mimic your training dataset.
GANs had been getting better and better, but we don’t have much control over the generated items, which is called unconditional generation.
This blog is about how to adapt your model inputs — without changing your model process — in order to control the output and get examples from a particular class or for those examples to take on certain features.
Conditional Generation Intuition.
As a quick recap with unconditional generation- getting outputs from random classes-. You can think of it as a gumball machine, where you input a coin and you get a random gumball. If you want a red gumball, you’ve to keep spinning coins till getting it.
In this example, the coin is the random noise vector that your GAN uses for generation, and the gumball machine is the generator. You can’t control what exact color of output you will get.
On the other hand, conditional generation allows you to ask for a specific class and you get what you ask for.
It’s like a vending machine. You input a coin but with a code for a specific item. e.g., A2 for a red soda.
Here the coin and the code are the inputs for the GAN and the vending machine is the generator and the soda is the generated output.
Note, you still don’t control certain features of the soda bottle, I.e., you can’t get the one with the latest expiration date or the bottle that’s least damaged or the one that’s filled up the most, you just get a random red soda. But it is a red soda, not anther.
With conditional GANs, you get a random example from the specified class.
Conditional VS Unconditional Generation.
With conditional generation, you can get generated examples from the classes you choose, while with unconditional generation, you get examples from a random class.
As a result, with conditional generation, we’ve to train our GAN with labeled datasets to learn how to produce examples from desired classes while unconditional generation doesn’t need any labels, just need a pile of real examples.
Conditional Generation Inputs.
you need to have a label dataset and somehow pass that class information to both the generator and the discriminator during training.
With unconditional generation, the generator needs a noise vector to produce random examples.
For conditional generation, we also need a vector to tell the generator from which class the generated examples should come from. Usually this is a one-hot vector, which means that there are zeros in every position except for one position corresponding to the desired class. i.e., let’s say all of these different cells in the one-hot class vector are different dog breeds and this 2nd cell corresponds to a husky. If we had 1 somewhere else and 0 on the husky cell, then we would want it to produce a different class; e.g., a golden retriever.
The noise vector is the one that adds randomness in the generation, to produce a diverse set of examples. But now it’s a diverse set of examples conditioned on the certain class. There’s one-hot vector to control what class to generate.
The input to the generator in a conditional GAN is a concatenated vector of both the noise and the one-hot class information (one huge vector).
e.g., with a class vector here representing a husky breed, the generator will ideally produce a husky dog from one noise vector.
But if we change that noise vector, it should produce a different husky dog, while the class information stays the same.
But how do you ensure the generator actually produces a husky? Why doesn’t it just produce any random dog?
Well, that’s actually because the discriminator will also be given this class information.
The discriminator in a similar way, will take the examples paired with the class information as inputs to determine if the examples are real/fake representations of that particular class. e.g., with class of golden retriever, but in image of a beagle dogs, so not a golden retriever being fed in. This discriminated arches actually say this is fake because that is not a real looking golden retriever, even though it’s a very real looking image of a beagle.
As a result, for the discriminator to predict that an example is real, it needs to look like the examples from that class in the training data set. The real images will have the correct class label, and this encourages the generated images to take on the right class as well.
How to add the class information?
Well, the image is fed in as 3 different channels (RGB) or just one channel if it’s a gray-scale image.
Then the one-hot class information could also be fed in as additional channels where all the channels take on values of all zeros.
In contrast to that one-hot vector, these are typically much larger matrices, all these white blocks of the same height and width as the image take on values of all zeros. Whereas this black one will take on values of ones.
There are many other less space consuming ways of doing this, e.g., compressing this information in another format. You can even create a separate NN head to do that, which would be prudent if you had many different classes.
Controllable Generation.
How to control what features you want in the output, even after the model has been trained and how it compares with conditional generation.
Controllable generation allows you to control some of the features that you want in your output examples. I.e., with a GAN that generates faces, you could control the age of the person’s looks in the image or if they have sunglasses or the direction they’re looking at in the picture, or they’re perceived gender.
You can do this by actually tweaking the input noise vector Z, that is fed to the generator after you train the model.
i.e., with the input noise vector Z, maybe you get this picture of a woman with red hair.
If you tweak one of the features from this input noise vector, you may get the same woman but with blue hair this time. Maybe because this 1st element, represents changing hair color.
Conditional Generation VS Controllable Generation.
These terms are typically what researchers mean if you check out the papers, though it’s not as clearly delineated.
Sometimes controllable generation can definitely include conditional generation because you’re still controlling the GANs in some way.
Vector Algebra in the Z-Space
How to manipulate noise vectors in order to control desired features in the outputs.
Controllable generation and interpolation are somewhat alike. With interpolation, you get intermediate examples between 2 generated observations.
Just to be clear, Z1 and Z2 are the 2 dimensions in this Z-space and there’s a noise vectors V1 and V2.
Say, when you feed V1 into the generator, will produce the 1st image, and when you feed V2 into the generator, will produce the other image.
Get AI & Data Science 604’s stories in your inbox
Join Medium for free to get updates from this writer.
If you want to get intermediate values between these 2 images, you can make an interpolation between their 2 input vectors, V1 and V2 in the Z-space.
This interpolation is often a linear interpolation. there are other ways to interpolate between these 2 vectors. Then you can take all these intermediate vectors and see what they produce from the generator.
Controllable generation also uses changes in the Z-space and takes advantage of how modifications to the noise vectors are reflected on the output from the generator. e.g., with the noise vector, you could get a picture of a woman with red hair and then with another noise vector, you could get a picture of the same woman but with blue hair.
The difference between these 2 noise vectors is just this direction in which we’ve to move in Z-space to modify the hair color of your generated images.
In controllable generation, the goal is to find these directions for different features you care about. E.g.., modifying hair color. With this known direction d in Z-space, you can now control the features on the output.
This means that if you then generate an image of a man with red hair produced by the same Generator g, with this input noise vector V1, you can modify the hair color of this man in the image by adding that direction vector d you found earlier to the noise vector, creating this new noise vector here, V1 + d, passing that into your generator and getting a resulting image where his hair is blue.
Challenges with Controllable Generation.
Controllable generation has a couple of challenges. Specifically, feature correlation and the alpha space in Z-space entanglement.
Feature Correlation.
When different features have a high correlation, it becomes difficult to control specific features without modifying the correlated ones.
I.e., ideally you want to control single features e.g., the amount of beard on a person’s face and if features in the data set don’t have a high correlation, you’d be able to take this picture of a woman and add a beard to her by moving in some direction in Z space.
However, it’s very likely that in the data set you use for training, features like the presence of a beard and how masculine the face looks will be strongly correlated. If you want to add a beard to the picture of a woman, you’d end up modifying many more features on the output.
But this is not desirable if you want to just change one feature about someone.
Entanglement in the Z space.
When the Z space is entangled, movement in the different directions has an effect on multiple features simultaneously in the output.
Even if those features aren’t necessarily correlated in your training data set. This is just how the noise space was learned (to be very entangled).
In this entangled Z-space, when you control if a person in the output has glasses. I.e., you also end up modifying whether she has a beard.
Or, when you try to modify her apparent age, you’ll end up also changing her apparent eyes and hair color too.
That’s not quite desirable. The same happens with other uncorrelated features. This means that changes in some of the components of the noise vectors change multiple features in the output at the same time and this makes it difficult to control the output.
This is a very common problem when the Z space doesn’t have enough dimensions relative to the number of features in the output, then it actually can’t map things one-to-one.
Classifier Gradients
A simple method used to find that direction using the gradient of trained classifiers. To find the direction in the z-space that modifies certain features on the output, e.g., the presence of sunglasses. You could use a trained classifier that identifies if a person in a picture has that feature.
you could take a batch of noise vector Z, that goes through the generator to get some images, and pass these images through a classifier, then use this information to modify your Z vectors without modifying the weights of the generator at all (the generator weights are frozen). When training is done, and you modify your Z vectors by moving in the direction of the gradient with the costs. That penalizes the model for every image classified as not having sunglasses.
Then, repeat this process until the images are classified as people with sunglasses.
However, there’s also the downside. You need a pre-trained classifier that accurately detects the feature that you want to control. You can also train your own. I.e., if you wanted to classifier that detected beards, you might have to train that on your own if that isn’t available off the shelf. you should always check if something is available out, because this could be a really simple and cool way to allow you to control.
Entanglement
Imagine these noise vectors (v1, v2) are coming from a disentangled z-space.
then each of these positions would correspond to a single feature in the output.
I.e., this 1st element, of the noise vector may correspond to hair color, and the 2nd dimension may correspond to hair length, and so on.
If you wanted to change the hair color in a picture, you would literally just change the 1st element on its noise vector or move in the Z1 direction on that Z space; and that’s all you have to do, change the hair color.
Then for hair length, you just move in/change the Z2 dimension (2nd position). Also, it’s quite possible that these noise vectors are disentangled with respect to just these 2-d features and the rest of the values don’t mean anything.
Typically, we want the noise vector size to be larger than the number of features you want to control. This makes it easier for the model to learn as it allows these values to take on different things over time during training.
I.e., if you only wanted to control these 2 features like hair color and hair length, then it’s probably prudent to make your noise vector larger than just 2-dimensions. These other values won’t control a specific feature, they just help the model adapt to in training.
Because the components of the noise vectors in the disentangled z-space allow you to change those features that you desire in the output, they’re often called latent factors of variation, where the word latent comes from the fact that the information from the noise vectors, is not seen directly on the output, but they do determine how that output looks. Sometimes you might hear noise vectors being referred more generally as latents.
Then factors of variation means that these are just different factors like hair color and hair length that you want to vary only one factor that you are varying, not anything else.
Essentially, a disentangled z-space means that there are specific indices on the noise vectors, these specific dimensions that change particular features on the output of your GAN.
I.e., if a person on the image generated by a GAN has glasses or not, or whether she has a beard or some features of her hair. Each of these cells will correspond to something that you desire to change, and you can just change the values of that dimension in order to adapt glasses, beard, or hair.
A crucial different between a disentangled z-space and an entangled z-space is that with a disentangled z-space, when you control one of the features on the output. e.g., glasses, the other features remain the same. The beard and hair will remain the same. Or if I change whether someone has a beard, the glasses and hair will stay the same.
What this means is that with a disentangled z-space, you’re much more likely to be able to add a beard to a person that looks very feminine without changing her hair or her facial features.
How to encourage your model to use disentangled z-spaces?!
One way to is to label your data and follow a similar process, the one used for conditional generation.
But in this case, the information from the class is embedded in the noise vector, I.e., you don’t need this extra one-hot class information. However, using this method could be problematic for continuous classes. Imagine having to label thousands of human faces with the length of their hair. Having a few different classes, might actually nudge your generator in the right direction.
Another way to encourage your model is to use a disentangled z-space without labeling any of your examples.
Instead, you add a regularization term to the loss function (e.g., BCE , W-loss) to encourage your model to associate each index from the noise vectors, 2 different features on the output. This regularization could come from classifier gradients and they’re also much more advanced techniques to do this in an unsupervised way, meaning without any labels.
