GANs Part8: StyleGAN
Analyze and implement the key advancements of StyleGAN, techniques associated with StyleGAN, Investigate the controllability, fidelity and diversity outputs.
StyleGAN Overview
It’s not only the state-of-the-art GAN, but also an inflection point improvement in GAN, particularly in its ability to generate extremely realistic human faces.
Let’s start by going over primary goals of StyleGAN
- Producing high-quality (high-resolution) images.
- Increasing diversity of images in the output.
- Increasing control over image features. I.e., adding features like hats or sunglasses or mixing styles from 2 different generated images together.
Getting high resolution that fools the eye is a huge feat that was very difficult to achieve. This was mainly due to small model capacity, low resolution datasets, and the challenge of high resolution just really wasn’t tackled until ProGAN (StyleGAN predecessor, 2018).
These generated faces from 2014. Look like bad sketches there, greenie an unlikely to convince anyone of their authenticity.
look at this more recent high-resolution face generated by StyleGAN in 2018, if you saw this image without any context, would you be able to guess that she wasn’t a real person? It would seem pretty apparent that's StyleGAN has achieved its goal of greater fidelity.
StyleGAN also wants to increase the control over your image feature. And this can be mixing the style of one image into another.
Where the upper-left face is a mix from the below and to the right where you can see hair color and styles coming from the picture from the right and various facial features coming from the face on the bottom.
On the bottom right corner, is a mixing of styles from the woman above, as well as styles from the boy on the left.
Control could also mean adding accessories such as glasses and style. StyleGAN accomplishes this by disentangling the latent space.
The word Style in the context of image generation is any variation in the image. And you can think of these variations as generally representing the look and feel of different levels in the image.
These different levels could mean larger, coarser styles such as general face shape or facial structure to finer, more detailed styles which are like color their hair or placement of certain wisps of hair.
The StyleGAN generator, is made up of blocks where earlier blocks roughly aligned with coarser features, e.g., facial structure or pose. And nearer to the output, finer details styles e.g., hair color and eyebrow shape, are controlled.
Progressive Growing (PG).
PG in StyleGAN, is trying to make it easier for the generator to generate higher resolution images by gradually training it from lower resolution to higher resolution images.
It starts with a very blurry image to generate a 4 by 4 image (only 16 pixels) to then a much higher resolution image overtime.
At the very beginning, the generator just needs to generate this 4 by 4 image, and the discriminator needs to evaluate whether it’s real or fake, and to make it not so obvious what’s real or fake, the reals will also be down sampled to a 4 by 4 image.
In the next step, everything is doubled, so the generated image is 8 by 8. It’s a higher resolution image than before, but still an easier task than a super high-resolution image, and the reals are also down-sampled to 8 by 8.
The discriminator will now take this 8 by 8 image and have this extra convolutional layer to take that in. The generator conversely, will also have this 8 by 8 extra convolutional layer that will then be able to generate this higher resolution image. By 8 by 8 for the convolutional layer (the expected output on generator and input for the discriminator, not the filters), we’re not looking at channels.
Over lots of different growing periods, the generator will be able to generate super high-resolution images, and the discriminator will look at that higher resolution image VS real images that will also be at this high resolution, so no longer down-sampled and be able to detect whether it’s real or fake.
PG: The Generator.
What happens initially when you want to generate a doubled size image, we could use nearest neighbors filtering, these are not learnable parameters, it’s just very basic up-sampling with 99% and do 1% of taking this into a convolutional layer, producing an 8 by 8 resolution image. So, we’ve some learnable parameters, but you only weight this direction by 1%.
Over time, you start to decrease the up-sampling part and to increase this 1% to rely more on the learned parameters. I.e., 50–50, and this image may start to look more like a face.
Until you completely do not rely on this up-sampling, and you just rely on these learned parameters, and this will just be part of the network, and the up-sampling section will not be there anymore.
More generally, you can think of this as an alpha parameter that grows over time where alpha starts out as 0, then alpha grows like 0.1, 0.2, 0.3 all the way up to 1. When alpha= 1 then the up-sampling = 0.
PG: The Discriminator.
For the discriminator, there’s something fairly similar, but in the opposite direction.
What’s happening in the discriminator is the reverse and with the same alpha that gradually uses these learned parameters as well. Of course, the discriminator takes in the image instead of generating the image at the very end, gives you just one output of real or fake and essentially a probability between 0 and 1.
Noise Mapping Network (NMN)
let’s take a look at the structure of the NMN, the reasons why it exists, and finally where its output (W) actually goes.
Get AI & Data Science 604’s stories in your inbox
Join Medium for free to get updates from this writer.
NMN is a simple NN composed of 8-FC layers with activations in between, it takes ε (or Z — 512 in size) and maps it into an intermediate noise vector (W) (512 in size). So, it just changes the values.
The motivation behind this is that mapping Z will get us a more disentangled representation. Entanglement in Z-Space is when Z doesn’t really map onto output features in one-to-one way. I.e., If we’re to change one of these Z vector values, we’ll actually change a lot of different output features. And this is bad, because this doesn’t allow for a lot of fine-grained control (feature level control) in the resulting image.
It wouldn’t be great if you’re trying to change the eyes of someone and you suddenly change their skin color.
Z-Space is often entangled, because the real data has a certain probability density. Meaning that there is a probability density around the actual features of the sample certain image e.g., having glasses on or not, having a beard, color of hair, eyes, or certain age associated with them, etc.
But because Z has normal prior, where you draw all values from this normal distribution. It can be hard for it to map this normal distribution to the correct density distribution of all output features. So, it’s a quite challenging for Z to be able to model this entire space of all different types of features associated with those images.
It will try to find some way to twist itself trying to map onto these desired features. And it often twists itself in a way that probably doesn’t quite make sense to us. So, this is kind of unrealistic for it to be completely disentangled, (be completely one-to-one mapping). So, it will probably learn a complex entangled set of mappings to achieve that.
However, having W, can give it a little bit of room. So, it can allow for that space (W Space) to match the real data’s density, and this makes it easier to learn one-to-one mapping.
So essentially Z doesn’t need to be as fixed to the statistics of the training data. But can instead learn these factors of variation that are now linear and easier to generate from. And this will reduce the chance of your styles being too closely correlated to ultimately helps to control and map features.
Where does NMN fit into this architecture?
In a traditional GAN generator, you taken noise vector (ε) into the generator, then the generator outputs an image.
Now in StyleGAN, ε is handled a little bit differently, so instead of feeding it directly into the generator, it goes through a mapping network to get an intermediate noise vector (W), then gets injected into the StyleGAN generator (multiple times) to produce an image. This adds stochastic variations onto the image.
Z is no longer fit in directly, but instead, it’s going through this NMN, and W is actually imputed at multiple different places along the network.
So instead of inputting W at the very beginning, what the model starts with is actually just a constant value. So, we start with just a constant value (size of (4 by 4 by 512) that doesn’t change for any generated image. So, any changes to any image being generated will occur where Z is being introduced. Which is in all of these blocks.
Adaptive Instance Normalization (AdaIN).
Within each block, there’s an up-sampling layer and 2 CONV layers to help learn additional features, and also to double the image in size. But this is not all of it, AdaIN actually comes in after each CONV layer.
In general, normalization helps to accelerate and stabilize the training process, as it takes the outputs from previous layer, and put them around 0 and 1.
The adaptive styles are coming from W, which is inputted into multiple areas of the network.
W (inform styles) is not directly inputted in the AdaIN layers. Instead, it goes through 2 FC layers and produces 2 parameters, ys (scaling parameter), and the other is yb (bias parameter).
And these statistics (ys and yb) are then imported into the AdaIN layers.
After doing IN step (The middle part in the formula below), then we multiply these values (which are now normalized around 0 and std of 1) by ys, then we add yb term. Which means we are to re-shift and rescale the values based on these statistics that are extracted from W. And they are called adoptive because if W is changed, those extracted values will be changed.
Now, we’ve talked about all the main components of StyleGAN, the last variation over the traditional GANs is random noise injection.
Random Noise Injection
The random noise injection occurs before every AdaIN block. To do this, you need to create a noise tensor that is the same size as the current feature map (image).
The noise tensor is not entirely random; it is initialized as one random channel that is then multiplied by learned weights for each channel in the image. I.e., imagine an image has 512 channels and its height and width are (4 x 4). You would first create a random (4 x 4) noise matrix with one channel. Then, your model would create 512 values — one for each channel. Next, you multiply the (4 x 4) matrix by each one of these values. This creates a “random” tensor of 512 channels and (4 x 4) pixels, the same dimensions as the image. Finally, you add this noise tensor to the image. This introduces uncorrelated noise and is meant to increase the diversity in the image.
New starting weights are generated for every new layer, or generator, where this class is used. Within a layer, every following time the noise injection is called, you take another step with the optimizer and the weights that you use for each channel are optimized (i.e. learned).
