GANs — Part1: Basic GAN.
An advanced DL class that generates realistic images, music and other things. It generates pictures of people that had never existed or make someone look younger or older. GANs can also turn a low-res video into a great-looking high-res video. On the supervised learning side, if we don’t have enough instances, GAN can synthesize more.
Agenda
Apps of GANs.
The intuition behind GANs.
How The Discriminator Works.
How The Generator Works.
Binary Cross Entropy loss function.
Putting all together.
To create a GAN, we need to implement 2 different NNs:
- The 1st is the generator (the art forger) who’s trying to mimic pieces of realistic artworks (e.g., Mona Lisa, Starry Night ...etc.).
- The 2nd is the discriminator (the art inspector) who’s looking at a pile of real famous art and also the fake art that’s forged by the art forger and trying to figure out which ones are real/fake and giving that feedback back to the generator to improve over time.
Real Life applications of GANs
We can see how GANs have progressed from black/white and rather inhuman looking faces in 2014 to much higher quality photorealistic faces in 2018.
They’ve been getting actually even better till nowadays. In early 2020 in video realistic GAN that could generate images that are so high resolution and look like professional photographs.
GANs can also perform image-to-image translation, which means they can take an image from one domain and transform it into another. E.g., transforming an image of a horse into a zebra and vice versa.
What’s really interesting is that we don’t need examples of a zebra and a horse doing the same things, and instead just transfer that style over.
GANs can help you draw. GauGAN model can take a rough drawing of a landscape and make it photorealistic. So, we can have set of different classes together e.g., a cloud, a mountain, or a lake. Then from this really rough sketch, GANs are able to produce something really photorealistic.
GAN can also take a still-life portrait, I.e., the Mona Lisa, and animate it using the motion of any real person’s face. They don’t even have to look like a Mona Lisa to play the part.
GANs don’t stop at 2D images, they can also generate 3D objects e.g., chairs and tables. These can be applied to fields like generative design where we can create cool furniture.
There’re diverse applications in medicine where we can use GANs for generating artificial medical data or detecting abnormalities in X-rays.
Several prominent companies have also started using GANs for a variety of applications.
Adobe is thinking about the next generation of Photoshop where novice artists can perform at an expert level. Google is using them for text generation also with images. IBM is using GANs for data augmentation, so using a GAN to generate synthetic examples to augment the dataset for a classifier downstream. Snapchat and TikTok put them to work in creative new filters which we’ve probably seen and used. Even Disney is using them for super-resolution.
The intuition behind GANs
GANs learn to create realistic images by making a generator (G) and a discriminator (D) compete against each other. That’s why they’re called adversarial. Both G and D models learn from their competition and force each other to get better, until the examples produced by G (the art forger) look realistic enough to fool D (the art inspector). But D will try to learn how to not get fooled, even by the closest replica.
We can see Starry Night and Scream look pretty good. Meanwhile D looks for a pile of both real famous paintings and the fake ones. And tries to tell which ones are real/fake.
To start this game, all we need is a collection of real images, e.g., famous paintings. At the very beginning of this game.
- G actually doesn’t know how to produce real looking artwork.
Additionally, G isn’t allowed to see the real images. It doesn’t know how this painting should even look. So, at the very beginning, G initially just paints a masterpiece of scribbles, and nobody ever told it what to do and what to generate.
- Also, D doesn’t know for sure what’s real and what’s fake.
But D is allowed to look at the real artwork. The real images are just jumbled up in a pile with the fake ones as well, and D doesn’t know which ones are real/fake. But we tell it as it learns which ones are real/fake and whether it’s right or wrong.
To start the competition, we train D using the real artworks. This way we get D that’s able to differentiate a poorly drawn image from the ones that are slightly better and also the real ones.
And when G produces many batches of paintings, G will know in what direction to go on and improve, by looking at the scores assigned by D.
This one looks a little bit more real, so G will start painting a little bit more realistically, not quite there yet, but almost.
D also improves over time as it receives more and more realistic images at each round from G (fake) and real images both, all jumbled up in a pile.
But essentially it tries to develop a keener eye as these images get better. when it says that this image created by G is 60% real. we tell it: “that it’s wrong that it’s not necessarily real, that it’s actually fake”.
After many rounds, G will start producing paintings that are harder to distinguish (if it’s impossible) from the real ones. And the game will end.
Now, we’ll deeply address each component individually.
The Discriminator (D)
D is a type of classifier, so I’ll start with a refresher on what classifier is and in probabilistic terms what classifiers learn.
The learning process can be summarized as follows. You’ve input features(X), and a set of classes(Y). The NN takes in those features and learns a set of parameters (θ). These parameters change overtime as the NN learns what each class looks like.
The parameters are trying to map (X) to (Y) by minimizing the differences between the predictions (Y^) and the true labels (Y).
And this is where a cost function (CF) comes in, and it’s computed by comparing how closely Y^ is to Y.
From CF, we can compute the gradients to update those parameters. Then, repeat this process till the classifier is in good shape.
let’s take a little ride into math land where the goal of D is to model the probability of each class. This is modeling the probability of class Y given input features X.
Now, bring it back to the GAN context. D is a classifier that inspects the examples and determines whether they’re real/fake. So, take in this fake Mona Lisa, and determine how fake this image is.
In probabilistic terms, D models the probability of an example being fake given X.
I.e., it’ll look at this picture and determined how far it is from the real one, giving it as feedback to G to improve its efforts.
The Generator (G) (The heart of GANs)
G is the one that we should invest in, hoping to achieve a high performance at the end of the training process.
Get AI & Data Science 604’s stories in your inbox
Join Medium for free to get updates from this writer.
If we trained G from the class of a cat, then it’s expected to do some computations and output a representation of a cat that looks real.
Ideally, we don’t want G to output the same cat every run. As, generating just one image is no fun and pointless. To ensure it will be able to produce different examples every single time, we’ll input different sets of random values, which is called a noise vector (ε).
ε is fed as input, sometimes with the class into G NN. Then G will compute a series of nonlinearities and return some features that look like a cat.
def get_noise(n_samples, z_dim):
return torch.randn(n_samples,z_dim, device=device)imagine this image has 1000 pixels, so the final layer of G should have 1000 nodes that represent each pixel’s value.
In another run, it can generate a sphinx cat, maybe in a funnel wrong, this savannah cat. These are all with different noise vectors.
let’s consider conceptually how G improves over time.
- ε is fed into G to produce a set of features (X^) that can pose a cat image.
- X^ is fed into D, which determines how real/fake it thinks, y^d representing D predictions.
- We can compute a CF that looks at how far the examples produced by G are being considered real by D. As G wants this to seem as real as possible.
def get_disc_loss(gen, disc, criterion, real, num_images, z_dim= 64):
fake_noise = get_noise(num_images, z_dim)
fake = gen(fake_noise)
disc_fake_pred = disc(fake.detach())
disc_fake_loss = criterion(disc_fake_pred, torch.zeros_like(disc_fake_pred))
disc_real_pred = disc(real)
disc_real_loss = criterion(disc_real_pred, torch.ones_like(disc_real_pred))
return (disc_fake_loss + disc_real_loss) / 2Basically, G wants y^ to be as close to 1 (real as possible). Whereas D is trying to get this to be 0 (i.e., fake).
- Then, use the difference between these 2 to then update the parameters of G, to get improved over time and know which direction to move its parameters to generate something that looks more real and will fool D.
Once you get a G that looks pretty good, we can save its weights.
Then we can load it back up then input some random ε into the saved G, it can generate different examples.
In the probabilistic term, G models the probability of the features of a cat image. E.g., leaking its paw or having cute whiskers are having different types of far, given this class of a cat (suppose we’re to generate one class which is a cat).
Binary cross entropy Cost Function (BCE)
BCE is used for training GANs. It’s especially designed for classification tasks, where there’re 2 categories.
Let’s break down each of BCE components. 1st at the beginning, we see a summation from 1 to m, as well as dividing by that m.
This basically just means taking the average over the number of examples (m) in the entire batch. I’ll go over what this negative sign means later on.
There are 2 different terms in brackets. Let’s look at each of them.
The left term, the true label (y) times the log of the prediction, which is h(x) parameterized by θ. To understand what this value is trying to get at, let’s look at some examples.
This term is mainly for when the prediction is just real (y=1), and where y=0 (fake), and whatever predictions we’ve (any), then this value actually results in 0 which means this term doesn’t matter.
If y=1 (real), and you’ve a really high h(x) that is close to 1, say 0.99, then you get a value that’s close to 0.
But in the case where y=1, but h(x) is 0 (thinking it’s fake, but it’s actually real), then this value is extremely large. This is largely caused by the log.
What this term says: “it makes it 0, if y=0 or h(y) is good, and it makes it negative infinity if h(y) is bad.”
Now looking at the 2nd term, it looks very similar.
The term is activated in this case y=0 only. As if y=1, then (1 — y) = 0 and whatever your prediction, this will evaluate to 0. And if h(x) gets close to 0, then this value is close to 0.
However, if it’s fake (0), but your prediction is really far off, and thinks it’s real, then this term evaluates to negative infinity.
Basically, each of these terms evaluates to negative infinity if for their relevant label, the prediction is really bad. That brings us to this negative sign.
If either of these values evaluates to something really big in the negative direction, this negative sign is crucial to making sure that is a positive (greater than, or equal to 0). Because for CF, what you typically want is a high value being bad, and your NN is trying to reduce this value as much as possible through backpropagation. Getting predictions that are closer, evaluating to 0 makes sense, because you want to minimize CF.
Now let’s look at what the loss function looks like for each of the labels over all possible predictions.
In this plot, prediction value on the x-axis, the loss associated with that training example is on the y-axis. In this case, the loss simplifies to the negative log of the prediction. When the prediction is close to 1, here at the tail, the loss is close to 0 because your prediction is close to the label.
When the prediction is close to 0 out here, unfortunately your loss approaches infinity, so a really high value because the prediction and the label are very different.
The opposite is true when the label is 0, and the loss function reduces to the negative log of 1 — y. When the prediction is close to 0, the loss is also close to 0. That means you’re doing great.
But when your prediction is closer to 1, but the ground truth is 0, it will approach infinity again.
Putting all covered concepts together
To train a basic GAN, we alternate the training on G and D.
For D, we get some fake examples (X^) produced by G from that input ε. And then those X^ and real ones (X) are both passed into D without telling it which ones are real /fake. Then D makes predictions Y^ of which ones are real/fake.
Or more specifically a probability of score of how fake/real each of these images are. After that, the predictions are compared using BCE loss with the desired labels for fake/real. And that helps update its parameters (θd). This only updates Weights of D, and not G.
For G, again it first generates X^. Then these X^ are again passed into D.
G only sees X^ and it doesn’t see the real examples at all. So these are being passed to D only.
class Generator(nn.Module):
def __init__(self, z_dim=10, im_dim=784, hidden_dim=128):
super(Generator, self).__init__()
self.gen = nn.Sequential(
self.generator_block(z_dim, hidden_dim),
self.generator_block(hidden_dim, hidden_dim * 2),
self.generator_block(hidden_dim * 2, hidden_dim * 4),
self.generator_block(hidden_dim * 4, hidden_dim * 8),
nn.Linear(hidden_dim * 8, im_dim),
nn.Sigmoid()
)
def generator_block(self, input_dim, output_dim):
return nn.Sequential(
nn.Linear(input_dim, output_dim),
nn.BatchNorm1d(output_dim),
nn.ReLU(inplace=True))
def forward(self, noise):
return self.gen(noise)
def get_gen(self):
return self.genAnd then D makes (Y^) of how real/fake these are. After that the predictions are compared using the BCE loss with all the labels equal to real. As G is trying to get these fake images to be equal to real as closely as possible.
class Discriminator(nn.Module):
def __init__(self, im_dim=784, hidden_dim=128):
super(Discriminator, self).__init__()
self.disc = nn.Sequential(
self.discriminator_block(im_dim, hidden_dim * 4),
self.discriminator_block(hidden_dim * 4, hidden_dim * 2),
self.discriminator_block(hidden_dim * 2, hidden_dim),
nn.Linear(hidden_dim, 1),
## nn.Sigmoid()
)
def discriminator_block(self, input_dim, output_dim):
return nn.Sequential(
nn.Linear(input_dim, output_dim),
nn.LeakyReLU(0.2, inplace=True))
def forward(self, image):
return self.disc(image)
def get_disc(self):
return self.disc- D wants X^ to seem as fake as possible, but G wants X^ to seem as real as possible (Adversarial).
- We alternate their training, only one model is trained at a time, while the other one is held constant.
After computing the cost, the gradient is then propagated backwards, and the parameters of G (θg) is getting updated in this process. Again now, it’s only G.
def get_gen_loss(gen, disc, criterion, num_images, z_dim= 64):
fake_noise = get_noise(num_images, z_dim)
fake = gen(fake_noise)
disc_fake_pred = disc(fake)
return criterion(disc_fake_pred, torch.ones_like(disc_fake_pred))it’s important to keep in mind that both models should improve together and should be kept at similar skill levels from the beginning of training. the reasoning behind this is if you had D that is superior to G, you’ll get predictions from it telling you that all the fake examples are 100% fake.
Well, that’s not useful for G (i.e., doesn’t know how to improve). Everything just looks super fake, there isn’t anything telling it to know which direction to go in. Meanwhile, if you had a superior G that completely out skills D, you’ll get predictions saying that all the generated images are 100% real.
And should be kept at similar skill levels from the beginning of training. And the reasoning behind this is largely because of D has a much easier task than G, it’s just trying to figure out which ones are real/fake, as opposed to model the entire space of what a class could look like.
So, having output from D be much more informative, like 0.87 fake or 0.2 fake as opposed to just 100% fake. Probability one fake is much more informative to the G in terms of updating its weights and having it learn to generate realistic images over time.
