GANs Part2: DCGAN
Using convolutions to tune the GAN architecture and apply them to build DCGAN.
The main features of DCGAN:
- Use convolutional layers instead of fully connected layers without any pooling layers
- Use BatchNorm in both the generator and the discriminator
- Use ReLU activation in the generator for all layers except for the output, which uses a Tanh activation.
- Use LeakyReLU activation in the discriminator for all layers except for the output, which does not use an activation
The activation function (AF) takes any real numbers as an input and its output is in a certain range using non-linear and differentiable operations.
A node takes in information from the previous layer and computes 2 things which is split with this dotted line below. And the nodes together compose the entire NN architecture.
z_i^[L] represents the input to the i-th neuron in the L-th layer. W_i^[l] denotes the weight associated with the connection from the i-th neuron in the previous layer L-1 to the i-th neuron in the current layer L. a_i^[l-1] is the activation (output) of the i-th neuron in the previous layer L-1.
The inputs to a neuron in a specific layer of a NN, are computed as a weighted sum of the outputs from the previous layer.
1st, It computes Z, where Z equals to a sum of various weights on the outputs from the previous layer.
This is typically called a linear layer, and both include scaling values of weights as well as a bias term which will shift those values.
2nd, compute a using AF given Z.
Why do we need differentiable and non-linear AF?!
- It needs to be differentiable as we use back propagation for training the NN and updating its weights by computing the derivative, so it needs to be able to differentiate and provide gradients to the previous layer.
- It also needs to be non-linear, to extract complex features.
If we didn’t use non-linear activations, a NN with multiple hidden layers could actually be collapsed into a simple linear regression.
Common Activation Functions.
Researchers often experiment with “what AF works best”. There’re a huge number of AF, but not all of them are ideal. We’ll discuss 4 of the most popular AF used in GANs.
Rectified Linear Unit (ReLU).
ReLU squashes out all the negatives. Any positive value coming into, the output will be the same. But, whenever Z is negative, then it’ll output 0. Note, ReLU is strictly speaking not differentiable at Z=0 (The flat part).
But by convention and implementation, the derivative of ReLU at Z=0 is often set to 0.
This can be problematic, as the learning process depends on the derivative to provide information on how to update the weights. With 0 derivative some nodes get stuck on the same value (stop learning). In fact, the previous components of the NN will all be affected as well.
This problem is known as “dying ReLU problem”.
LeakyRelu.
Leaky ReLU maintains the same form of ReLU when Z is positive, it keeps the same value, but it adds a little leak or a slope in the line when Z is negative. Broadly the slope is treated as a hyperparameter, but it’s typically set to 0.1, meaning the leak is quite small relative to the positive slope. This solves the “dying ReLU problem”.
Sigmoid
It has a smooth as shape and outputs values between 0 — 1. So, it is often used in binary classification models in the last layer to indicate a probability between 0- 1.
When Z>=0, the sigmoid activation outputs a value between 0.5 and 1. When Z<0, the sigmoid outputs a value between 0 and 0.5.
Sigmoid isn’t used very often in hidden layers because the derivative approaches zero at the tails. This produces “vanishing gradient problems” or saturated outputs at the tails.
You can imagine that sigmoid continues to go in both directions because it can take any real value as input.
Hyperbolic Tangent(tanh)
In contrast with the sigmoid however, it outputs values between -1–1.
When z is positive, it outputs a value between 0–1 and when z is negative it outputs and negative value between -1–0.
One key difference from the sigmoid is that tanh keeps the sign of the input Z, so negatives are still negative. That can be useful in some applications. Because it’s shape is similar to the sigmoid however, the same saturation and vanishing gradient issue does occur.
Again, the tails do extend on both sides of bridging 1 at the top and -1 at the bottom.
Review of Convolutions
We’ll go over a recap on what convolutions are and how they work. Convolution allows to detect key features in different areas using filters.
These filters scan across the image, to find various features. I.e., it will tell which parts of the image contains eyes, a nose, etc.
Each of these filters is just a matrix of real values learned during training. In reality, they’re much more abstract than eye or nose filters, but they do pick up on fairly high-level features. The convolution operation is very straightforward. Take a look at this 5-by-5 grayscale image where each square is a pixel with a value between 0 and 255.
Where 0 represents entirely black, 255 represents entirely white pixels and gray values are in between. Say you’ve a learned 3-by-3 filter with 1 in the 1st column, 0 in the 2nd column, and -1 in the last column.
The convolution operation using this filter on the grayscale image, this means that you want to apply this filter across this grayscale image.
What that would look like, is starting in the top left corner and multiplying all the elements from the filter on top of those pixels.
Then, take the sum of these products and put them into a resulting matrix (the 1st cell). After that, move the filter one position to the right to get the element-wise product, sum the results again, and store that value in the output matrix and so on. Then, go down a slot till covering the entire image with the filter. We get 3-by-3 result at the end.
There is a vertical line in the image, and the filter see some high values and it’s trying to say, “There’s a vertical edge here.” These filters could be of many different values and composing multiple layers of these filters will get higher-level features.
Get AI & Data Science 604’s stories in your inbox
Join Medium for free to get updates from this writer.
A CNN uses different layers to perform tasks in a hierarchical way, where:
- Initial layers discover edges and contours.
- Middle layers: Detect complex shapes (created by edges)
- Final layers: Uncover distinctive image features characteristic of the image type that you want the network to classify.
There’re some tweaks you can make to this simple operation that will be helpful for moving forward, which are padding and stride.
Padding & Stride
Take this grayscale image from the previous example.
Moving 1 pixel to the right and 1 pixel down is called using a stride = 1.
So you’re only moving 1 pixel at a stride of 1 each time. It doesn’t have to be the same stride value for going right versus going down as well.
The larger the strides are, the less the image will be coverage be filters, but at the same time it will make the computation faster. It is a trade-off.
A stride of 2, would mean that every time you actually take 2 steps to the right, so the filter starts in the top left corner, but now you go 2 pixels to the right each time, and then because it’s 2 both for right and down, you go 2 pixels down, and so you only do 4 computations.
As you can see, the stride determines how many sections the image your filter will visit. In this case, the result of convolution is now just a smaller 2 x 2 matrix, because you visited 4 of those 3 x 3 sections with the filter. 2 at the top and 2 at the bottom.
Padding
Padding is another tweak to the basic convolution. Take this 3 x 3 image and 2 x 2 filter. The values here are not important in the pixel. So I’ve left them out. When computing the convolutions with stride =1 in this case, you visit 4 sections of the image.
the top-left corner, the top-right corner, the bottom-left and 1 in the bottom-right. The pixel in the center, it gets visited 4 times, while the pixels in the corners are only visited once, and the other pixels are visited twice.
This means that the information kept in the center, gets more attention than the information on the edges. This is a problem if you’ve features on the edges. We want to ensure that the filter puts equal emphasis across the entire image.
To address this, we can add padding around the image, so all the information appears at the center of this whole new picture, exactly like in a framed picture. The padding size depends on the filter size, and the image size.
The value used for the padding can be any, but it’s often set to zero, which is called zero-padding. When you do those before computing the convolution, the filter scans the image along with its frame, but every pixel in the image is visited the same number of times. It’s just the padding is visited fewer times.
Pooling and Up-sampling.
Pooling is used to reduce the size of the input while up-sampling is used to increase the size of that input.
Pooling is used to lower the dimension of the input images by taking the mean or finding the maximum value of different areas.
I.e., a picture of this cat after a pooling layer will result in this blurry image with a lower size or lower resolution.
we can see the color palette and the color distribution over both images is still very similar. The shapes on the blurry image still resemble the original. It’ll be less expensive to do computations on this pooled layer than it’s on this original image.
Let’s take a look at one of the most popular types of pooling called max pooling. Take 4-by-4 grayscale image and we use a 2-by-2 max pooling filter in a sense and I’m saying filter, but there’re no learned weights. It’s just the window that’s being passed across and it’ll have a stride=2, so that I won’t visit any pixel more than once.
Starting in the top left corner of the image, we take the maximum value in this window(20) and put it in the output. Next you do it to the same in the top right corner (not overlapping any pixels). Then you keep doing that until you’ve covered the entire image. At the end, we’ve 2-by-2 output containing the maximum of each of these 2-by-2.
What max pooling does, is getting the most salient information from this image, which are these high values. This can be really important for distilling information where you really care about only the most salient information.
Sometimes this is not an image, and this is just an intermediate layer, so this is really just extracting the weights with the highest values on this intermediate section.
There’re other pooling other than max pooling.
There’s average pooling, where we just take the average instead of the max and min pooling where you took the minimum. Again, it’s really important to know that pooling doesn’t have any learnable parameters. That’s where it’s really different from convolutions. There’s was just a simple rule applied across the image.
Up-Sampling.
up-sampling has the opposite effect of pooling. Given this lower resolution image, up-sampling has a goal of outputting one that has higher resolution.
It actually requires inferring values for the additional pixels and there are a few different methods to do this. One easy way to is known as nearest neighbors up-sampling, where we can copy the values of the pixels from the input multiple times to fill the output. I.e., take this 2-by-2 image and up-sample to 4-by-4 output.
First, we assign the value in the top left corner from the input to the top left pixel in the output.
The other values from the input to pixels that are added distance of 2 pixels from that top left corner and for other cases, this distance might be different depending on the size of the enlargement that you would want.
There’ll be exactly one pixel in between each of these, including this diagonal. Then assign the same value to every other pixel as it is to its nearest neighbor. For every 2-by-2 corner in the output, the pixel values will look the same.
we can also think of this as putting these values into the corners first and for every other pixel finding its nearest neighbor. Again, pooling for up-sampling, there are many different ways to do up-sampling, like linear and bi-linear interpolation.
Transposed convolutions (TConv)
I.e., a 2-by-2 input that you want to up-sample to a 3-by-3 output. Using a TConv, we can use a 2-by-2 filter (learned paramters) with stride=1 to accomplish this task by following a really similar procedure to the one for convolutions.
You start by taking the top-left value from your input and getting its product with every value in the 2-by-2 filter. Then save this value in the top 2-by-2 left corner of the output.
Note, some values in the output are influenced much more heavily by the values from the input. i.e., the center pixel in the output is influenced by all the values in the input, while the corners are influenced by just once.
The center pixel is visited 4 times and is influenced by all pixels while the other ones aren’t, and this causes a common issue that arises when using TConv. This image shows a real output from a TConv where the circle is not actually generated. It’s just zoom in to this checkerboard issue. Notice that the image has a pattern resembling a checkerboard and this arises because when you up-sample with a filter, some pixels are influenced much more heavily while the ones around it are not.
