Normalization Strategies
Clearly explain the differences among Batch normalization, Layer normalization, Instance normalization and Group Norm
Normalization has been a standard technique for vision-related tasks for a while, and there are dozens of different strategies out there. It can be overwhelming to try to understand each of them.
Recall that no matter what strategy we pick, the goal of normalization is to “shift” the target samples into a certain distribution. This is usually good for stabilizing training, as normalization standardizes the input.
Let’s look at how the top 4 most used normalization strategies work, and why you might choose one over another.
Throughout this post will be looking at an example with an input dimension of N x C x H x W, where N is the batch size, C is channels, and H x W is the size of an image.
Batch Normalization (BN).
BN is the 1st normalization strategy ever introduced to the world of DNNs. Benefits of using BN include faster training speed and greater input stability, which significantly reduces the chance of gradient explosion/vanishing.
This technique involves normalizing on the “batch layer” (represented as the green shade) N x H x W across the channel dimension C.
Quoting from this article, think of the structure of an image as a book. Each book (C x H x W) has multiple pages (C), with each page having height and width H x W. Now think of a row of N books on a bookshelf as a batch of images (N x C x H x W). To apply BN, we pick the 1st page of each book and apply normalization. We then repeat this process for the rest of C - 1 pages.
A problem with BN is that it yields poor results when there are not enough samples (N). It can also be time-consuming due to the way it is implemented. This is not ideal in certain scenarios such as tasks with low batch size, or tasks with variable batch size (online learning).
Layer Normalization (LN).
LN fixes the sample size issue that BN suffers from. This technique involves normalizing on the “layers” (yellow shade) C x H x W, which is basically a single image sample, across the batch dimension N. Note that layer norm is not batch dependent.
Continuing our analogy to books, think of the structure of an image as a book. Each book (C x H x W) has multiple pages (C), with each page having height and width H x W. Now think of a row of N books on a bookshelf as a batch of images (N x C x H x W). To apply layer norm, we pick the first book (C x H x W) and apply normalization. We then repeat this process for the rest of N - 1 books.
Get AI & Data Science 604’s stories in your inbox
Join Medium for free to get updates from this writer.
Layer norm is widely used in sequential modeling (RNN) and the transformer architecture, as NLP tasks often have variable batch sizes (sentence length), it is more intuitive to just normalize on a single word with layer norm.
Instance Normalization (IN)
IN, also known as contrast normalization, is first used in the StyleGANs paper for transferring image style. This technique involves normalizing on a single “layer” (green shade) H x W across both the channel (C) and batch (N) dimensions.
Again, think of the structure of an image as a book. Each book (C x H x W) has multiple pages (C), with each page having height and width H x W. Now think of a row of N books on a bookshelf as a batch of images (N x C x H x W). To apply IN, we normalize on the first page of the first book (H x W). We then repeat this process for the rest of the C - 1 pages for N - 1 books.
BN Vs IN?
Batch Normalization (BN)
BN will compute the mean and std for each of the channels across the mini batch. This means that all pixels in the same channel from all images in the batch are used to calculate the normalization parameters.
Instance Normalization (IN)
IN normalizes the inputs for each instance (image) independently. IN computes the mean and variance for each channel independently for each image.
- I.e, if you have a batch of N images with B number of channels for each, IN will calculate N sets of means and variances (one set per image) for each of the B channels. which means N * B in each mini-batch.
- This allows each image to be processed independently, which can be advantageous for tasks where individual characteristics of each image matter e.g., style transfer and image generation where the style needs to be adapted individually per instance.
Group Normalization (GN)
GN is a mixture of instance and layer norm. The idea is to split the channel dimensions into “groups” (yellow and purple shades). Normalization is then computed on each group across the batch N dimension just like instance norm.
For easy comparison, the 2 groups (yellow and purple shades) given in the diagrams above have different channel counts. This is usually not the case.
Same thing, think of the structure of an image as a book. Each book (C x H x W) has multiple pages (C), with each page having height and width H x W. Now think of a row of N books on a bookshelf as a batch of images (N x C x H x W). To apply group norm, we first divide the pages into C / G groups. We then normalize within every group in the first book. Finally, repeat this process for the rest of N - 1 books.
