Batch Normalization
Training Deep Neural Networks is complicated due to the fact that the distribution of each layer’s input changes according to the variation of previous layer parameters: Internal covariance shift. In DNN, even small changes can entail a major variation in the configuration of the network, leading to changes in the input distribution to hidden layer.
Get Paola Benedetti’s stories in your inbox
Join Medium for free to get updates from this writer.
Internal covariance shift is addressed by normalizing the layer input with the batch normalization technique. The idea behind BN is to introduce a normalization step that fixes the mean and variances of a layer input. Also, the examples are processed by batches instead of performing the training on the entire dataset as one. This also affects positively the gradient flow through the network because it reduces the dependence of gradients on the scale of the parameters and the initialization. Limiting the activation results to a certain range, it allows us to use a higher learning rate, it also reduces overfitting thanks to its regularization effect due to the introduction of noise to each hidden layer’s activation.
Normally batch normalization is combined with other technique, such as drop out.
