Normalization: Clear Explaining
Normalization is a DL technique used scale input features to a common range. The primary aim of normalization is to ensure that the NN trains effectively and efficiently, leading to faster convergence and improved overall performance.
AGENDA
Introduction
What is Batch Normalization (BN)?
Calculations in Batch Normalization
Before or After Activation Functions?
How does BN work in the training phase?
Benefits of BN.
Challenges and Considerations.
Introduction
Training DNNs is complicated by the fact that the distribution of each layer’s inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift.
Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. BN allows us to use much higher learning rates and be less careful about initialization, and in some cases eliminates the need for Dropout.
What is batch normalization?
Batch Normalization developed in 2015 and quickly became essential in DL. Generally, BN is used to standardize the inputs and/ or outputs to each layer (not limited to the input layer), improving the training process and model performance.
Its primary goal is to address the issues of internal covariate shift, which refers to the changes in the distribution of inputs to a layer as the parameters of the previous layers change during training. This shift can significantly slow down the convergence of the network and make training more unstable.
Batch normalization works by normalizing the inputs to a layer for each mini-batch, ensuring that they have a mean of zero and a variance of one. This normalization step is achieved by computing the mean and variance for the current mini-batch and then using these statistics to scale and shift the inputs. The key idea here is to maintain a stable distribution of inputs throughout the training process, which allows the network to learn more effectively.
One of the significant advantages of batch normalization is that it enables the use of higher learning rates. With stable input distributions, models can converge faster and sometimes achieve better performance. Additionally, batch normalization acts as a form of regularization, reducing the need for other techniques such as dropout. By incorporating normalization layers, deep networks can also become less sensitive to weight initialization, allowing for more robust training across different architectures.
Calculations in Batch Normalization.
Here are the steps of performing BN on a batch.
- Step 1: The algorithm calculates the mean and variance of a mini-batch.
Here, μB is the mini-batch mean, m is the number of instances in the mini-batch and xi represents each input instance.
Here, σ 2 B is the mini-batch variance, μB is the mini-batch mean, m is the number of instances in the mini-batch and xi represents each input instance.
- Step 2 (Normalization Step): normalize each instance in the mini-batch.
Here, xˆᵢ represents each normalized instance. σ² ᵦ is the mini-batch variance and μB is the mini-batch mean which were calculated earlier.
ε is a very constant small (e.g. 0.001) (often referred to as a numerical stabilizer) added to the variance in order to avoid division by zero if σ is zero in some cases.
- Step 3 (Scale and Shift Step): The normalized instances are then multiplied by γ (gamma) and shifted by β (beta). The γ and β are new learnable parameters alongside the original network parameters. Their optimal values will be obtained from the input data during training just like we update weights and biases in the network!
Here, yi is the output of the BN layer.
Step 3 is optional when the next layer is linear as scaling and shifting handle by that layer itself.
In Step 2, the mean of the outputs is 0 and the standard deviation is 1. However, the values of the intermediate layers generally prefer to be standardized around a mean other than 0 and have a variance other than 1. That’s why we perform Step 3 in the batch normalization process.
Before or After Activation Functions.
You might be wondering about where to apply BN in our NN.
BN can be performed either before or after the activation functions, and both approaches are used in practice.
Before Activation Function: BN is applied to the output of the linear transformation (i.e., the weighted sum of inputs). This helps in stabilizing the distribution of inputs to the activation function, potentially leading to faster convergence and better overall performance.
After Activation Function: BN is applied after the activation function is less common, but it can still be effective in certain scenarios. It normalizes the post-activation outputs, which can also help in stabilizing the training process.
Here’s the advice of some Deep Learning experts:
Andrew Ng says: “BN should be applied immediately before the nonlinearity of the current layer”. The authors of the BN paper said that as well, but according to Francois Chollet on the Keras thread, the BN paper authors use BN after the activation layer. On the other hand, there are some benchmarks (…) that show BN performance better after the activation layers.
There is thus no clear answer to this question. You will have to try experimentally what works best. As, in many architectures, it is more typical to apply BN before the activation function. This is often referred to as “pre-activation” normalization. Ultimately, whether to apply batch normalization before or after the activation function can depend on the specific architecture, the nature of the data, and the desired properties of the network. Experimentation is often key to finding the optimal configuration for a particular problem.
How does BN work in the testing phase?
When it comes to prediction, we usually predict a single instance instead of a batch of instances. How does the algorithm calculate mini-batch mean and variance during testing when there are no batches?
The algorithm solves this by simply calculating the moving average and moving standard deviation of the mini-batch during training and storing these values to use in the test phase. The momentum argument is also associated with that process.
Get AI & Data Science 604’s stories in your inbox
Join Medium for free to get updates from this writer.
Now, there are 4 parameters contained in a BN layer which are γ (gamma), β (beta), moving average and moving standard deviation
The γ (gamma) and β (beta) are trainable parameters as their optimal values are learned during training.
The moving average and moving standard deviation are non-trainable parameters as their values are derived from the data at once rather than updating values during training.
Benefits of Batch Normalization
At this point, you may have a good understanding of what the benefits of Batch Normalization are, but let’s outline them formally:
Faster Convergence and Speed up Training.
Batch Normalization accelerates the training process by allowing the use of higher learning rates. It achieves this by stabilizing the distribution of the inputs to each layer throughout training. With inputs that have a fixed mean and variance, the network’s weights are less likely to diverge even with higher learning rate settings. This normalization reduces the number of epochs needed to reach convergence, effectively speeding up the training process considerably.
After normalizing, the values are small and homogeneous (have the same range). This makes learning (converging) faster.
When data is not normalized, the features may have different scales. This can lead to a situation where the cost function has a non-symmetric shape, making it difficult for optimization algorithms like gradient descent to find the minimum efficiently. If one feature dominates the others due to its scale, the gradients can become skewed, causing the optimization path to oscillate and take longer to converge. Without normalization, the cost surface might have elongated contours, leading to a zigzagging path towards the minimum.
Normalizing the data (e.g., scaling features to a common range such as [0, 1] or standardizing to have a mean of 0 and a standard deviation of 1) helps to create a more symmetric cost function. This symmetry allows the gradient descent algorithm to make more uniform steps towards the minimum, reducing oscillations and leading to a more stable and faster convergence.
Normalizing makes the cost function more symmetric, which aids in more efficient training and much easier to converge because the gradient descent algorithm goes straight towards the minimum error faster rather than oscillating, which needs more time.
When the features are on a similar scale, the learning rate can be more consistent across all dimensions. This helps prevent overshooting in one direction while making smaller steps in others. As a result, the convergence tends to be more direct and efficient, meaning that the algorithm can reach the minimum loss faster.
Reduce the vanishing gradient problem
This problem may happen when there are many layers in the NN. During training, layer weights are often multiplied by very small numbers less than 1. When this happens in many layers, the resulting outputs become even smaller so that, at some point, computer hardware cannot represent these small values anymore. So, the vanishing gradient problem arises.
Overcoming Covariate Shift
Normalization is key in NNs to overcome various training hurdles, especially internal covariate shifts, which describes how the distribution of inputs for each network layer changes during training as the parameters from previous layers are adjusted. Such shifts can slow down training because each layer must constantly adapt to new data distributions. It’s like trying to hit a moving target, complicating the training process and often necessitating lower learning rates and meticulous parameter initialization to achieve model convergence.
In the context of NNs, managing data flow over time presents significant challenges, such as controlling vanishing or exploding gradients. The issues of shifting data distributions further complicate this, potentially destabilizing the network over long training periods. While some deep learning models like LSTMs do employ mechanisms like forget gates to lessen some effects of input shifts, these are specifically tailored to sequence modeling and don’t tackle the fundamental issue of shifts within the network’s hidden layers.
Increased Stability
By normalizing the inputs across mini-batches, BN helps in reducing the internal covariate shift which is the phenomenon where the distribution of layer inputs changes due to changes in network parameters during training. This shift often complicates training by requiring lower learning rates and careful parameter initialization. By ensuring consistent activation distributions, BN adds a level of stability to the training process, making the network less sensitive to parameter scale and initialization.
Reducing the Need for Careful Initialization
Before the advent of BN, careful initialization of network weights was crucial to prevent activations from becoming too large or too small, which could cause vanishing or exploding gradients. This technique alleviates this issue by re-scaling activations, which means that the initial weights no longer need to be precisely tuned. This relaxation in the requirement for initialization precision allows for more flexibility in choosing weight initialization methods, simplifying the setup of NN training significantly.
Influence on Learning Dynamics
Batch Normalization not only improves training metrics but also alters the learning dynamics of a NN. It decouples the parameters of preceding layers from those of the succeeding layers, making each layer more independent of others in terms of learning updates. This independence can lead to better learning patterns as the layers do not need to compensate for the changing scales of inputs during the training process. Additionally, the scale and shift parameters learned alongside the normalized inputs can help the network better model complex relationships in the data by dynamically adjusting the range of activations throughout the network.
Challenges and Considerations
Despite its many benefits, Batch Normalization has its challenges and limitations, which need careful consideration.
Despite its many benefits, batch normalization is not without its drawbacks. For instance, it can be less effective in certain situations, like in recurrent neural networks (RNNs) or when processing very small batch sizes. Researchers have explored alternatives like layer normalization and group normalization to address these challenges.
Dependency on Batch Size
Batch Normalization’s effectiveness is closely tied to the size of the mini-batches used during training. With too small batch sizes, the estimates of the mean and variance used for normalization can become inaccurate, leading to unstable training behavior. Conversely, very large batches require significant memory overhead and can slow down the training process. Finding the right balance in batch size is crucial to leveraging the full potential of Batch Normalization.
Impact on Training Dynamics
While Batch Normalization stabilizes training, it can also introduce a certain level of unpredictability in batch statistics, especially when applied to recurrent architectures like LSTMs or in situations where the batch size is small. This unpredictability can sometimes lead to degradation in model performance if not managed correctly.
Mitigation Strategies
To mitigate these challenges, one approach is to use sufficiently large mini-batches to ensure stable estimates of mean and variance. Alternatively, techniques such as Ghost Batch Normalization, which divides larger batches into smaller “ghost” batches, can be employed to maintain normalization effectiveness without the computational overhead of large batch sizes. Additionally, careful monitoring of training and validation performance should be conducted to adjust hyperparameters like learning rate and batch size dynamically based on the observed training behavior.
