Batch Normalisation
It is an important Normalisation technique generally used in neural networks and Deep Learning
It scales down activation of each layer for each mini- batch data set
Mean and variance are calculated for each input and x_normalised = xi — mean/(square_root(variance² + epsilon)) and then yi_hat = gamma* xi + beta
where yi_hat is the predicted output we predict using our model and the data it is trained upon, gamma and beta are two variables where gamma : learnable scaling parameter, beta : learnable shifting parameter
We Have momentum and epsilon as it’s hyperparameter, But dropout rate for Dropout method,but both Dropout and Batch Normalisation are different methods, Where Dropout reduces the chances for overfitting, Batch Normalisation makes our model more robust and stable towards changes and allows model to reconsider genuine data points which once might be declared as outliers to rescale and might be considered in the calculation
Get Akshat Gupta’s stories in your inbox
Join Medium for free to get updates from this writer.
Further benefits of using Batch Normalisation are :
- Makes model more robust and stable
- helps in regularisaiton of activation in each layer
- Re -Initialise weights/ parameter in better manner
- Better optimization and faster convergence
- Allow higher learning rates
But a downside is that it can behave poorly for small mini-batch datasets
Batch Regularisaiton only regularises during training and uses that result while predicting individual instance or batch dataset during inference
Although it is not gauranteed that this algorithm or method might gaurantee best results and always perform optimization that lead to better results, we must consider best option according to out requirements and parameters by comparing different models and algorithms by cross validation
