Dropout
Deep neural networks are really powerful machine but also very slow to use due to the large amount of parameters and the related problem of overfitting. We already presented the Batch Normalization technique, which can help us to solve this problem, normally more than just one technique are combined in order to address this issue, Drop Out is one of those.
The term ’Dropout’ refers to randomly drop out units (hidden and visible) in a neural network during the training phase. By doing this, we temporarily remove it from the network along with its incoming and outgoing connections. In the simplest case, each unit is associated with an independent probability p evaluated using a validation set or set to 0.5 by default.
Get Paola Benedetti’s stories in your inbox
Join Medium for free to get updates from this writer.
The application of dropout to a standard neural network amounts to sampling a thinner network from it which is composed of all units that survived dropout. A neural network with n units can be seen as a collection of 2^n possible number of thinned neural networks. For each presentation of training case, a new thinned network is sampled and trained meaning, training a neural network with dropout is the same as training a collection of 2^n thinned networks with extensive weight sharing, where each thinned network gets trained very rarely.
At test time it’s not feasible to explicitly average the reduction from exponentially many thinned models however, a very simple averaging method works well in practice: a single neural net without dropout where weights are a scaled down versions of the trained ones. The scaling enable the combination of 2^n networks with shared weights into a single neural network used at test time.
Training a network with dropout and using this approximate averaging method at test time, compared to training with other regularization methods, leads to significantly lower generalizations error on a wide variety of classification problems.
