REGULARISATION (L1 ,L2)
Regularisation is a process of introducing additional information in order to prevent overfitting. The focus for this article is L1 and L2 regularisation.
Overfitting is a phenomenon that occurs when a machine learning or statistics model is tailored to a particular dataset and is unable to generalise to other datasets. This usually happens in complex models, like deep neural networks.
Let’s go!
What’s L1 and L2?
L1 and L2 regularisation owes its name to L1 and L2 norm of a vector w respectively.
1-norm (also known as L1 norm)
2-norm (also known as L2 norm or Euclidean norm)
A linear regression model that implements L1 norm for regularisation is called lasso regression.
Get Nehra Vikas’s stories in your inbox
Join Medium for free to get updates from this writer.
one that implements (squared) L2 norm for regularisation is called ridge regression.
Loss function with no regularisation
We define the loss function L as the squared error, where error is the difference between y (the true value) and ŷ (the predicted value).
Let’s assume our model will be overfitted using this loss function.
Loss function with L1 regularisation
Based on the above loss function, adding an L1 regularisation term to it looks like this:
where the regularisation parameter λ > 0 is manually tuned. Let’s call this loss function L1. Note that |w| is differentiable everywhere except when w=0, as shown below. We will need this later.
Loss function with L2 regularisation
Similarly, adding an L2 regularisation term to L looks like this:
