L1, L2 and Elastic Net
Regression
In regression we use penalty to overcome the problem of underfitting and overfitting.
By regularization we can overcome the problem of overfiting you can see in image.
In this figure you can see the model is working best for training data which is green color focus on 1st figure and for test data set it is not perfoming well. if you are confused how can i say it is not performing well for the test dataset so let me tell you the error (mse) is higher in first than second fig(regularized) and our goal in any ml algorightm to reduce the sqaured error so that’s why we can say after regularization our model is performing well.
There are three type of regularization technique we can use
l1
It can decrease weight of feature to 0 if that is not important overall to our model and it add the term of absolute summation of weight multiplied by alpha we can a parameter tuning constant in loss function.
l2
It does not decrease the weight of feature to 0 instead of this it decrease so that it tends to 0 but not exact to 0 and it add the term of square of weight
Elastic Net
This is the concept inspired from L1and L2. In elastic net both term L1 and L2 added to loss function.
Difference between L1 and L2
a) Value of weight of some feature can be equal to 0 in case of L1 while in case of l2 it only tends to 0 not exact 0
Get Yahia’s stories in your inbox
Join Medium for free to get updates from this writer.
b) When we increasae value of alpha L1 reduce all weight to 0 and become straight horizontal that implies it does not depend on any feature while in case of L2 it cannot possible to dec the all weight to 0 even single cannot be reduced to 0
L1 regression(ridge regression)
L2 regression (lasso regression)
in fig you can see we cannot horizontal can be obtained in ridge regression(l1).
c) l1 is robust to outlier while l2 is not as such as l1
d) A new figure
in this fig we can see in the case of l1 the loss function minimize at w=0 and this is what we do means we want to reduce loss function and in l1 we can achieve it by w=0 while in l2 we cannot minimize loss function at w=0 in any possibility since in x axis possible weight is taken
