Regularization

A good way to reduce overfitting is to regularize the model (i.e., to constrain it): the fewer degrees of freedom it has, the harder it will be to overfit the data.

Introduction

• Regularization basically adds the penalty as model complexity increases.
• Regularization parameter (lambda) penalizes all the parameters except intercept so that model generalizes the data and won’t overfit.

We will now look at ridge regression, lasso regression, and elastic net regression, which implement three different ways to constrain the weights. Also, we will be covering why Early Stopping is important.

Assumption

• Both regularizations assume that the model can be expressed as an additive combination of the features.
• The target variable is a linear combination of the features plus some noise.
• The observations in the dataset are assumed to be drawn from the same underlying distribution, meaning they are independently and identically distributed.

Ridge Regression — L2 regularization

• L2 regularization is a method to mitigate overfitting in machine learning models.
• It can effectively handle multicollinearity in a dataset if model interpretability isn’t a concern.
• Helps to distribute the effects of correlated variables evenly across various features.
• Emphasizes stability and retains all features.
• It can reduce coefficient values toward zero but never exactly to zero. This also means that L2 regularization cannot perform feature selection.

L2 | Ridge Regression

• A regression model which uses L2 technique is called Ridge Regression.
• Puts weight on certain values to discourage the model from fitting training data too closely, and to reduce overall model complexity.
• L2 regularization adds the squared sum of coefficients (or the “squared magnitude” of the coefficient) as the penalty term to the model’s SSE loss function.
• It tends to distribute the weight among correlated features, reducing the risk of overfitting compared to models without regularization. Since L2 regularization does not drive coefficients to zero, it can provide more stable predictions, which helps in scenarios with small sample sizes or noisy data.
• If the regularization parameter is set too high, L2 can also lead to underfitting. In this case, the coefficients shrink towards zero too much, which can hinder the model’s ability to learn the underlying patterns in the data.

Lasso Regression — L1 regularization

• Lasso regression can reduce coefficient values to zero, enabling feature selection and removal.
• L1 regularization works well for feature selection in case we have a huge number of features.
• L1 regularization can actually cause model underfitting based on the lambda value. If lambda is zero, then regularization is disabled.

L1 | Lasso Regression

• A regression model that uses L1 technique is called Lasso Regression.
• L1’s penalty term is the absolute value of the sum of coefficients (or the “absolute value of magnitude” of the coefficient), and is added to the model’s sum of squared errors (SSE) loss function.
• Outliers can significantly impact L1 and L2 regularization, with L1 generally being more robust to their influence.
• L1 regularization is generally less prone to overfitting because it adds a penalty for large coefficients, which helps to simplify the model. By driving some coefficients to zero, it effectively reduces the model’s complexity and focuses on the most significant features.
• If the regularization parameter is set too high, L1 regularization can lead to underfitting by overly simplifying the model and driving too many coefficients to zero.

Get Rahul Singh’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Elastic Net Regression

• Elastic net regression is a middle ground between ridge regression and lasso regression.
• The regularization term is a weighted sum of both ridge and lasso’s regularization terms, and you can control the mix ratio r.

Elastic net cost function

• When r = 0, elastic net is equivalent to ridge regression.
• when r = 1, it is equivalent to lasso regression
• It is almost always preferable to have at least a little bit of regularization, so generally you should avoid plain linear regression.
• Ridge is a good default.
• If you suspect that only a few features are useful should prefer lasso or elastic net because they tend to reduce the useless features weights down to zero.
• In general, elastic net is preferred over lasso because lasso may behave erratically when the number of features is greater than the number of training instances or when several features are strongly correlated.

from sklearn.linear_model import ElasticNet
elastic_net = ElasticNet(alpha=0.1, l1_ratio=0.5)
elastic_net.fit(X, y)
elastic_net.predict([[1.5]])
>>> array([1.54333232])

Early Stopping

• A very different way to regularize iterative learning algorithms such as gradient descent is to stop training as soon as the validation error reaches a minimum. This is called early stopping.

Early stopping regularization

• A complex model being trained with batch gradient descent on the quadratic dataset.
• As the epochs go by, the algorithm learns, and its prediction error (RMSE) on the training set goes down, along with its prediction error on the validation set.
• After a while, though, the validation error stops decreasing and starts to go back up. This indicates that the model has started to overfit the training data.
• With early stopping you just stop training as soon as the validation error reaches the minimum.
• It is such a simple and efficient regularization technique that Geoffrey Hinton called it a “beautiful free lunch”.

Here is a basic implementation of early stopping:

from copy import deepcopy
from sklearn.metrics import mean_squared_error
from sklearn.preprocessing import StandardScaler

X_train, y_train, X_valid, y_valid = [...] # split the quadratic dataset

preprocessing = make_pipeline(PolynomialFeatures(degree=90, include_bias=False),
StandardScaler())

X_train_prep = preprocessing.fit_transform(X_train)
X_valid_prep = preprocessing.transform(X_valid)
sgd_reg = SGDRegressor(penalty=None, eta0=0.002, random_state=42)
n_epochs = 500
best_valid_rmse = float('inf')

for epoch in range(n_epochs):
sgd_reg.partial_fit(X_train_prep, y_train)
y_valid_predict = sgd_reg.predict(X_valid_prep)
val_error = mean_squared_error(y_valid, y_valid_predict, squared=False)
if val_error < best_valid_rmse:
best_valid_rmse = val_error
best_model = deepcopy(sgd_reg)

1. code first adds the polynomial features and scales all the input features, both for the training set and for the validation set (the code assumes that you have split the original training set into a smaller training set and a validation set).
2. Then it creates an SGDRegressor model with no regularization and a small learning rate.
3. In the training loop, it calls partial_fit() instead of fit(), to perform incremental learning.
4. At each epoch, it measures the RMSE on the validation set.
5. If it is lower than the lowest RMSE seen so far, it saves a copy of the model in the best_model variable.
6. This implementation does not actually stop training, but it lets you revert to the best model after training.
7. Note that the model is copied using copy.deepcopy(), because it copies both the model’s hyperparameters and the learned parameters. In contrast, sklearn.base.clone() only copies the model’s hyperparameters.

Find me on LinkedIn.