Motivation for Regularization in Machine Learning
A simple intuition for how regularization reduces overfitting and improves generalization
Regularization is one of those machine learning concepts that shows up everywhere, but the motivation behind it is not always obvious at first.
In this article, I want to explain regularization using a simple intuition: how it helps move a model away from overfitting and toward better generalization. I had also created a youtube video here for visual explanation.
This assumes you already have some basic familiarity with machine learning concepts like models, loss functions, training error, and test error.
Linear vs. Nonlinear Models
Imagine we have a classification dataset with two types of points:
- Positive examples
- Negative examples
Now suppose we try to fit two different models on the same dataset.
The first model is linear. It tries to draw a straight decision boundary between the positive and negative examples.
The second model is nonlinear. It has enough flexibility to bend and curve around the data points.
Figure 1: A simple linear model may make a few mistakes, while a highly flexible nonlinear model can bend around the training data (aka overfitting).
The nonlinear model may be able to classify every training example correctly. It can create a complex decision boundary that separates all positives from all negatives.
At first, that might sound great. After all, zero training error seems like a good thing. But this is where overfitting enters the picture!
What Is Overfitting?
An overfitted model learns the training data too closely.
Instead of learning the general pattern, it starts learning the noise or randomness in the specific training sample. As a result, it may perform extremely well on training data but poorly on new, unseen data.
Overfitted models usually have high variance. This means that if we sampled a new dataset from the same distribution and trained the model again, we might get a very different decision boundary.
A simpler model, such as a linear model, may misclassify a few training examples, but it is often more stable. It may generalize better to real-world data.
So the goal is not always to minimize training error as much as possible. The real goal is to build a model that performs well on unseen data.
Where Regularization Comes In
Regularization helps control how complex a model becomes.
At a high level, regularization penalizes large weights or coefficients in the model. By doing this, it discourages the model from relying too heavily on any particular feature or from becoming unnecessarily complex.
A simpler linear model might look like:
A more flexible nonlinear model might look like:
The nonlinear model has an additional term, x², which allows it to capture curved patterns in the data. This extra flexibility can be useful, but it also increases the risk of overfitting.
The Loss Function Without Regularization
In machine learning, we usually train a model by minimizing a loss function.
Get Akashkadel’s stories in your inbox
Join Medium for free to get updates from this writer.
For simplicity, imagine a squared error loss:
This loss only cares about reducing the difference between the actual values and predicted values.
Without regularization, the model is free to make the weights as large as needed if that helps reduce training loss. This can lead to very complex decision boundaries and overfitting.
Adding Regularization
With regularization, we add an extra penalty term to the loss function.
For L1 regularization, we penalize the absolute values of the weights:
For L2 regularization, we penalize the squared values of the weights:
Here, λ controls how much importance we give to the regularization term.
If λ is small, the penalty is weak, and the model can still become quite flexible.
If λ is large, the penalty is strong, and the model is pushed toward smaller weights.
Effect of lambda on model complexity
Figure 2: As regularization strength increases, model weights shrink and the decision boundary usually becomes simpler. Source: link
How This Reduces Overfitting
Let’s go back to the nonlinear model:
The x² term is what gives the model its nonlinear flexibility.
When we apply regularization, the model is encouraged to reduce unnecessary weights. If the coefficient w₁ becomes very small, the nonlinear term contributes less to the final prediction.
In an extreme case, if w₁ becomes zero, the model effectively becomes:
This is now similar to a linear model.
So regularization can reduce the influence of complex terms and help the model move from an overfitted state toward a more generalized state.
L1 vs. L2 Regularization
Both L1 and L2 regularization penalize large weights, but they behave slightly differently.
L1 regularization can push some weights all the way to zero. This makes it useful when we want feature selection, because some features may effectively be removed from the model.
L2 regularization usually shrinks weights toward zero but does not necessarily make them exactly zero. It is often useful when we want to reduce model complexity while still keeping most features active.
Figure 3: L1 regularization can push some weights exactly to zero
Final Intuition
Regularization helps a model focus on the signal instead of memorizing the noise.
By penalizing large weights, it discourages unnecessary complexity and reduces overfitting. The result is a model that may not fit the training data perfectly, but is more likely to generalize well to unseen data.
