ML-How to choose Lambda
About the lambda in regularization term-Andrew Ng
Effect of lambda value
In the previous post about regularization, we used a term called the “regularization term” to prevent overfitting. In this post, let’s see how we choose the value lambda in our regularization term,
Let’s assume that our model uses this hypothesis function, and linear regression for analysis.
The cost function would be like this with our regularization term at the end.
Get Jaehoon Jang’s stories in your inbox
Join Medium for free to get updates from this writer.
Let’s see effects of the changes of lambda value.
How to choose lambda
It’s similar to our model selection method that we covered before. We change our lambda values and calculate the weights by minimizing the cost function. Then we calculate the errors using the cross-validation set, choose the parameters that have the lowest error on the cross-validation set, then apply these parameters to our test set to see how well it performs.
How cross-validation error/train error change due to the changes in lambda
If lambda has small value, the hypothesis function would be more complicated and would have more polynomial terms, so the train error would be low. As lambda increases, the hypothesis function would be more simplified and the train error would increase.
However, cross-validation error changes a bit differently. When the lambda is large, it may cause underfitting(high bias situation) so the cross-validation error would be large. If lambda is small, it will cause overfitting, and it also leads to increase in error.
