Mastering Model Selection with Cross-Validation

Think that you have some 50 to 100 data points in your dataset. We cant say that they are lying linear. They may be of any random sort.

After observing the points you may get a doubt that :

Which model to choose? (linear/quadratic/cubic or higher order)

Now think of this scenario! What if you train the data right from 1st order polynomial to 10th order polynomial? You would get the output of all the test data from 1 to 10.

Just look at all of the test results you got after testing with 10 different-order polynomials. If the test set of 5th order polynomial gives you the lowest value, then you might decide that the value would be the better model for your application.

Get santhoshini’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

But, the key issue here is something called “data leakage” — you’re essentially using your test set to make modeling decisions, which violates a fundamental principle of machine learning evaluation. Here’s why this is problematic:

1. The Test Set Should Be Completely Hidden

• The test set is meant to be your final, unbiased estimate of model performance
• If you use it to make modeling decisions (like choosing polynomial order), you’re “leaking” information from your test set into your model selection process
• This means your final performance estimate will be optimistically biased

1. The Problem with Your Approach When you:

• Train models of order 1–10
• Pick the one with the lowest test error (5th order in your example)
• Report that error as your final performance

The optimal procedure for model selection would be splitting the data into training, validation, and test sets. Nearly 60% of the data should be included in training data, and the remaining 40% should be used for validation. Half of this would be used for validation, i.e. 20%, and the other half would be used for the test set.

1. Training Set (Usually 60–70% of data)

• This is where your model learns from the data
• Used to fit/estimate the model parameters
• The model sees and learns from these examples directly
• Example: In a polynomial regression, this data is used to find the coefficients

Validation Set (Usually 15–20% of data)

• Used for tuning hyperparameters and making model decisions
• Helps prevent overfitting by evaluating model performance on unseen data
• Used repeatedly during the model development process
• Examples of decisions made using validation data:
• Choosing the number of layers in a neural network
• Selecting the best learning rate
• Picking the optimal number of trees in a random forest
• Deciding when to stop training (early stopping)

1. Test Set (Usually 15–20% of data)

• The final, untouched dataset
• Used ONLY ONCE to evaluate the final model
• Provides an unbiased estimate of model performance
• Never used for model training or tuning
• Simulates how your model will perform on completely new data

The very next baby step would be the calculation of training error, cross-validation error, and test error.

Where m is the number of data points included in that particular set.

Armed with these 3 measures of learning algorithm performance, one can now then go for carrying out model selection.

Instead of carrying out the result on a test set — like the way we thought above, we would instead evaluate these on our cross-validation set.

Perform cross-validation error on 1st, 2nd,…, until 10th order polynomial and the one which has a lower value of error. The preserves the test set hidden and untouched and when you test the data finally, it gives you a more accurate estimation of the model.

If 4th 4th-order polynomial is the lowest, then you would pick this 4th-order polynomial as a ‘good model’ for your application.

Finally, if you want to report an estimation of the generalization error, for how well this model would do on new data, use the test set of data to calculate the J(test) for w4, b4.