HOLDOUT in Machine Learning
In this Abstract we will see all about HOLDOUT in the most simplest manner. This abstract is completely for beginners who want to understand different concepts in Machine Learning in the easiest and fastest way!
Overview of what exactly is Holdout:
In Machine Learning, we have to follow a set of steps for building a Machine Learning Model for good predictions:
1) Data Extraction
2) Data Cleaning
3) EDA (Exploratory Data Analysis)
4) Feature Engineering (Selection , Extraction)
5) Feature Scaling (Standardization , Normalization)
6) Machine Learning Model Development
a) Model Training
b) Model Evaluation
c) Model Testing
And in the 6th step is where our topic comes into picture!
In the Machine Learning Model Development (Training, Testing) is where a method named HOLDOUT is implemented.
Training, Testing has 2 approaches:
1) HOLDOUT
2) K-Fold Cross Validation
Let’s dive deeper into HOLDOUT:
As mentioned earlier, HOLDOUT is a method used in Training, Testing i.e., this is where our data is completely pre-processed & ready for implementation and evaluation (for further improvements, maybe use Hyperparameter Training)
What exactly happens in HOLDOUT…?
Our ready data has to be evaluated so for that, we divide our data into 2 sections:
1) Training Set
2) Testing Set
Now you maybe asking why do we do this?
For ex. assume I have a set of Mangoes. 10 ripe mangoes and 7 unripe mangoes. And I have to teach a child the 2 different states of the fruit. So there are 2 ways of doing that
- Show all the Mangoes (ripe & unripe) to the child and teach which is ripe and which isn’t. And then asking the child to classify [from the already shown Mangoes] the ripe and unripe Mangoes (Evaluating the Training performed on the child)
- Perform HOLDOUT and show some (70% i.e. 12 Mangoes) of Mangoes (ripe & unripe) and teach the child. And then asking the child to classify [from the remaining Mangoes i.e., 30% of Mangoes] the ripe and unripe mangoes.
Considering the 1st possibility:
Since the child has already seen and learnt all the Mangoes, it will classify all the Mangoes absolutely correctly
But if we show the child some other Mangoes, it may find some difficulty for classifying the same.
This method is highly Theoretical and very less Practical.
Considering the 2nd possibility:
Now here, the child is shown some of the Mangoes and asked to classify the remaining Mangoes (which the child hasn’t seen). The child may take some time and make some mistakes as well, but will understand much better.
This method is Highly Practical!
The same goes for any Machine Learning Model!
But again there is a problem in the 2nd possibility as well, now what’s that problem about?
Considering an actual Machine Learning Model here, when we divide the Dataset into Training and Testing, we divide it randomly. Hence we can’t exactly control how it gets divided i.e., high chance of Bias taking place.
Considering the same examples of Mangoes above in a ML Model, when we divide it randomly, it gets divided such that 12 Mangoes for Training & 5 Mangoes for Testing.
Get Brian Daniel Thomas’s stories in your inbox
Join Medium for free to get updates from this writer.
Once again for the division there are again different possibilities:
1) All 10 ripe Mangoes and 2 unripe Mangoes are in the Training Set | The remaining 5 unripe mangoes are in Testing Set.
The Model will perform poorly here in Testing as it has learnt all the ripe mangoes and in Testing all the unripe Mangoes are present.
2) 8 ripe & 4 unripe Mangoes are in the Training Set | The remaining 2 ripe & 3 unripe mangoes are in the Testing Set.
Here the model will perform considerably better, but again while considering real-life datasets, the distribution here could be variable. When we check the accuracy of real-life datasets they differ each time we divide the dataset intro Training and Testing.
I’ll illustrate the same for you :
Consider the Hand-Written digits dataset from the sklearn’s repository.
Importing the packages, the Dataset and different Machine Learning algorithms
Python3
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifierimport numpy as np
from sklearn.datasets import load_digits
import matplotlib.pyplot as pltdigits = load_digits()
Checking the Data (Features) and the Target Values.
Python3
print("Data : \n", digits.data)
print ("")
print("Target : \n", digits.target)
Showing the actual Handwritten Image:
Python3
import matplotlib as mpl
import matplotlib.pyplot as pltplt.gray()
plt.matshow(digits.images[88])
plt.show()
Confirming the Number in the Image:
Python3
print("Data : \n", digits.data[88])
print("")
print("Target : ", digits.target[88])
Now we will perform the HOLDOUT Method i.e., the Train, Test Split!
Python3
from sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split(digits.data, digits.target, test_size=0.3)
We get the following ‘SCORES’ by implementing the HOLDOUT method once:
Python3
#Logistic Regression
lr = LogisticRegression(solver='liblinear',multi_class='ovr')
lr.fit(X_train, y_train)
lr.score(X_test, y_test)#Support Vector Machines
svm = SVC(gamma='auto')
svm.fit(X_train, y_train)
svm.score(X_test, y_test)#Random Forest
rf = RandomForestClassifier(n_estimators=40)
rf.fit(X_train, y_train)
rf.score(X_test, y_test)
When we implement HOLDOUT method once more (the Training & Testing Data gets changed)
Hence we get the following output!
The scores have changed right?
Why does this happen..?
When we perform the ‘train_test_split’ we divide certain random part of our dataset into the Training and Testing Set!
And when we run the ‘train_test_split’ once more, we again divide the data randomly into the Training and Testing Set!
So when the data keeps on changing in this manner the model learns even more and gets better at predictions! This is also an important concept named K-Fold Cross Validation!
