Synthetic Minority Over-sampling Technique (SMOTE)

Introduction

6 min readJan 12, 2024

In the data science industry , we often have to tackle with imbalanced data. It is one of the biggest problem in the data science. There are many solutions and methods to solve this problem like RUS, ROS , SMOTE. There are many blogs on this different balancing techniques. But in this blog we are going to focus on a hybrid method which is a combination of the SMOTE and under sampling method .I am actually working on a project where I have used this method during working on a mental health related problem. I am going to share its code , my insights on this method during working on this method.

Why balancing is required ?
What is SMOTE ?
SMOTE with Undersampling
Code of this method with my dataset
What are the results on my dataset ?
Conclusion

Why balancing is required ?

Whenever you train your model with imbalanced data , you often get wrong results after training. In this case , the model becomes bias towards the majority class and the model often becomes unable to capture the complexness of the data. In this cases , we often get high accuracy which leads to some misleading insights of the model. So we use different evaluation metrics like recall, precision and F1 score in this case . Lets see the classification report in my case on the imbalanced dataset along with confusion matrix.

In the above figure , we can see , though the accuracy is high the True Positive class has very less in number in compared to True Negative. Lets say you have fraud credit card detection task , then in this case true positive plays a crucial role than true negatives. `

What is SMOTE

SMOTE = Synthetic Minority Oversampling Technique .

Generally in Random Over sampling Technique we randomly choose minority samples to balance the data , as a result , the model come across the data multiple times and the overfitting occurs , reducing the performance of the model.

In SMOTE , using the KNN algorithm ,we synthesize the data to balance the data. It selects the minority examples that are close to the feature space. Then, it draws the line between the examples in the features space and draws a new sample at a point along that line.

Get Kousik Barnwal’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

In simple words, the algorithm selects the random example from the minority class and selects a random neighbor using K Nearest Neighbors. The synthetic example is created between two examples in the feature space.

Xs is synthesized sample, S is the class of the Minority sample.

See the illustrated Pictures below

Lets see the classification report and the confusion matrix on my dataset

You can see the better results than previous , the model is now predicting True positives value in a better way at the cost of precision. The reason is that using SMOTE, as it does not consider the majority class while creating synthetic examples. This can cause issues where there is a strong overlap between the classes. So the original paper suggests to use the combination of the SMOTE with Under sampling.

SMOTE with Undersampling

The hybrid resampling (SMOTE-+RUS) worked by synthesizing samples of the minority class based on their nearest neighbor, and then, RUS randomly reduced the majority sample to match the size of the minority class. As also in this case , we do the under sampling of the majority class , so the number of the synthesized sample generated during oversampling will be less in number compare to previous result, as a result less chance of noise introduced in the data. So the overall f1 score will be better.

In simple words,

Oversampling -> Recall gets better , precision gets lower as more minority samples are generated , who very less correlated with the majority class.

Undersampling -> Precision gets better, recall gets lower as it occurs in the information loss.

This trade off can be shown by Precision recall curve:

Here Train base line is basically balanced using SMOTE + RUS , test dataset is also imbalanced.

We maintain this trade off in this method optimally, to get a optimum model and get the higher f1 score.

Code of this method with my dataset

Basically I here use a ensemble model as a classifier ,which is a voting classifier with Xgboost, randomforest and adaboost classifier. Then using gridsearchCV I try to get the optimum hyperparameters for the oversampling, undersampling and the model . Lets see the code below:

# Define the pipeline with a RandomForestClassifier, SMOTE, and RandomUnderSampler
pipeline = Pipeline([
    ('under', RandomUnderSampler(random_state=42)),
    ('over', SMOTE(random_state=42)),
    ('model', RandomForestClassifier())
])

# Define parameter grid for grid search
param_grid = {
    'under__sampling_strategy': [0.25, 0.5, 0.75],
    'over__sampling_strategy': ['not majority','auto','not minority'],
    'model__n_estimators': [50, 100, 200]
}

# Perform grid search with cross-validation
grid_search = GridSearchCV(pipeline, param_grid, cv=5, scoring='f1', n_jobs=-1,error_score='raise')
grid_search.fit(X1, y)

# Print the best parameters
# best_params = grid_search.best_params_
# print(f'Best Parameters: {best_params}')

# Evaluate the model on the test set
y_pred = grid_search.predict(X_test)

# Display classification report
print("Classification Report:")
print(classification_report(test_y, y_pred))

What are the results on my dataset ?

The results are improved than the previous two ,lets see it.

Precision increases and recall decreases to get the maximum f1 score which is the harmonic mean of the precision and recall.

Conclusion

It appears that the precision in depression detection (my project) is relatively high, suggesting that when the model predicts depression, it is often correct. However, the recall and the area under the ROC curve (AUC) aren’t as high as desired. Classifiers commonly encounter challenges in optimizing both precision and recall, especially when dealing with imbalanced datasets, such as those in depression detection. It’s crucial to weigh the costs of different types of errors in the context of the problem at hand. In this scenario, a false negative(Type 2 error) (failing to identify depression when it’s present) may have significant consequences for an individual’s well-being, while a false positive(type 1 error) (incorrectly indicating depression) might lead to unnecessary distress or interventions, impacting the user’s mental health experience. But in this case , false negative is worse than the false positive.

Hope you understand this blog. If you have any doubts , write in the comments below. I will answer them. Keep Learning & Keep Growing !!!