Don’t Let Missing Data Trip You Up: Your Guide to Handling It
In the realm of machine learning, the quality of your dataset plays a pivotal role in the performance and accuracy of your models. However, real-world data is often messy, containing missing values that can impede the learning process. When preparing training and test datasets, it’s crucial to address these missing values effectively to ensure robust model performance. In this article, we’ll delve into various approaches for handling missing values in your datasets, accompanied by Python code snippets for each technique.
Understanding Missing Values
Before diving into the approaches, let’s understand what missing values are. Missing values occur when no data is stored for a particular observation in one or more variables. These gaps can arise due to various reasons such as data corruption, human error during data collection, or simply because certain information was not collected.
Approach 1: Removing Rows or Columns with Missing Values
This approach involves straightforwardly removing rows or columns from the dataset that contain missing values. The rationale behind this method is to ensure that the dataset used for training or testing the machine learning model is complete, albeit at the cost of losing potentially valuable information.
import pandas as pd
# Load the Titanic dataset from publicly available URL.
# This dataset contains information about passengers on the Titanic ocean liner.
titanic_df = pd.read_csv('https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv')
# Display the first few rows of the dataset to observe its structure and contents.
print("Original Titanic Dataset:")
print(titanic_df.head())
# Count number of rows and columns before removal
rows_before_rows_removed, cols_before_rows_removed = titanic_df.shape
print("\nNumber of Rows Before Removal:")
print(rows_before_rows_removed)
print("\nNumber of Columns Before Removal:")
print(cols_before_rows_removed)
# Check for missing values in the dataset
print("\nMissing Values Before Removal:")
print(titanic_df.isnull().sum())In the provided Python code snippet, the Pandas library is utilized to load the dataset into a DataFrame (titanic_df). We then count the number of rows and columns in the original dataset. We use the isnull().sum() method to check for missing values in the dataset. This provides a count of missing values for each column.
Number of Rows Before Removal:
891
Number of Columns Before Removal:
12
Missing Values Before Removal:
PassengerId 0
Survived 0
Pclass 0
Name 0
Sex 0
Age 177
SibSp 0
Parch 0
Ticket 0
Fare 0
Cabin 687
Embarked 2
dtype: int64# Approach 1: Removing Rows with Missing Values
# Drop rows with missing values
cleaned_rows_df = titanic_df.dropna(axis=0)
# Count number of rows and columns after removing rows with missing values
rows_after_rows_removed, cols_after_rows_removed = cleaned_rows_df.shape
print("\nNumber of Rows After Removal:")
print(rows_after_rows_removed)
print("\nNumber of Columns After Removal:")
print(cols_after_rows_removed)
# Check for missing values in the cleaned dataset
print("\nMissing Values After Removal of Required Rows:")
print(cleaned_rows_df.isnull().sum())We use the dropna() method with axis=0 to remove rows containing missing values. Setting axis=0 ensures that rows are dropped. The cleaned dataset is stored in cleaned_rows_df.
Number of Rows After Removal:
183
Number of Columns After Removal:
12
Missing Values After Removal of Required Rows:
PassengerId 0
Survived 0
Pclass 0
Name 0
Sex 0
Age 0
SibSp 0
Parch 0
Ticket 0
Fare 0
Cabin 0
Embarked 0
dtype: int64On checking the cleaned dataset, we find that there are 0 null values in the cleaned dataset. However, the number of rows have reduced from 891 to just 183.
# Approach 2: Removing Columns with Missing Values
# Drop columns with missing values
cleaned_cols_df = titanic_df.dropna(axis=1)
# Count number of rows and columns after removing columns with missing values
rows_after_columns_removed, cols_after_columns_removed = cleaned_cols_df.shape
print("\nNumber of Rows After Removal:")
print(rows_after_columns_removed)
print("\nNumber of Columns After Removal:")
print(cols_after_columns_removed)
# Check for missing values in the cleaned dataset
print("\nMissing Values After Removal of Required Columns:")
print(cleaned_cols_df.isnull().sum())We use the dropna() method with axis=1 to remove columns containing missing values. Setting axis=1 ensures that columns are dropped. The cleaned dataset is stored in cleaned_cols_df.
Number of Rows After Removal:
891
Number of Columns After Removal:
9
Missing Values After Removal of Required Columns:
PassengerId 0
Survived 0
Pclass 0
Name 0
Sex 0
SibSp 0
Parch 0
Ticket 0
Fare 0
dtype: int64On checking the cleaned dataset, we find that there are 0 null values in the cleaned dataset. However, the number of columns have reduced from 12 to 9 (‘Age’, ‘Cabin’, & ‘Embarked’ columns which had missing values are removed).
Approach 2: Imputation Techniques
Imputation techniques aim to fill in missing values with substituted values derived from the observed data. There are several strategies for imputation, including:
I. Mean/Median/Mode Imputation:
These techniques involve replacing missing values with the mean, median, or mode of the respective feature/column. They are simple and effective, especially for numerical or categorical variables with no underlying trend or distribution.
Mean Imputation: Missing values are replaced with the mean of the non-missing values in the column.
# Approach: Mean Imputation
# Filter out non-numeric columns
numeric_cols = titanic_df.select_dtypes(include=['int64', 'float64']).columns
# Impute missing values with the mean of each column
mean_imputed_df = titanic_df.copy()
mean_imputed_df[numeric_cols] = titanic_df[numeric_cols].fillna(titanic_df[numeric_cols].mean())
# Display the number of missing values after mean imputation
print("\nMissing Values After Mean Imputation:")
print(mean_imputed_df.isnull().sum())
# Display the first few rows of the dataset after mean imputation
print("\nDataset After Mean Imputation:")
print(mean_imputed_df.head())Missing Values After Mean Imputation:
PassengerId 0
Survived 0
Pclass 0
Name 0
Sex 0
Age 0
SibSp 0
Parch 0
Ticket 0
Fare 0
Cabin 687
Embarked 2
dtype: int64The imputation works on the numerical column with missing values i.e. ‘Age’. Similar process happens when median and mode imputation techniques are applied.
Median Imputation: Missing values are replaced with the median of the non-missing values in the column. This is less sensitive to outliers compared to mean imputation and is often preferred when the data is skewed.
# Approach: Median Imputation
# Impute missing values with the median of each column
median_imputed_df = titanic_df.copy()
median_imputed_df[numeric_cols] = titanic_df[numeric_cols].fillna(titanic_df[numeric_cols].mean())
# Display the number of missing values after median imputation
print("\nMissing Values After Median Imputation:")
print(median_imputed_df.isnull().sum())
# Display the first few rows of the dataset after median imputation
print("\nDataset After Median Imputation:")
print(median_imputed_df.head())Missing Values After Median Imputation:
PassengerId 0
Survived 0
Pclass 0
Name 0
Sex 0
Age 0
SibSp 0
Parch 0
Ticket 0
Fare 0
Cabin 687
Embarked 2
dtype: int64Mode Imputation: Used for categorical variables, missing values are replaced with the most frequent category (mode) observed in the column.
# Approach: Mode Imputation
# Impute missing values with the mode of each column
mode_imputed_df = titanic_df.copy()
mode_imputed_df[numeric_cols] = titanic_df[numeric_cols].apply(lambda x: x.fillna(x.mode()[0]))
# Display the number of missing values after mode imputation
print("\nMissing Values After Mode Imputation:")
print(mode_imputed_df.isnull().sum())
# Display the first few rows of the dataset after mode imputation
print("\nDataset After Mode Imputation:")
print(mode_imputed_df.head())Missing Values After Mode Imputation:
PassengerId 0
Survived 0
Pclass 0
Name 0
Sex 0
Age 0
SibSp 0
Parch 0
Ticket 0
Fare 0
Cabin 687
Embarked 2
dtype: int64II. Forward Fill (ffill) and Backward Fill (bfill):
Get Gaurav Jain’s stories in your inbox
Join Medium for free to get updates from this writer.
These techniques involve propagating the last observed non-missing value forward or backward to fill in missing values.
Forward Fill (ffill): Missing values are replaced with the value of the previous non-missing element in the column. This is useful for datasets where missing values occur in sequences or time series data.
import pandas as pd
import numpy as np
# Create a synthetic time series dataset with missing values
dates = pd.date_range(start='2024-01-01', end='2024-01-10')
data = {'value': [1, np.nan, 3, 4, np.nan, 6, np.nan, 8, 9, np.nan]}
time_series_df = pd.DataFrame(data, index=dates)
# Display the original time series dataset
print("Original Time Series Dataset:")
print(time_series_df)Original Time Series Dataset:
value
2024-01-01 1.0
2024-01-02 NaN
2024-01-03 3.0
2024-01-04 4.0
2024-01-05 NaN
2024-01-06 6.0
2024-01-07 NaN
2024-01-08 8.0
2024-01-09 9.0
2024-01-10 NaN# Apply Forward Fill Technique
ffill_time_series_df = time_series_df.ffill()
# Display the time series dataset after applying forward fill
print("\nTime Series Dataset After Forward Fill:")
print(ffill_time_series_df)Time Series Dataset After Forward Fill:
value
2024-01-01 1.0
2024-01-02 1.0
2024-01-03 3.0
2024-01-04 4.0
2024-01-05 4.0
2024-01-06 6.0
2024-01-07 6.0
2024-01-08 8.0
2024-01-09 9.0
2024-01-10 9.0Backward Fill (bfill): Missing values are replaced with the value of the next non-missing element in the column. Similar to forward fill, this method is helpful for sequential or time-based datasets.
# Apply Backward Fill Technique
bfill_time_series_df = time_series_df.bfill()
# Display the time series dataset after applying backward fill
print("\nTime Series Dataset After Backward Fill:")
print(bfill_time_series_df)Time Series Dataset After Backward Fill:
value
2024-01-01 1.0
2024-01-02 3.0
2024-01-03 3.0
2024-01-04 4.0
2024-01-05 6.0
2024-01-06 6.0
2024-01-07 8.0
2024-01-08 8.0
2024-01-09 9.0
2024-01-10 NaNIII. Interpolation:
Interpolation techniques estimate missing values based on the surrounding data points. They are particularly useful when the data exhibits a trend or pattern.
import pandas as pd
# Load the Iris dataset from seaborn
iris_df = pd.read_csv('https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv')
# Introduce missing values
iris_df.loc[1:3, 'sepal_length'] = None
iris_df.loc[3:5, 'petal_length'] = None
# Display the first few rows of the dataset
print("Original Iris Dataset with Missing Values:")
print(iris_df.head())
# Check for missing values in the dataset
print("\nMissing Values Before Interpolation:")
print(iris_df.isnull().sum())We load the Iris dataset using Pandas read_csv() function. We then introduce missing values into the dataset by setting certain ranges of ‘sepal_length’ and ‘petal_length’ columns to None. The first few rows of the dataset is printed to observe its structure and contents. We finally use the isnull().sum() method to check for missing values in the dataset before applying interpolation. This provides a count of missing values for each column.
Original Iris Dataset with Missing Values:
sepal_length sepal_width petal_length petal_width species
0 5.1 3.5 1.4 0.2 setosa
1 NaN 3.0 1.4 0.2 setosa
2 NaN 3.2 1.3 0.2 setosa
3 NaN 3.1 NaN 0.2 setosa
4 5.0 3.6 NaN 0.2 setosa
Missing Values Before Interpolation:
sepal_length 3
sepal_width 0
petal_length 3
petal_width 0
species 0
dtype: int64Linear Interpolation: Missing values are estimated as the linearly interpolated value between the nearest non-missing data points. It assumes a linear relationship between consecutive data points.
# Linear Interpolation
linear_interpolated_df = iris_df.interpolate(method='linear')
# Display the first few rows of the dataset after linear interpolation
print("\nDataset After Linear Interpolation:")
print(linear_interpolated_df.head())
# Check for missing values after linear interpolation
print("\nMissing Values After Linear Interpolation:")
print(linear_interpolated_df.isnull().sum())We use the interpolate() method with method='linear' to perform linear interpolation.
Dataset After Linear Interpolation:
sepal_length sepal_width petal_length petal_width species
0 5.100 3.5 1.400 0.2 setosa
1 5.075 3.0 1.400 0.2 setosa
2 5.050 3.2 1.300 0.2 setosa
3 5.025 3.1 1.325 0.2 setosa
4 5.000 3.6 1.350 0.2 setosa
Missing Values After Linear Interpolation:
sepal_length 0
sepal_width 0
petal_length 0
petal_width 0
species 0
dtype: int64Polynomial Interpolation: This method fits a polynomial function to the non-missing data points and estimates missing values based on the polynomial curve. It can capture more complex patterns compared to linear interpolation.
# Polynomial Interpolation
polynomial_interpolated_df = iris_df.interpolate(method='polynomial', order=2)
# Display the first few rows of the dataset after polynomial interpolation
print("\nDataset After Polynomial Interpolation:")
print(polynomial_interpolated_df.head())
# Check for missing values after polynomial interpolation
print("\nMissing Values After Polynomial Interpolation:")
print(polynomial_interpolated_df.isnull().sum())We use the interpolate() method with method='polynomial' and order=2 to perform polynomial interpolation of order 2.
Dataset After Polynomial Interpolation:
sepal_length sepal_width petal_length petal_width species
0 5.100000 3.5 1.400000 0.2 setosa
1 4.646594 3.0 1.400000 0.2 setosa
2 4.478792 3.2 1.300000 0.2 setosa
3 4.596594 3.1 1.239484 0.2 setosa
4 5.000000 3.6 1.241698 0.2 setosa
Missing Values After Polynomial Interpolation:
sepal_length 0
sepal_width 0
petal_length 0
petal_width 0
species 0
dtype: int64Approach 3: Advanced Techniques
Advanced techniques involve leveraging more complex algorithms, such as machine learning models, to predict missing values based on the available data. These methods are more sophisticated and may offer better performance but require more computational resources.
One such method is K-Nearest Neighbors (KNN) imputation, where for each missing value, the algorithm identifies the k nearest neighbors based on similarity measures (e.g., Euclidean distance) and computes the missing value as the average (or weighted average) of these neighbors’ values.
import pandas as pd
from sklearn.datasets import load_iris
from sklearn.impute import KNNImputer
# Load the Iris dataset
iris = load_iris()
# Create a DataFrame from the dataset
iris_df = pd.DataFrame(iris.data, columns=iris.feature_names)
# Introduce missing values
iris_df.loc[1:2, 'sepal length (cm)'] = None
iris_df.loc[3:5, 'petal length (cm)'] = None
# Display the first few rows of the dataset
print("Original Iris Dataset with Missing Values:")
print(iris_df.head())
# Check for missing values in the dataset
print("\nMissing Values Before Imputation:")
print(iris_df.isnull().sum())
# KNN Imputation
# Instantiate KNN imputer
knn_imputer = KNNImputer(n_neighbors=5)
# Perform KNN imputation
iris_imputed_array = knn_imputer.fit_transform(iris_df)
# Create a DataFrame from the imputed array
iris_imputed_df = pd.DataFrame(iris_imputed_array, columns=iris_df.columns)
# Display the first few rows of the imputed dataset
print("\nIris Dataset After KNN Imputation:")
print(iris_imputed_df.head())
# Check for missing values after imputation
print("\nMissing Values After KNN Imputation:")
print(iris_imputed_df.isnull().sum())In the provided Python code, scikit-learn’s KNNImputer is utilized to perform KNN imputation. The n_neighbors parameter specifies the number of neighbors to consider when imputing missing values. The fit_transform() method fits the imputer to the dataset and replaces missing values with predicted values based on the nearest neighbors.
Original Iris Dataset with Missing Values:
sepal length (cm) sepal width (cm) petal length (cm) petal width (cm)
0 5.1 3.5 1.4 0.2
1 NaN 3.0 1.4 0.2
2 NaN 3.2 1.3 0.2
3 4.6 3.1 NaN 0.2
4 5.0 3.6 NaN 0.2
Missing Values Before Imputation:
sepal length (cm) 2
sepal width (cm) 0
petal length (cm) 3
petal width (cm) 0
dtype: int64
Iris Dataset After KNN Imputation:
sepal length (cm) sepal width (cm) petal length (cm) petal width (cm)
0 5.10 3.5 1.40 0.2
1 4.60 3.0 1.40 0.2
2 4.72 3.2 1.30 0.2
3 4.60 3.1 1.46 0.2
4 5.00 3.6 1.40 0.2
Missing Values After KNN Imputation:
sepal length (cm) 0
sepal width (cm) 0
petal length (cm) 0
petal width (cm) 0
dtype: int64Handling missing values is a critical step in data preprocessing for machine learning tasks. Each approach has its strengths and weaknesses, and the choice depends on the specific characteristics of the dataset and the requirements of the problem at hand. Experimentation and careful consideration are key to determining the most suitable method for your data. With the techniques discussed in this article and the accompanying Python code snippets, you’re equipped to tackle missing values effectively in your machine learning projects.
