Missing Data? No Problem! Improve your Data Analysis with Iterative Imputer: A Practical Guide for Beginners
We all know that feeling — you finally get your hands on a juicy data set, ready to uncover some hidden insights, only to find that it’s riddled with missing values. It’s enough to make even the most seasoned analyst want to pull their hair out! Fortunately for us, there is a powerful tool in data analysis arsenal that can help overcome this hurdle — Iterative imputation.
Iterative Imputation is the process of filling in missing data points with estimates based on the available data, and it can be a game-changer for analysis.
In this practical guide for beginners, we’ll be diving deep into one of the most effective and user-friendly imputation techniques out there — iterative imputation. Whether you’re a seasoned pro or just starting out on your data analysis journey, this guide will provide you with the knowledge and tools you need to improve your analysis and make the most out of your data.
In this tutorial, we’ll be working with a dataset called the California Housing Prices dataset. This dataset was created by Kaggle to help people learn about Machine Learning algorithms. However, we made some changes to the dataset to suit our needs for this tutorial. If you want to follow along with this tutorial and use the same dataset we’re using, you can download the modified version from this link.
# Packages
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import random
# Read Data
data1 = pd.read_csv("California_Housing_Data_Part1.csv")
print(data1.head(5))
In order to illustrate the challenge, we intentionally introduced around 300 missing values in our dataset, at random. This represents approximately 30% of the data, and the missing values were only added to the column that contains information about the median house value.
# About 300 random indices were chosen
indices = random.sample(range(len(data1)), 300)
# Indices were utilized to replace values with NaN
data1.loc[indices, 'median_house_value'] = np.nan
# Check Null values
data1.info()
Next, we need to conduct standardization on our dataset. Standardization is a process that involves converting our data to a standardized format, which uses a consistent scale that’s easier to compare and analyze. This step is particularly important for both data visualization and machine learning purposes, as it can be difficult to compare features and relationships that are on different scales. By standardizing our dataset, we can ensure that all of our features are on the same scale and can be compared accurately. This can help us to identify patterns and relationships effectively, and help us improve the accuracy of our models.
# Standardize dataset
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
data1[data1.columns] = scaler.fit_transform(data1[data1.columns])
print(data1.head(5))
Visualizing missing values is a good practice, as it provides a clear indication of any problems or issues with our data. In this step, we’ll perform two visualizations: a heatmap for analyzing missing values in columns, and a line chart for observing the null values in the Median House Value column (the only one with missing values, for now).
# Visualization - Before Imputation
# Bar Plot for Missing Values in each column
plt.figure(figsize=(15, 8))
sns.heatmap(data1.isnull(), cmap='viridis', cbar_kws={'ticks': [0, 1]}, vmin=0, vmax=1)
plt.title('Heat Map Describing Missing Data')
plt.xlabel('Dataset Columns')
plt.ylabel('Observation/Index values')
plt.show()
# Line Plot for Missing Values Column
plt.figure(figsize=(15, 8))
plt.plot(data1['median_house_value'], label='Median House Value', linewidth=2, color = 'Green')
plt.xlabel('Index Values')
plt.ylabel('Median House Value')
plt.title('Line Plot Describing Median House Data')
plt.show()
When we analyze the heatmap, we can observe that only the Median House Values column contains missing data. Moreover, these missing values are not concentrated in one area, but are randomly spread out throughout the data. This is confirmed when we visualize the column using a line plot and observe that there are multiple breaks in the line chart, indicating the presence of missing values at different indexes.
Get Usama Naeem Toor’s stories in your inbox
Join Medium for free to get updates from this writer.
Iterative Imputation is an advanced method for filling in missing values in a dataset by creating a model based on the non-missing values and then using that model to make predictions for the missing values. By default, Iterative Imputation uses the ‘BayesianRidge’ estimator to create this model, but it is also possible to use other estimators like Linear Regression, ElasticNet, and more. Check documentation!
# Iterative Imputer from Scikit-learn
from sklearn.impute import IterativeImputer
# Setup the Imputer and perform Imputation
imputer = IterativeImputer()
data1[data1.columns] = imputer.fit_transform(data1[data1.columns])
data1.info()
After importing the Iterative Imputer class from scikit-learn, we create an instance of the class called “imputer”. We then use the fit_transform method to train the imputer on our data, which includes the missing values. During the process, the imputer creates a model based on the estimator in the IterativeImputer class, to predict the missing values using the Non-Null values. We then replace the missing values with these predicted values, and, by doing so, we obtain data that is based on the patterns it learns from the existing data.
But, are these new values appropriate? Do they make sense according to the data? We perform visualization once again to check!
# Visualization - After Imputation
# Bar Plot for Missing Values in each column
plt.figure(figsize=(15, 8))
sns.heatmap(data1.isnull(), cmap='viridis', cbar_kws={'ticks': [0, 1]}, vmin=0, vmax=1)
plt.title('Heat Map Describing Missing Data')
plt.xlabel('Dataset Columns')
plt.ylabel('Observation/Index values')
plt.show()
# Line Plot for Missing Values Column
plt.figure(figsize=(15, 8))
plt.plot(data1['median_house_value'], label='Median House Value', linewidth=2, color = 'Green')
plt.xlabel('Index Values')
plt.ylabel('Median House Value')
plt.title('Line Plot Describing Median House Data')
plt.show()
Great news! The visualizations indicate that we have successfully filled in the missing values with the imputation method. Moreover, the new values fit in well with the other non-null values, which means that the estimated values from imputation are reliable.
Source code can be found here.
