Data Preprocessing - Imputation

In this article, I will explain you what Imputation and the various techniques available to handle the null values in the data (Imputation).

What is Imputation?

Imputation refers to the process of filling/substituting the missing values in the data to retain all the information of the dataset. In the data, Missing values can occur in various forms, such as blank entries, NaN (Not a Number) values, or placeholders like “N/A” or “Unknown”.

Get Leela Jagadeeswar P’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

It is one of the important things in machine learning because most of the algorithms in ml cannot handle missing values.

https://upload.wikimedia.org/wikipedia/commons/e/e8/Imputation.jpg

The techniques available for Imputation:

• Fillna with Mean/Median/Mode/Fixed Value/min or max value: In this case, we are going to fill the missing values in the data either with constant value or using statistical methods such as mean, median or mode.

#Filling with Mean
df[col_name] = df[col_name].fillna(df[col_name].mean())

#Filling with median
df[col_name] = df[col_name].fillna(df[col_name].median())

#Filling with Mode
df[col_name] = df[col_name].fillna(df[col_name].mode()[0])

#Filling with Constant value
df[col_name] = df[col_name].fillna(constant_value)

#Filling with Constant value
df[col_name] = df[col_name].fillna(df[col_name].min())

#Filling with Constant value
df[col_name] = df[col_name].fillna(df[col_name].max())

• Forward/Backward Fill: Here, Forward fill (Last observation carried forward) fills the missing values in the data with the previous known value. Backward fill (Next observation carried backward) fills missing values with the next known value. These techniques are mainly useful when missing values occur in sequences or it is a time-series data.

#Filling with Forward fill
df[col_name] = df[col_name].fillna(method= "ffill")

#Filling with Backward fill
df[col_name] = df[col_name].fillna(method= "bfill")

• K-Nearest Neighbors (KNN) Imputer: It is a multivariate data imputation technique used for filling the missing values in the data by using the K-Nearest Neighbours approach.

import pandas as pd
from sklearn.impute import KNNImputer

# initialize KNN imputer object with k=3
imputer = KNNImputer(n_neighbors=3)

# fit and transform the dataframe to impute missing values
imputed_df = pd.DataFrame(imputer.fit_transform(df), columns=df.columns)

#Imputed dataframe
imputed_df.head()

• SimpleImputer: SimpleImputer is just like filling the missing values in the data with Fillna using constant value or statistical methods. So, we just need to specify the type of statistical method to the parameter ‘strategy’.

from sklearn.impute import SimpleImputer
simple_imputer = SimpleImputer(missing_values= np.nan, strategy= statistical_method)
df_imputed = simple_imputer.fit_transform(df)

• Interpolation: Mainly in this method, we estimate the missing values based on the values of other data points. Common interpolation techniques include linear interpolation, polynomial interpolation, and spline interpolation. These techniques assume a linear or smooth relationship between data points and can be effective when missing values are missing at random (MAR).

#By default it will take 'linear' method to impute
df[col_name] = df[col_name].interpolate()