Missing Data in Python? No Problem!

Master the Best Techniques to Detect and Impute Missing Data for Best Practices

6 min readNov 11, 2024

Feel free to join my LinkedIn group here

In the world of data analysis and machine learning, missing values are a common challenge that can significantly impact the accuracy and reliability of your results.

Identifying and handling missing values is essential for building robust models and accurate analyses.

Hi, my name is CyCoderX and in this article, we’ll explore how to detect, analyze, and address missing values in datasets using Python’s popular libraries, Pandas and NumPy.

Let’s dive in!

I write articles for everyone to enjoy, and I’d love your support by following me for more Python, SQL, Data Engineering and Data Science content.😊

Identifying Missing Values

Before dealing with missing values, it’s essential to understand where and how they exist in your dataset. Missing values are typically represented as NaN (Not a Number) in Pandas and NumPy.

Creating a Sample Dataset

import pandas as pd
import numpy as np

# Creating a sample dataset
data = {
    'Name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'],
    'Age': [25, np.nan, 30, np.nan, 28],
    'Salary': [50000, 54000, np.nan, 62000, 58000]
}
df = pd.DataFrame(data)

print("Sample DataFrame:\n", df)

Total sum of Missing Values

To determine how many missing values you have in the whole dataframe you can use the following query.

import pandas as pd
import numpy as np

# Creating a sample dataset
data = {
    'Name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'],
    'Age': [25, np.nan, 30, np.nan, 28],
    'Salary': [50000, 54000, np.nan, 62000, 58000]
}
df = pd.DataFrame(data)

# print("Sample DataFrame:\n", df)
# Calculating the total number of null values in the DataFrame
total_null_values = df.isnull().sum().sum()


print("Total number of null values in the DataFrame:", total_null_values)

# Output: Total number of null values in the DataFrame: 
# 3

Total sum of Missing Values in Each Column

To determine how many missing values you have you can use the following query.

import pandas as pd
import numpy as np

# Creating a sample dataset
data = {
    'Name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'],
    'Age': [25, np.nan, 30, np.nan, 28],
    'Salary': [50000, 54000, np.nan, 62000, 58000]
}
df = pd.DataFrame(data)

# print("Sample DataFrame:\n", df)



# Summing up all the null values in each column
null_values_sum = df.isnull().sum()


print("Sum of null values in each column:\n", null_values_sum)

"""
Output:
Sum of null values in each column:
 Name      0
Age       2
Salary    1
dtype: int64
"""

Checking the Percentage of Missing Values

Sometimes its a good practice to determine the severity of missing data, by calculating the percentage of missing values per column.

# Percentage of missing values per column
missing_percentage = (df.isnull().sum() / len(df)) * 100
print("\nPercentage of missing values per column:\n", missing_percentage)

Essential Pandas Cheatsheet

Master Pandas basics for data creation, loading, selection, and analysis.

Handling Missing Values

Once missing values are identified, there are several strategies for handling them depending on the dataset and analysis goals. Here are some of the most common approaches:

Dropping Missing Values

In some cases, it might be suitable to drop or delete the rows or columns with the missing values, particularly when dealing with a significantly large dataset and a few missing values.

Drop Rows with Missing Values: This removes any row that has at least one missing value.

# Drop rows with any missing values
df_dropped_rows = df.dropna()
print("\nDataFrame after dropping rows with missing values:\n", df_dropped_rows)

Drop Columns with Missing Values: This removes entire columns that contain missing values.

# Drop columns with any missing values
df_dropped_columns = df.dropna(axis=1)
print("\nDataFrame after dropping columns with missing values:\n", df_dropped_columns)

Arguments to consider:

Get CyCoderX’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

how: This determines if a row is dropped when all or any values are NaN.

any: (default) drops the row if any value is NaN.
all: drops the row if all values are NaN.

thresh: Requires at least that many non-NaN values to retain the row.

thresh=n: Keep rows with at least n non-NaN values.

subset: Specifies which columns to look for missing values.

subset=['col1', 'col2']: Only consider these columns/rows for NaNs.

inplace: If True, performs operation in place without returning a new DataFrame.

inplace=True: Modify the DataFrame in place.

Essential NumPy Cheatsheet

Quickly learn and reference essential NumPy functions and techniques.

Hey, did you know we have a LinkedIn Group? Join us to connect with like-minded professionals and stay ahead of industry trends!

Filling Missing Values

When dropping values is not an option (e.g., if a column has many missing values), you can replace or “impute” them with various techniques.

Fill with a Specific Value: You can fill all missing values with a specified value (e.g., 0 or "Unknown").

# Fill missing values with a specific value
df_filled_zero = df.fillna(0)
print("\nDataFrame after filling missing values with 0:\n", df_filled_zero)

Fill with Mean, Median, or Mode: For numerical data, it’s common to fill missing values with the mean, median, or mode.

# Fill missing values with the mean of the column
df_filled_mean = df.copy()
df_filled_mean['Age'].fillna(df['Age'].mean(), inplace=True)
print("\nDataFrame after filling 'Age' missing values with mean:\n", df_filled_mean)

# Sometimes you might need to use np.mean instead of .mean()

# Fill missing values with the mean of the 'Age' column using np.mean
df_filled_mean = df.copy()
df_filled_mean['Age'].fillna(np.mean(df['Age']), inplace=True)
print("\nDataFrame after filling 'Age' missing values with mean:\n", df_filled_mean)

Note: sometimes you will need to use np.mean() instead of .mean() and also have in mind you cant use axis=1 for rows and axis=0 for columns.

Use np.mean for NumPy arrays and for operations requiring high performance on large datasets or multi-dimensional arrays.
Use .mean() for Pandas DataFrames and Series, especially when you need to leverage DataFrame-specific functionalities like handling missing values.

Forward and Backward Filling: Another approach is to use neighboring values to fill in the gaps.

Forward Fill (ffill): Fills with the previous row’s value.
Backward Fill (bfill): Fills with the next row’s value.

# Forward fill (fill missing values with previous row's value)
df_ffill = df.fillna(method='ffill')
print("\nDataFrame after forward fill:\n", df_ffill)

# Backward fill (fill missing values with next row's value)
df_bfill = df.fillna(method='bfill')
print("\nDataFrame after backward fill:\n", df_bfill)

Mastering Duck Typing in Python

Learn how Python’s duck typing allows for dynamic, adaptable, and clean code.

Advanced Techniques for Handling Missing Values

For datasets where the simple methods above don’t suffice, more advanced techniques can provide accurate imputation.

Using Interpolation for Numerical Data

Interpolation estimates missing values based on other values in the dataset, which can be useful for time series or continuous data.

# Interpolate missing values
df_interpolated = df.interpolate()
print("\nDataFrame after interpolation:\n", df_interpolated)

Using Scikit-Learn’s Imputer for Machine Learning

The SimpleImputer class in scikit-learn provides various strategies for imputing missing values, such as mean, median, and most frequent values.

from sklearn.impute import SimpleImputer

# Impute missing values in the Salary column with the median
imputer = SimpleImputer(strategy='median')
df[['Salary']] = imputer.fit_transform(df[['Salary']])

print("\nDataFrame after imputing 'Salary' with median:\n", df)

How to Generate Requirements.txt for Your Python Project

A guide to generating requirements.txt for Python projects

Conclusion

Handling missing values is a crucial part of any data analysis pipeline. Pandas and NumPy offer a wide range of techniques, from simple replacements to advanced imputations, allowing you to handle missing data effectively.

Selecting the right method depends on the characteristics of your dataset and the goals of your analysis. By addressing missing values thoughtfully, you’ll create more accurate and reliable data models and analyses.

Happy Coding!

Note that I am not sponsored or affiliated with any services or products mentioned; my advice comes from my own experiences. Always conduct your own research and verify online information.

4 Mistakes Every Beginner Developer Should Avoid

Learn how to avoid the most frequent errors new developers often make.

Final Words:

Thank you for taking the time to read my article.
This article was first published on Medium by CyCoderX.

Hi, I’m CyCoderX! I’m a data engineer passionate about sharing knowledge, I write articles about Python, SQL, Data Science, Data Engineering, lifestyle and more!