Visualizing Null Values with Heatmaps

Python Quickies #40

5 min readSep 20, 2025

Missing data is one of the most common challenges in data analysis. Whether you’re dealing with survey responses, sensor readings, or web analytics, null values can significantly impact your analysis and model performance. While pandas provides methods like info() and isnull() to detect missing values, sometimes you need a more intuitive, visual approach to understand the patterns of missingness in your dataset.

Enter heatmaps — a powerful visualization technique that can transform your understanding of null value patterns from abstract numbers into clear insights.

Advantages of Null Value Visualistions

Before diving into the how, let’s understand the why of it. Visualizing null values helps you to:

Identify patterns: Are certain columns consistently missing together? Do nulls cluster in specific time periods?
Assess data quality: Quickly spot systematic data collection issues.
Make informed decisions: Choose appropriate imputation strategies based on missingness patterns.
Communicate findings: Show stakeholders data quality issues in an intuitive format.

Now let us see an example implementation.

Setting Up Your Environment

First, let’s import the necessary libraries:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from datetime import datetime, timedelta

Creating Sample Data with Realistic Missing Patterns

Let’s create a dataset that mimics real-world scenarios where null values often occur:

# using random number generator with random seed for reproducibility
rng=np.random.default_rng(42)

# Create date range
dates = pd.date_range('2023-01-01', periods=100, freq='D')
# Generate sample data
data = {
    'date': dates,
    'temperature': rng.normal(20, 5, 100),
    'humidity': rng.normal(60, 10, 100),
    'pressure': rng.normal(1013, 20, 100),
    'wind_speed': rng.exponential(5, 100),
    'rainfall': rng.exponential(2, 100),
    'user_rating': rng.integers(1, 6, 100),
    'comments': ['Comment ' + str(i) for i in range(100)]
}
df = pd.DataFrame(data)
# Introduce realistic missing patterns
# Pattern 1: Random missing values (equipment malfunction)
df.loc[np.random.choice(df.index, 15, replace=False), 'temperature'] = np.nan
# Pattern 2: Consecutive missing values (maintenance period)
df.loc[30:35, 'pressure'] = np.nan
# Pattern 3: Weekend missing values (no staff to collect data)
weekend_mask = df['date'].dt.dayofweek.isin([5, 6])
df.loc[weekend_mask & (rng.random(len(df)) < 0.3), 'humidity'] = np.nan
# Pattern 4: Correlated missing values (same sensor system)
corr_missing_idx = rng.choice(df.index, 8, replace=False)
df.loc[corr_missing_idx, ['wind_speed', 'rainfall']] = np.nan
# Pattern 5: Optional fields (user didn't provide)
df.loc[rng.choice(df.index, 25, replace=False), 'user_rating'] = np.nan
df.loc[rng.choice(df.index, 35, replace=False), 'comments'] = np.nan
print("Dataset shape:", df.shape)
print("\nMissing values per column:")
print(df.isnull().sum())

Output:

Basic Heatmap Visualization

Now let’s create our first heatmap to visualize null values:

# Create a boolean mask for null values
null_mask = df.isnull()

# Create the heatmap
plt.figure(figsize=(12, 8))
sns.heatmap(null_mask, 
            cbar_kws={'label': 'Missing Data'},
            cmap='viridis')
plt.title('Missing Values Heatmap', fontsize=16, pad=20)
plt.xlabel('Columns', fontsize=12)
plt.ylabel('Rows', fontsize=12)
plt.tight_layout()
plt.show()

Output:

This basic heatmap gives you an immediate overview, but we can make it much more informative.

Enhanced Heatmap with Better Insights

Let’s create a more sophisticated visualization that provides deeper insights:

# Create a figure with subplots
fig, axes = plt.subplots(2, 2, figsize=(16, 12))

# 1. Overall missing pattern
axes[0, 0].set_title('Missing Values Pattern')
sns.heatmap(df.isnull(), cbar_kws={'label': 'Missing Data'},ax=axes[0, 0])

# 2. Missing values percentage by column
axes[0, 1].set_title('Percentage of Missing Values by Column')
axes[0, 1].set_xlabel('Percentage Missing')
missing_percent = (df.isnull().sum() / len(df)) * 100
missing_df = pd.DataFrame({'Column': missing_percent.index, 'Missing_Percentage': missing_percent.values})
sns.barplot(data=missing_df, y='Column', x='Missing_Percentage', hue='Column', palette='viridis', ax=axes[0, 1])

# 3. Missing values over time
axes[1, 0].set_title('Missing Values Over Time')
axes[1, 0].set_xlabel('Date')
axes[1, 0].set_ylabel('Number of Missing Values')
axes[1, 0].tick_params(axis='x', rotation=45)
df_time = df.copy()
df_time['missing_count'] = df_time.isnull().sum(axis=1)
sns.lineplot(data=df_time, x='date', y='missing_count', marker='o', ax=axes[1, 0])
plt.xticks(rotation=45)

# 4. Co-occurrence of missing values
# Create correlation matrix of missing values
axes[1, 1].set_title('Missing Values Correlation')
missing_corr = df.isnull().corr()
mask = np.triu(np.ones_like(missing_corr, dtype=bool)) # upper triangle mask
sns.heatmap(missing_corr, mask=mask, annot=True, square=True, ax=axes[1, 1],cmap='viridis')
plt.tight_layout()
plt.show()

Output:

Advanced Heatmap: Matrix Plot for Missing Patterns

For even more detailed analysis, let’s create a matrix plot that shows the exact missing patterns:

def create_advanced_missing_heatmap(dataframe, figsize=(14, 10)):
    """
    Create an advanced heatmap for missing value analysis
    """
    # Prepare the data
    missing_data = dataframe.isnull()
    
    # Create figure with custom layout
    fig = plt.figure(figsize=figsize)
    gs = fig.add_gridspec(3, 3, height_ratios=[0.8, 2, 0.3], 
                         width_ratios=[2, 0.3, 0.8])
    
    # Main heatmap
    ax_main = fig.add_subplot(gs[1, 0])
    
    # Create custom colormap
    colors = ['#2E86AB', '#A23B72']  # Blue for present, Red for missing
    n_bins = 2
    cmap = sns.blend_palette(colors, n_colors=n_bins, as_cmap=True)
    
    sns.heatmap(missing_data.T, 
                cmap=cmap,
                cbar=False,
                ax=ax_main,
                xticklabels=False)
    
    ax_main.set_ylabel('Columns', fontsize=12)
    ax_main.set_xlabel('Row Index', fontsize=12)
    
    # Top plot: missing count per row
    ax_top = fig.add_subplot(gs[0, 0], sharex=ax_main)
    missing_per_row = missing_data.sum(axis=1)
    ax_top.bar(range(len(missing_per_row)), missing_per_row, 
               color='#A23B72', alpha=0.7)
    ax_top.set_ylabel('Missing\nCount', fontsize=10)
    ax_top.set_title('Missing Values Analysis Dashboard', fontsize=16, pad=20)
    
    # Right plot: missing count per column
    ax_right = fig.add_subplot(gs[1, 1], sharey=ax_main)
    missing_per_col = missing_data.sum(axis=0)
    ax_right.barh(range(len(missing_per_col)), missing_per_col, 
                  color='#A23B72', alpha=0.7)
    ax_right.set_xlabel('Missing\nCount', fontsize=10)
    ax_right.set_yticks(range(len(missing_per_col)))
    ax_right.set_yticklabels(missing_data.columns)
    
    # Summary statistics
    ax_summary = fig.add_subplot(gs[2, :])
    ax_summary.axis('off')
    
    total_cells = dataframe.shape[0] * dataframe.shape[1]
    total_missing = missing_data.sum().sum()
    missing_percentage = (total_missing / total_cells) * 100
    
    summary_text = f"""
    Dataset Overview:
    • Total cells: {total_cells:,}
    • Missing cells: {total_missing:,}
    • Missing percentage: {missing_percentage:.2f}%
    • Columns with missing data: {(missing_data.sum() > 0).sum()} out of {len(dataframe.columns)}
    • Rows with missing data: {(missing_data.sum(axis=1) > 0).sum()} out of {len(dataframe)}
    """
    
    ax_summary.text(0.1, 0.5, summary_text, transform=ax_summary.transAxes,
                   fontsize=11, verticalalignment='center',
                   bbox=dict(boxstyle="round,pad=0.3", facecolor="lightgray", alpha=0.5))
    
    plt.tight_layout()
    return fig

# Create the advanced heatmap
fig = create_advanced_missing_heatmap(df)
plt.show()

Output:

Practical Tips for Interpretation

When analyzing your heatmaps, look for these patterns:

Get Prathik C’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Vertical Patterns: Missing values in the same rows might indicate:

Data collection issues on specific dates
Systematic problems during certain periods
Batch processing failures

Horizontal Patterns: Missing values in the same columns might indicate:

Sensor malfunctions
Optional survey questions
Data integration issues

Block Patterns: Rectangular areas of missing data might indicate:

System maintenance periods
Seasonal data collection issues
Planned data collection gaps

Heatmaps transform the abstract concept of missing data into clear, visual patterns that can guide your data cleaning and inferences. By implementing heatmaps, you can:

Quickly identify problematic areas in your dataset
Make informed decisions about imputation strategies
Communicate data quality issues effectively to stakeholders
Spot systematic problems that might otherwise go unnoticed

The patterns you discover in these heatmaps should directly inform your data preprocessing strategy and help you build more robust, reliable models.

Start incorporating these techniques into your data analysis workflow, and you’ll find that understanding and addressing missing data becomes much more intuitive and therefore, more effective. Until next time:)