Missing Values ??

Whenever missing values come up, the first instinct of most naive data scientists is: “Hey, let’s just average out the values, or fill in the median or the mode, or maybe just remove the rows.”

But have you ever questioned yourself which of these will you choose, and more importantly, how will you choose?

Before touching a single missing cell, you need to understand why the data is missing. There are three types of missingness, and the right treatment depends entirely on which type you’re dealing with:

1. MCAR : Missing Completely At Random
2. MAR : Missing At Random
3. NMAR : Not Missing At Random

Let’s decode each one.

1. Missing Completely At Random (MCAR)

“The probability of being missing is the same for all observations.”

Ya ya, I know a fancy statement from a really good book. But what does it actually mean? Let’s decode it.

1.1 Definition

The probability of a value being missing is completely unrelated to anything not to other columns, and not to the missing value itself. It’s pure accident. Random noise.

1.2 Real-World Example

A hospital collecting patient blood pressure readings:

Patient | Age | Income | Blood Pressure
--------|-----|--------|----------------
1       | 45  | 50K    | 120  ✓
2       | 52  | 75K    | [MISSING]  ← random recording error
3       | 38  | 60K    | 118  ✓
4       | 61  | 90K    | [MISSING]  ← random recording error

Here the data went missing maybe because the nurse simply forgot to record it. The forgetting was completely random not related to whether the patient’s BP was high or low, and not related to age, income, or any other factor.

The test for MCAR: if you compare patients with missing BP against patients with recorded BP, the two groups look statistically identical on every other variable. No pattern anywhere.

1.3 How to Handle MCAR

This is the best-case scenario, because deleting or simply imputing doesn’t introduce bias. Your options:

a) Delete the rows (listwise deletion) : Safe for MCAR, but only recommended if the missing rows are a small fraction of the data. Otherwise you throw away a lot of good information in the other columns.

b) Mean / Median / Mode imputation : Simple and unbiased for the average under MCAR. One caveat worth knowing, it artificially shrinks the variance of the column (every imputed row gets the exact same value), so don’t rely on it if variance or correlations matter downstream.

c) Forward / Backward fill : Best for time-series data, where the previous (or next) value is a sensible stand-in:

Date    | Sales
--------|-------
Jan 1   | 100
Jan 2   | [MISS]  ← forward fill: 100
Jan 3   | 120
Jan 4   | [MISS]  ← forward fill: 120
Jan 5   | 150

2. Missing At Random (MAR)

“The probability of being missing is the same within groups defined by the observed data.”

2.1 Definition

The missingness depends on other observed variables in your dataset but not on the missing value itself.

Careful: “at random” is misleading here. MAR data is not randomly missing overall there is a visible pattern. The pattern just happens to be fully explained by columns you can see.

2.2 Real-World Example: Hospital Visits

Patient | Age | Health Status   | Hospital Visits
--------|-----|-----------------|------------------
1       | 30  | Healthy         | 2      ✓
2       | 72  | Chronic illness | [MISSING]
3       | 45  | Healthy         | 1      ✓
4       | 68  | Chronic illness | [MISSING]
5       | 25  | Healthy         | 3      ✓
6       | 75  | Chronic illness | [MISSING]
7       | 40  | Healthy         | 2      ✓
8       | 70  | Chronic illness | [MISSING]
9       | 35  | Chronic illness | 4      ✓
10      | 28  | Chronic illness | 5      ✓

What you see at first glance: “All the elderly, chronically ill patients have missing data!” looks like a strong pattern , maybe even bias.

The story behind it: elderly patients with chronic illness struggled to keep their visit records. So the missingness is driven by Age + Health Status two columns we can fully observe.

2.3 “Once we account for these variables, the missingness becomes random” Decoded

This is the statement that confuses everyone, so let’s decode it properly.

It means: if we re-group the dataset based on the observed variables (age and health status), the missingness within each group has no further pattern.

Concrete examplem suppose our full dataset has 60 patients aged 65+ with chronic illness, and 50 of them have missing hospital visit data.

• At the group level, missingness is predictable: this group has a very high missing rate (50/60 ≈ 83%) compared to, say, young healthy patients (near 0%).
• At the individual level, missingness is random: which specific 50 out of the 60 are missing? There’s no further bifurcation. Among these 60 patients:
• Males and females are missing at similar rates
• Rich and poor are missing at similar rates
• No other observed variable predicts who’s missing

We’ve already accounted for Age and Health Status. Beyond that, it’s a coin flip. That is what “random within groups” means.

(Contrast this with NMAR, coming up next: there, even within the group, the missing ones would specifically be, say, the patients with the most visits the value itself drives the missingness.)

2.4 How to Handle MAR

MAR requires smarter methods, because plain deletion or mean-filling would bias your results (you’d systematically lose/flatten the elderly chronic patients). The good news: since the missingness is explained by observed columns, we can use those columns to impute intelligently.

Our working dataset:

Patient | Age | Health Status | Income | Hospital Visits
--------|-----|---------------|--------|------------------
1       | 35  | Healthy       | 50K    | 2
2       | 68  | Chronic       | 40K    | ???
3       | 42  | Diabetes      | 55K    | 3
4       | 75  | Chronic       | 35K    | ???
5       | 28  | Healthy       | 60K    | 1

You can’t just ignore Patients 2 and 4. You need smart imputation not guessing.

2.4.1 MICE (Multiple Imputation by Chained Equations)

The idea: think of MICE as variables having a conversation to fill in the blanks.

Variables are related. If I know your age, health status, and income, I can predict your hospital visits. If I know your hospital visits, age, and income, I can predict your health status. MICE leverages these relationships through multiple iterations.

Get Swayam’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Step 1 : Initial placeholder. MICE starts with a naive guess, e.g. the mean of the observed values:

Mean Hospital Visits = (2 + 3 + 1) / 3 = 2 (placeholder)
Patient 2 → 2 (placeholder)
Patient 4 → 2 (placeholder)

Step 2 : Build a predictive model. MICE fits a regression on the complete cases, learning something like:

Hospital Visits = 0.49 + 0.05(Age) + 2.02(if Chronic)
                  + 0.81(if Diabetes) − 0.00002(Income)

(The coefficients here are illustrative , in reality they’re estimated from your data by minimizing prediction error, not chosen by hand.)

Step 3 : Predict the missing values with the learned model:

Patient 2: 0.49 + 0.05(68) + 2.02(1) − 0.00002(40000) ≈ 5.1 visits
Patient 4: 0.49 + 0.05(75) + 2.02(1) − 0.00002(35000) ≈ 5.6 visits

Notice how much more sensible this is than the naive placeholder of 2 the model recognized that elderly + chronic means more visits.

Round 4+ : Iterate until stable. MICE cycles through the variables (typically ~10 rounds). If other columns also had missing values, each would get its own model in turn, and each round’s improved estimates feed the next round’s models. The imputed values (and model coefficients) shift slightly each pass, then converge changes become tiny and MICE stops.

Why MICE works well:

• Captures relationships : understands that older, chronically ill patients visit more
• Iterative refinement : each pass improves on the last.
• Realistic results : values come from actual patterns in your data.
• Multi-column : handles missing data across several columns simultaneously.

2.4.2 KNN Imputation

“If you’re similar to me, we probably have similar values for the missing attribute.”

Step 1 : Define “similar”. For Patient 2 (Age 68, Income 40K), compute the distance to every other patient, e.g. Euclidean distance:

Distance = √[(Age difference)² + (Income difference)²]

To Patient 1: √[(68−35)² + (40K−50K)²] ≈ 10,000
To Patient 3: √[(68−42)² + (40K−55K)²] ≈ 15,000
To Patient 4: √[(68−75)² + (40K−35K)²] ≈  5,000  ← closest!
To Patient 5: √[(68−28)² + (40K−60K)²] ≈ 20,000

Side note you can already spot : income (in tens of thousands) completely dominates age (in tens) in this distance. This is why you should scale/normalize features before KNN imputation otherwise “similarity” is decided almost entirely by the largest-unit column.

Step 2 : Pick the K nearest neighbors that actually have the value. Patient 4 is the closest, but its Hospital Visits is also missing so it can’t donate a value. With K = 3, the nearest usable neighbors are:

Patient 1 → 2 visits
Patient 3 → 3 visits
Patient 5 → 1 visit

Step 3 : Average their values:

Patient 2's imputed Hospital Visits = (2 + 3 + 1) / 3 = 2.0

Why KNN works:

• Simple and intuitive : similar people, similar values
• No model assumptions : no linearity required
• Fast : one pass through the data
• Local : captures neighborhood patterns

One honest caveat : in this tiny example, KNN gives 2.0 visits while MICE gave ~5.1 for the same patient. KNN can underestimate when the “nearest” complete-case neighbors happen to be younger/healthier people it only knows distance, not relationships. MICE tends to do better when variables have strong, learnable relationships (which is exactly the MAR situation).

3. Not Missing At Random (NMAR)

“If the data is not MCAR and not MAR.”

3.1 Definition

Data is missing because of the value itself. The fact that the value is missing tells you something about what it would have been usually that it’s extreme or sensitive.

3.2 Simple Analogy

Ask people: “How much do you owe on your credit card?” People with huge debts are the most likely to refuse to answer. The missingness is directly caused by the value being high.

3.3 Real-World Example: Income Survey

Person | Age | Education   | Income
-------|-----|-------------|------------------
1      | 25  | High School | 30K   ✓
2      | 45  | Bachelor    | [MISSING]  ← probably very high
3      | 35  | High School | 35K   ✓
4      | 55  | Master's    | [MISSING]  ← probably very high
5      | 28  | Bachelor    | 45K   ✓

Why it’s NMAR: high earners refuse to answer. The missingness is driven by the income value itself. Even after accounting for education (the observed variable), the missing ones within each education group would still specifically be the highest earners that’s what separates this from MAR.

3.4 How to Handle NMAR

Honestly, this is the hardest type to handle. You can’t just remove the rows you’d lose the most informative segment of your data (the high earners would silently vanish, and your “average income” would be badly underestimated). And filling with mean/median/mode is worse: it actively injects wrong values and destroys the one signal you do have the missingness pattern itself.

So instead of trying to recover the value, the strategy is to preserve the missingness as information:

Option 1: Add a missing-indicator column

Add a flag column Income_Was_Missing (1 if missing, 0 if reported), then fill the original column with a neutral placeholder.

Person | Education | Income | Income_Was_Missing
2      | Bachelor  | 0      | 1  ← flagged
4      | Master's  | 0      | 1  ← flagged

Why it works: the flag turns missingness into a feature. To be precise about what the model learns it does not magically figure out that Person 2 earns 200K. What it learns is that Income_Was_Missing = 1 is itself predictive of whatever you're modeling downstream. If people who hide income systematically behave differently (default more, spend more, churn less whatever your target is), the model captures that pattern through the flag instead of being poisoned by a fake imputed income. You've kept the signal honest instead of erasing it.

Option 2: Use models that handle missing values natively

XGBoost uses a sparsity-aware split with a learned default direction. Here’s what actually happens: at every split in every tree, XGBoost tries sending the missing-value samples down the left branch, then down the right branch, and keeps whichever direction reduces the loss more. That direction becomes the default path for missing values at that node.

So in our income example, XGBoost never estimates what the hidden incomes are. It simply learns, split by split, that “rows with missing income perform best when routed toward the branch containing high earners” because the training labels of those rows pull it there. The missingness pattern gets exploited automatically, without a single imputed number. This is exactly why XGBoost’s built-in handling is genuinely useful for NMAR: when missingness is systematic, the learned default direction carries real signal. (When data is MCAR, that direction is basically noise no harm, but no magic either.)

Random Forest offers proximity-based imputation: it first fills missing values roughly, builds a forest, then measures how often each pair of rows lands in the same leaf (the proximity matrix) and re-imputes each missing value as a proximity-weighted average of similar rows iterating a few times. It’s essentially a smarter, model-driven cousin of KNN.

One honest caveat for NMAR overall: no method fully “solves” it, because the information you need is hidden by definition. Indicator columns and native-handling models let you use the missingness signal; for serious analysis, pair them with a sensitivity analysis (test your conclusions under “missing = low” vs “missing = high” assumptions) and be transparent about the uncertainty.

Quick Decision Guide

Is the missingness unrelated to everything (pure accident)?
└─ MCAR → deletion (if few rows) / mean-median-mode / ffill-bfill

Is the missingness explained by OTHER observed columns?
└─ MAR → MICE / KNN imputation (use those columns to impute smartly).
Is the missingness driven by the missing VALUE itself?
└─ NMAR → missing-indicator flag + XGBoost/RF native handling
          + sensitivity analysis, and stay honest about uncertainty.

The next time someone says “just fill it with the mean”, you’ll know exactly which question to ask first: why is it missing?