NumPy Part 6: Data Preparation for Machine Learning

In the previous parts of this NumPy series, we learned how to create and manipulate arrays, work with indexing and slicing, perform mathematical and statistical operations, use axes and broadcasting, and generate random data.

Now we’re going to take the final step:

How does NumPy actually help us prepare data for Machine Learning?

Before a Machine Learning model can learn from data, the data usually needs to be cleaned, transformed, and organized.

In this final part, we’ll learn how NumPy can help us with:

• Missing values (NaN)
• Detecting missing values
• Replacing missing values
• Standardization
• Normalization
• Features and targets (X and y)
• Preparing numerical features
• One-hot encoding concepts
• Train/test split concepts
• An end-to-end ML-style data preparation example
• Common mistakes
• The final NumPy → ML connection

1. What Is Data Preparation?

Real-world data is rarely perfect.

A dataset might contain:

• Missing values
• Different numerical scales
• Text/categorical values
• Outliers
• Incorrect data types
• Features with very different ranges

For example:

import numpy as np

data = np.array([
    [25, 50000],
    [30, 60000],
    [35, np.nan],
    [40, 90000],
    [45, 100000]
])

Here:

• Column 1 → Age
• Column 2 → Salary
• One salary value is missing.

Before giving this data to a Machine Learning model, we need to prepare it.

2. Understanding NaN

NaN stands for:

Not a Number

In NumPy, we represent it as:

np.nan

Example:

salary = np.array([50000, 60000, np.nan, 80000, 90000])

print(salary)

Output:

[50000. 60000.    nan 80000. 90000.]

NaN is commonly used to represent missing numerical data.

3. Detecting Missing Values with np.isnan()

NumPy provides:

np.isnan()

to detect NaN values.

Example:

salary = np.array([50000, 60000, np.nan, 80000, 90000])

print(np.isnan(salary))

Output:

[False False  True False False]

True means the value is NaN.

So:

50000 → False
60000 → False
NaN   → True
80000 → False
90000 → False

We can also count missing values:

np.sum(np.isnan(salary))

Output:

1

This tells us that there is one missing value.

4. Missing Values in a 2D Dataset

Consider:

data = np.array([
    [25, 50000],
    [30, np.nan],
    [35, 70000],
    [40, np.nan]
])

We can detect all missing values:

np.isnan(data)

Output:

[[False False]
 [False  True]
 [False False]
 [False  True]]

To count all missing values:

np.sum(np.isnan(data))

Output:

2

We can also count missing values column-wise:

np.sum(np.isnan(data), axis=0)

Output:

[0 2]

This means:

Age     → 0 missing values
Salary  → 2 missing values

This is where our previous understanding of axis becomes useful.

5. Why Normal mean() Doesn't Work with NaN

Suppose:

salary = np.array([50000, 60000, np.nan, 80000, 90000])

If we calculate:

np.mean(salary)

we get:

nan

Why?

Because the normal NumPy aggregation functions don’t automatically ignore NaN values.

Instead, NumPy provides special functions for working with missing values.

6. np.nanmean()

np.nanmean() calculates the mean while ignoring NaN values.

Example:

salary = np.array([50000, 60000, np.nan, 80000, 90000])

print(np.nanmean(salary))

The calculation is effectively:

(50000 + 60000 + 80000 + 90000) / 4

Result:

70000

The missing value is ignored.

7. np.nanmedian()

Similarly, NumPy provides:

np.nanmedian()

which calculates the median while ignoring NaN values.

Example:

salary = np.array([50000, 60000, np.nan, 80000, 90000])

print(np.nanmedian(salary))

Output:

70000

The NaN is ignored when calculating the median.

8. Mean vs Median for Missing Values

When replacing missing values, we often have to decide whether to use the mean or median.

Mean

np.nanmean(data)

Useful when the data doesn’t contain significant outliers.

Median

np.nanmedian(data)

Often more robust when the data contains outliers.

For example:

Salaries:

50000
55000
60000
65000
1000000

The extremely high salary can pull the mean upward.

The median is less affected by that outlier.

Therefore:

Mean is sensitive to outliers, while median is generally more robust.

9. Replacing Missing Values

Once we identify missing values, we may want to replace them.

For example:

salary = np.array([50000, 60000, np.nan, 80000, 90000])

We can calculate the mean:

mean_salary = np.nanmean(salary)

Then replace the missing values:

salary[np.isnan(salary)] = mean_salary

Now:

print(salary)

Output:

[50000. 60000. 70000. 80000. 90000.]

This technique is called imputation.

10. What Is Imputation?

Imputation means:

Replacing missing values with a reasonable value.

Common approaches include:

• Mean
• Median
• Mode
• A fixed value
• Forward/backward filling
• Model-based imputation

With NumPy, a simple numerical example is:

data[np.isnan(data)] = np.nanmean(data)

However, in real Machine Learning projects, we need to be careful.

We should generally calculate imputation values from the training data, not from the entire dataset.

This helps avoid data leakage.

We’ll discuss this concept later.

11. Feature vs Target

One of the most important concepts in Machine Learning is understanding:

Features (X)

and

Target (y)

Suppose we have:

Age    Experience    Salary
25     2             50000
30     5             60000
35     8             80000
40     10            90000

If we’re trying to predict salary:

Age + Experience → Salary

Then:

X = features
y = target

We could create:

X = np.array([
    [25, 2],
    [30, 5],
    [35, 8],
    [40, 10]
])

y = np.array([
    50000,
    60000,
    80000,
    90000
])

Here:

X → input data
y → value we want to predict

12. Shape of X and y

This is extremely important in Machine Learning.

For our example:

X.shape

gives:

(4, 2)

Meaning:

4 samples
2 features

And:

y.shape

gives:

(4,)

Meaning:

4 target values

A useful mental model is:

X
↓
Rows    → Samples
Columns → Features

So:

X.shape = (number_of_samples, number_of_features)

13. Why Feature Scaling Is Important

Consider two features:

Age       → 20–60
Salary    → 30,000–150,000

The numerical ranges are very different.

Some Machine Learning algorithms can be affected by this difference in scale.

Therefore, we often transform features so that they are on comparable scales.

Two common techniques are:

1. Standardization
2. Normalization

14. Standardization

Standardization transforms data so that it is centered around:

Mean = 0

and has approximately:

Standard deviation = 1

The formula is:

z = (x - mean) / standard_deviation

For example:

data = np.array([10, 20, 30, 40, 50])
mean = np.mean(data)
std = np.std(data)
standardized = (data - mean) / std
print(standardized)

The resulting values are centered around zero.

For example, values might look approximately like:

[-1.41, -0.71, 0, 0.71, 1.41]

The important idea is:

Standardization tells us how far a value is from the mean in terms of standard deviations.

15. Why Standardization Helps

Suppose:

Age:
25
30
35
40
45

and:

Salary:
50000
60000
70000
80000
90000

After standardization, both features are represented in terms of their distance from their respective means.

This prevents a feature’s large numerical units from automatically dominating simply because its raw numbers are larger.

16. Normalization

Normalization is another type of scaling.

One common form is Min-Max normalization.

The formula is:

(x - min) / (max - min)

This transforms values to a range between:

0 and 1

Example:

data = np.array([10, 20, 30, 40, 50])

normalized = (
    (data - np.min(data))
    / (np.max(data) - np.min(data))
)
print(normalized)

Output:

[0.   0.25 0.5  0.75 1.  ]

So:

10 → 0
20 → 0.25
30 → 0.5
40 → 0.75
50 → 1

17. Standardization vs Normalization

These two concepts are often confused.

Standardization

Typically:

Mean ≈ 0
Standard deviation ≈ 1

Formula:

(x - mean) / std

Min-Max Normalization

Typically:

Minimum → 0
Maximum → 1

Formula:

(x - min) / (max - min)

Quick comparison

Technique — — — — — — — — Typical result

Get Siddhartha Gaurav’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Standardization — — — — — — Mean ≈ 0, STD ≈ 1

Min-Max Normalization — — — Values between 0 and 1

Neither is universally “better.” The appropriate choice depends on the Machine Learning algorithm and data.

18. Preparing Numerical Features

Suppose we have:

X = np.array([
    [25, 50000],
    [30, 60000],
    [35, 70000],
    [40, 80000],
    [45, 90000]
])

We can calculate feature-wise mean:

mean = np.mean(X, axis=0)

Output:

[35. 70000.]

Notice what axis=0 does.

It calculates the mean of each column/feature.

Then:

std = np.std(X, axis=0)

gives the standard deviation of each feature.

We can standardize the entire feature matrix:

X_scaled = (X - mean) / std

Here, broadcasting allows:

X shape       → (5, 2)
mean shape    → (2,)
std shape     → (2,)

The (2,) arrays are applied feature-wise across all rows.

This is a perfect example of how the concepts we learned in Part 4 — axis and broadcasting — are directly useful for Machine Learning.

19. Categorical Data

Not every feature is numerical.

Consider:

City
----
Delhi
Mumbai
Bangalore
Delhi

Machine Learning models generally cannot directly work with strings such as:

"Delhi"
"Mumbai"
"Bangalore"

We need to convert categorical values into numerical representations.

One common technique is:

One-Hot Encoding

20. One-Hot Encoding Concept

Suppose we have:

Color
-----
Red
Green
Blue

We can represent them as:

Red    → [1, 0, 0]
Green  → [0, 1, 0]
Blue   → [0, 0, 1]

Each category gets its own column.

Example:

Color    Red    Green    Blue
Red       1       0        0
Green     0       1        0
Blue      0       0        1

This is called one-hot encoding.

NumPy can help construct numerical arrays, but in real ML workflows, one-hot encoding is usually handled using tools such as encoders from ML libraries.

The important thing for now is understanding the concept.

21. Train/Test Split

Suppose we have:

1000 samples

We shouldn’t train and evaluate our model using exactly the same data.

Instead, we generally split the dataset into:

Training data
Testing data

For example:

80% → Training
20% → Testing

The training data is used to learn patterns.

The testing data is used to evaluate how well the model performs on unseen data.

Conceptually:

Original Dataset
       |
       ↓
 ┌───────────────┐
 │ Train / Test  │
 └───────────────┘
       |
       ├── 80% → Training
       |
       └── 20% → Testing

In practical ML projects, libraries such as scikit-learn provide utilities for performing this split.

22. Why Data Leakage Is Dangerous

Imagine we calculate the average salary using:

Training data + Test data

and then use that average to fill missing values in the training data.

The training process has indirectly received information from the test set.

That’s called data leakage.

The general rule is:

Learn preprocessing parameters from the training data, then apply them to both training and test data.

For example:

Training data
      ↓
Calculate mean/std
      ↓
Transform training data
      ↓
Apply same mean/std to test data

Not:

Training + Test
      ↓
Calculate mean/std

This distinction becomes very important in real Machine Learning projects.

23. End-to-End ML-Style Data Preparation

Let’s combine everything we’ve learned.

Suppose we have:

data = np.array([
    [25, 50000],
    [30, 60000],
    [35, np.nan],
    [40, 90000],
    [45, 100000]
])

The columns represent:

Age
Salary

Step 1 — Detect missing values

np.isnan(data)

We can count them:

np.sum(np.isnan(data), axis=0)

Step 2 — Calculate column-wise means

mean = np.nanmean(data, axis=0)

The NaN value is ignored.

Step 3 — Replace missing values

We can replace NaN values using the corresponding feature mean.

Conceptually:

for column in range(data.shape[1]):
    column_mean = np.nanmean(data[:, column])
    data[np.isnan(data[:, column]), column] = column_mean

Now our dataset no longer contains missing values.

Step 4 — Separate features and target

Suppose salary is our target.

Then:

X = data[:, 0:1]
y = data[:, 1]

Now:

X → Age
y → Salary

Step 5 — Standardize the features

Calculate:

mean = np.mean(X, axis=0)
std = np.std(X, axis=0)

Then:

X_scaled = (X - mean) / std

Now the feature has been standardized.

Step 6 — Split the data

Conceptually:

X, y
 ↓
Train/Test Split
 ↓
X_train, X_test
y_train, y_test

The training data is used to train the model.

The test data is used for final evaluation.

24. Common Mistakes

Mistake 1 — Treating NaN as zero

A missing value is not necessarily zero.

np.nan

doesn’t mean:

0

They represent completely different things.

Mistake 2 — Using mean() when NaN values exist

This:

np.mean(data)

may return:

nan

Use appropriate NaN-aware functions when required:

np.nanmean(data)
np.nanmedian(data)

Mistake 3 — Confusing Standardization with Normalization

Remember:

Standardization
→ mean ≈ 0
→ std ≈ 1

while:

Min-Max Normalization
→ usually values between 0 and 1

Mistake 4 — Using the wrong axis

For a dataset:

rows    → samples
columns → features

If we want the mean of each feature:

np.mean(X, axis=0)

If we want the mean of each sample:

np.mean(X, axis=1)

Mistake 5 — Calculating preprocessing statistics from the entire dataset

Don’t calculate your scaling or imputation statistics using both training and test data.

Instead:

Training data
     ↓
Learn parameters
     ↓
Transform train + test

This prevents data leakage.

Mistake 6 — Thinking NumPy does everything in an ML pipeline

NumPy is extremely important, but it isn’t the entire Machine Learning ecosystem.

A typical ML workflow might use:

NumPy
   ↓
Pandas
   ↓
Scikit-learn
   ↓
Machine Learning Model

Each tool has a different role.

25. NumPy → Machine Learning Connection

Now let’s connect everything we’ve learned throughout this series.

Part 1 — Fundamentals

We learned:

Arrays
Shape
Dimensions
Data types

These are the basic building blocks of ML data.

Part 2 — Indexing & Manipulation

We learned:

Indexing
Slicing
Reshape
Transpose
Concatenate
Split

These allow us to select and restructure data.

Part 3 — Mathematical & Statistical Operations

We learned:

Mean
Median
Variance
Standard deviation
Percentiles
Min/Max

These help us understand and transform datasets.

Part 4 — Axis & Broadcasting

We learned:

axis=0
axis=1
broadcasting

These become extremely useful when performing feature-wise calculations.

For example:

mean = np.mean(X, axis=0)
X_scaled = (X - mean) / std

Part 5 — Random Data

We learned:

randint()
rand()
random()
randn()
normal()
uniform()
choice()
shuffle()
seed()

Randomness is fundamental in Machine Learning — for example, generating synthetic data, sampling, shuffling datasets, and initializing algorithms.

Part 6 — Data Preparation

Now we learned:

Missing values
NaN handling
Imputation
Standardization
Normalization
Features
Targets
Categorical encoding concepts
Train/test split
Data leakage

This is where NumPy starts looking much more like Machine Learning data preparation.

26. The Bigger Picture

A simplified Machine Learning workflow looks like:

Raw Data
   ↓
Data Cleaning
   ↓
Missing Value Handling
   ↓
Feature Preparation
   ↓
Encoding
   ↓
Scaling
   ↓
Train/Test Split
   ↓
Machine Learning Model
   ↓
Prediction
   ↓
Evaluation

NumPy plays an important role in many of these steps.

But in real-world projects, we’ll usually combine it with other libraries.

27. Final Takeaway

After completing this six-part NumPy series, you should now understand much more than just NumPy syntax.

You should understand how arrays represent Machine Learning data.

The most important mental model is:

Rows    → Samples
Columns → Features
X       → Input features
y       → Target

And the preprocessing pipeline:

Raw Data
   ↓
Handle Missing Values
   ↓
Prepare Features
   ↓
Encode Categories
   ↓
Scale Features
   ↓
Split Train/Test
   ↓
ML Model

The goal of learning NumPy wasn’t simply to memorize functions.

It was to understand how numerical data is represented, manipulated, transformed, and prepared for Machine Learning.

🎯 Final NumPy Checklist

Before moving on, you should be comfortable with:

• Creating NumPy arrays
• shape, ndim, size, dtype
• Indexing and slicing
• Boolean indexing
• np.where()
• reshape()
• flatten() / ravel()
• Transpose
• Concatenation
• Splitting arrays
• Copy vs View
• Element-wise operations
• Vectorization
• Universal functions
• Mean, median, variance, standard deviation
• Percentiles
• argmin() / argmax()
• unique()
• Axis
• Broadcasting
• Random number generation
• randint()
• rand()
• random()
• randn()
• normal()
• uniform()
• choice()
• shuffle()
• Missing values / NaN
• np.isnan()
• np.nanmean()
• np.nanmedian()
• Imputation
• Standardization
• Normalization
• Features and targets
• One-hot encoding concept
• Train/test split concept
• Data leakage

🚀 What’s Next?

With NumPy complete, the natural next step is to continue with Pandas.

And this is especially useful because Pandas builds directly on many of the concepts we’ve just learned.

We’ll move from:

NumPy
 ↓
Numerical arrays

to:

Pandas
 ↓
Real-world tabular datasets
 ↓
Data cleaning
 ↓
Data analysis
 ↓
Feature preparation
 ↓
Machine Learning

So don’t think of NumPy as an isolated topic.

NumPy is the numerical foundation underneath a large part of the Python Data Science and Machine Learning ecosystem.

🎓 NumPy Series Complete

Part 1 → Fundamentals

Part 2 → Indexing, Slicing & Array Manipulation

Part 3 → Mathematical & Statistical Operations

Part 4 → Axis, Aggregation & Broadcasting

Part 5 → Random Numbers & Random Data

Part 6 → Data Preparation for Machine Learning

Six parts. One foundation. NumPy → Data Science → Machine Learning.