Numerical Data Preprocessing

Bhargav

8 min read3 days ago

Numerical data preprocessing is the process of converting raw numerical data into a suitable format so that machine can learn effectively and produce better predictions.

Numerical Data Preprocessing Techniques :

Handling Missing Values:

Missing value handling is a data preprocessing technique used to identify and treat unavailable values in a dataset before training a Machine.

Types of Missing Values

a. MCAR — Missing Completely At Random :

Missing values have no relationship with any feature in the dataset.
example : A computer randomly fails while recording some customer ages.

b. MAR — Missing At Random :

Missingness depends on another observed variable.
example :
Younger customers are less likely to provide their income details.
Here, missing income values depend on the observed Age column.

c. MNAR — Missing Not At Random :

Missingness depends on the missing value itself or an unobserved factor.
example :
High-income people may intentionally avoid providing their salary details because their salary is high.

Identifying Missing Values

In python

Techniques to Handle Missing Values

Deletion Technique : Deletion means removing rows or columns containing missing values.

a. Row Deletion

Remove entire rows containing at least one missing value.

df.dropna(axis=0, inplace=True)

When to use?

Very few rows contain missing values.
Missingness is reasonably random.
Removing rows will not introduce significant bias.

2. Imputation Techniques :

Imputation means replacing missing values with estimated or meaningful values.

There are several methods are available.

A. Mean Imputation: Replace missing values with the average of available values.

Python Implementation

Advantages :

Simple to implement: Easy to calculate and replace missing values.
Preserves dataset size: We don’t need to delete rows containing missing values.
Fast execution: Requires very little computational time.
Useful for numerical data: Works with continuous numerical features.
Maintains completeness: Helps create a dataset without missing values.

Limitations :

Sensitive to outliers: Extreme values can significantly affect the mean.
Reduces variance: Replacing missing values with the same mean makes the data less spread out.
Changes distribution: The original distribution of the feature may be distorted.
Affects correlation: Relationships between variables may become weaker or biased.
Not suitable for skewed data: Mean may not represent the typical value when data is highly skewed.
Ignores relationships: It does not consider relationships between different features.
Can introduce bias: If data is not missing completely at random (MCAR), mean imputation can produce biased estimates.

When should we use mean imputation?

When the numerical feature has a nearly normal distribution.
when there are low missing values.
No outliers.
Symmentrical Data.

B. Median Imputation : Replace missing values with the middle value after sorting.

Python Implementation

Advantages :

Outliers cannot effect the median.
Suitable for skewed data.
Preserves dataset size.

Limitations :

Reduces variance: Replacing missing values with the same median reduces data variability.
Changes distribution
Affects correlation: May weaken or bias relationships between features.
Ignores feature relationships: Does not consider relationships between different columns.
Not suitable for categorical data: Median is generally used for numerical data, not nominal categories.

When should we use median imputation?

When numerical data contains outliers.
When the distribution is highly skewed.
When the data is not normally distributed.
When missing values are relatively few.
When the median represents the central tendency better than the mean.

C. Mode Imputation : Replace missing values with the most frequently occurring value.

Python Implementation :

Advantages :

Simple to implement.
Suitable for categorical or dscrete numerical data.
Preserves dataset size.
Maintains valid categories: Replaces missing values with an existing. category.
Fast execution.

Limitations :

Creates bias: The most frequent category gets more representation.
Changes distribution: Increases the frequency of the mode category.
Ignores feature relationships: Does not consider relationships between other columns.
Not suitable for high-cardinality features: When many unique categories exist, the mode may not represent the data effectively.

When should we use mode imputation?

When categorical columns contain missing values.
When one category occurs significantly more frequently than others.
When the number of missing values is relatively small.
When replacing missing values with an existing category is reasonable.

2. Handling Outliers

It is an important step in Numerical Data Preprocessing.

What is Outlier :

Get Bhargav’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

outlier is data point and it is an unusually high or unusually low value compared to other data points.

example :

marks = [45, 50, 52, 48, 55, 51, 49, 53, 98]

Most students scored between 45 and 55.

But one student scored 98.

Here, 98 may be an outlier.

Why we do detect outliers?

Outlier detection helps us understand whether unusual data points are affecting the patterns our ML model is trying to learn.

Outliers Handling Methods:

IQR (Interquartile Range) (popular)
Z-Score
Capping / Winsorization
Removing outliers when justified

IQR Method (Interquartile Range) — Outlier Detection :

The IQR (Interquartile Range) method is a statistical technique used to identify outliers in numerical data.

It identifies unusually small or large values by measuring the spread of the middle 50% of the dataset.

Quartiles:

Before understanding IQR, first we need to understand about quartiles.

Quartiles divide sorted numerical data into four equal parts.

Q1 (First Quartile): 25% of observations are below or equal to this value.
Q2 (Second Quartile): 50% of observations are below or equal to this value. This is the median.
Q3 (Third Quartile): 75% of observations are below or equal to this value.
Q4 (Fourth Quartile): 100% of observations are below or equal to this value.

IQR Formula : The Interquartile Range measures the distance between the third and first quartiles.

Example :

X = [10, 12, 15, 18, 20, 22, 25, 100]

step 1 : Sort the data.

10, 12, 15, 18, 20, 22, 25, 100

step 2 : Find Q2 (Median).

step 3 : Find Q1.

step 4 : Find Q3.

step 5 : Calculate IQR.

Lower Bound :

Calculating the lower and upper bound :

These are important steps in the outlier detection.

Upper Bound :

Identifying the Outlier :

The decision rule is :

Press enter or click to view image in full size

Python Implementation :

3. Feature Scaling

Feature Scaling is a data preprocessing technique used to transform numerical features into a similar scale or range.

Why Feature Scaling?

Feature Scaling brings numerical features to a comparable scale so that machine learning algorithms can learn more effectively.

Types :

Standard Scaler (Standardization)
Min-Max Scaler (Normalization)
Robust Scaler

Standard Scaler (Standardization) :

StandardScaler is a feature scaling technique in Machine Learning that standardizes numerical features by removing the mean and scaling them to unit variance.

It works well when the data is free from extreme outliers.
It transforms features so that their mean becomes 0 and standard deviation becomes 1.
It helps ML algorithms perform better when features have different scales.

Python Implementation :

Functions :

fit() – Calculates the mean and standard deviation.
transform() – Applies the scaling.
fit_transform() – Performs both operations.

Min-Max Scaler (Normalization) :

MinMaxScaler is a feature scaling technique in Machine Learning that transforms numerical features into a fixed range, usually between 0 and 1.

It scales numerical features into a specified range, usually 0 to 1.
It is sensitive to outliers because extreme minimum or maximum values affect the scaling.
It is useful when we need all features within a fixed range.

Python Implementation :

Functions :

fit() – Finds the minimum and maximum values.
transform() – Scales the data.
fit_transform() – Performs both operations.

Robust Scaler :

RobustScaler is a feature scaling technique in Machine Learning that scales numerical features using the median and Interquartile Range (IQR) instead of the mean and standard deviation.

It is especially useful when the dataset contains outliers.
It uses the median and IQR to scale numerical features.
It is less affected by extreme values compared to StandardScaler and MinMaxScaler.