Feature Scaling Explained: Min-Max, Robust and Standard Scalers

Boost ML model’s performance by mastering data normalization and standardization.

Photo by Tim Oun on Unsplash

⬅️ Previous: [Feature Transformations Explained]

Feature Scaling

Feature scaling is a method used in data pre-processing for machine learning and data science that involves scaling the range of variables or features of data. The goal of feature scaling is to normalize the range of independent variables or features of data within a specific range, such as between -1 and 1 or 0 and 1, or to standardize the features to have a mean of 0 and a standard deviation of 1. This process is important because features on different scales can distort the distance between data points in some algorithms and can also affect the performance of algorithms negatively.

Why do we need feature scaling?

1. Improves the performance of Distance-based algorithms like KNN.
2. Improves optimization, as gradient descent converges slowly and inaccurately on datasets with different scales.
3. Improves numerical stability in deep learning.

Which algorithms are affected if the features are not scaled?

1. Gradient Descent based algorithms like Linear Regression, Logistic Regression, Neural Network, etc.
2. SVM
3. Distance based algorithms like KNN, K-Means, etc.
4. Dimensionality Reduction Techniques like PCA, LDA, etc.
5. Regularization Techniques like Ridge Regression, Lasso Regression, etc.

Which algorithms are not affected?

1. Tree-based algorithms, because they use logical comparison. like CART, Random Forest, AdaBoost, Gradient Boosting, XGBoost, etc.
2. Naïve Bayes, because it uses probability.

Types of Feature Scaling:

1. Univariate: Scaling is done column wise.
2. Multivariate: Scaling is done row wise.

Types of Feature Scaling Techniques:

(i.) Standardization

Standardization is a combination of two steps: mean centering and scaling. The resulting data will have a mean = 0 and a standard deviation = 1, with no change in covariance (meaning the shape of the resultant will not necessarily be circular).

NOTE: The distribution of all columns remains the same if the covariance is 0; otherwise, it will change. However, distribution of single feature will not change after standardization.

Distribution has changed because covariance of all features is not 0.

No effect on outliers means outliers will remain the same, as the distribution will not change if the covariance is 0.

Advantages:

• Simple and efficient to implement.

Disadvantages:

• Does not work with algorithms that assume non-negative values, like Multinomial Naive Bayes (as probability cannot be negative) and Relu in deep learning.

from sklearn.preprocessing import RobustScaler, MinMaxScaler, StandardScaler, MaxAbsScaler, normalize
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
import numpy as np

df = pd.read_csv('https://raw.githubusercontent.com/campusx-official/100-days-of-machine-learning/main/day24-standardization/Social_Network_Ads.csv')
df = df.iloc[:,2:]
df.sample(5)


X_train, X_test, y_train, y_test = train_test_split(df.drop('Purchased', axis=1), df['Purchased'], test_size=0.3, random_state=0)

scaler = StandardScaler()

scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)

scaler.mean_ # array([3.78642857e+01, 6.98071429e+04])

X_train_scaled = pd.DataFrame(X_train_scaled, columns=X_train.columns)
X_test_scaled = pd.DataFrame(X_test_scaled, columns=X_test.columns)

np.round(X_train.describe(), 1)
np.round(X_train_scaled.describe(), 1)

fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(12, 5))
ax1.scatter(X_train['Age'], X_train['EstimatedSalary'])
ax1.set_title("Before Scaling")
ax2.scatter(X_train_scaled['Age'], X_train_scaled['EstimatedSalary'], color='red')
ax2.set_title("After Scaling")
plt.show()


fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(12, 5))
ax1.set_title('Before Scaling')
sns.kdeplot(X_train['Age'], ax=ax1)
sns.kdeplot(X_train['EstimatedSalary'], ax=ax1)
ax2.set_title('After Standard Scaling')
sns.kdeplot(X_train_scaled['Age'], ax=ax2)
sns.kdeplot(X_train_scaled['EstimatedSalary'], ax=ax2)
plt.show()

Distribution has not changed because covariance of all features is 0.

Also, the distribution has not changed because the covariance of all features is 0. But the first figure looks weird because Age is concentrated between 0 and 100, while EstimatedSalary is spread between 0 and 175,000. Thus, the relationship is not clearly visible.

fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(12, 5))
ax1.set_title('Age Distribution Before Scaling')
sns.kdeplot(X_train['Age'], ax=ax1)
ax2.set_title('Age Distribution After Standard Scaling')
sns.kdeplot(X_train_scaled['Age'], ax=ax2)
plt.show()


fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(12, 5))
ax1.set_title('Salary Distribution Before Scaling')
sns.kdeplot(X_train['EstimatedSalary'], ax=ax1)
ax2.set_title('Salary Distribution Standard Scaling')
sns.kdeplot(X_train_scaled['EstimatedSalary'], ax=ax2)
plt.show()


lr_scaled = LogisticRegression()
lr_scaled.fit(X_train_scaled, y_train)
y_pred_scaled = lr_scaled.predict(X_test_scaled)

accuracy_score(y_test, y_pred_scaled) # 0.8666666666666667

The distribution of a single feature has not changed.

(ii.) Normalization

1. Min Max Scaler

Min Max Scaler is a data pre-processing technique used in machine learning to adjust the values of numeric columns in the dataset to a common scale, without distorting differences in the ranges of values or losing information. Unlike standardization, which rescales data to have a mean of 0 and a standard deviation of 1, normalization typically rescales the data into the range between 0 and 1. This process is important for modeling algorithms that are sensitive to the scale of input features.

It maintains the distribution of data and relative distances. It is highly sensitive to outliers so don't use if there are outliers in the data.

1. Fixed Range: After applying Min-Max scaling, all feature values are transformed to lie within a predefined range, typically [0, 1]. If X is a feature, the transformed value ′X′will be in the range where the minimum value of the feature becomes 0, and the maximum value becomes 1.
2. Relative Distances Preserved: The relative distances between values in each feature are preserved. While the absolute values change to fit within the new range, the proportions relative to the minimum and maximum of the original data are maintained.
3. No Impact on Shape of the Data Distribution: Min-Max scaling does not alter the shape of the feature’s distribution. If the original data is normally distributed, skewed, or has any other distribution, the scaled data will maintain that distribution, albeit within the new range.
4. Dependency on Min and Max Values: The scaled values are highly sensitive to the minimum and maximum values in the data. If there are outliers, they will compress the rest of the data into a very small portion of the range, potentially reducing the effectiveness of the scaling.
5. Zero Values Transformation: If the original data contains zero values, they may no longer be zero after scaling unless the minimum value in the original data is 0. The transformation shifts all values proportionally between the new minimum and maximum. Thus, it is not useful where data is sparse, or where zero means the absence of something (like when a column is a count of something, etc).
6. Impact on Algorithms: Min-Max scaling can significantly impact the performance of machine learning algorithms that are sensitive to the magnitude of the input values, such as gradient descent-based algorithms, nearest neighbors, and neural networks. The scaling ensures that features contribute equally to the model’s learning process, preventing features with larger scales from dominating the learning.
7. Reversibility: The Min-Max scaling process is reversible. Given the scaled value, along with the original minimum and maximum, you can calculate the original value. This is important for interpreting model outputs in the original scale.

df = pd.read_csv('https://raw.githubusercontent.com/campusx-official/100-days-of-machine-learning/main/day25-normalization/wine_data.csv', header=None, usecols=[0,1,2])
df.columns=['Class label', 'Alcohol', 'Malic acid']
df

X_train, X_test, y_train, y_test = train_test_split(df.drop('Class label', axis=1), df['Class label'], test_size=0.3, random_state=0)

scaler = MinMaxScaler()
scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)

X_train_scaled = pd.DataFrame(X_train_scaled, columns=X_train.columns)
X_test_scaled = pd.DataFrame(X_test_scaled, columns=X_test.columns)

np.round(X_train.describe(), 1)
np.round(X_train_scaled.describe(), 1)

fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(12, 5))
# before scaling
ax1.scatter(X_train['Alcohol'], X_train['Malic acid'], c=y_train)
ax1.set_title("Before Scaling")
# after scaling
ax2.scatter(X_train_scaled['Alcohol'], X_train_scaled['Malic acid'], c=y_train)
ax2.set_title("After Scaling")
plt.show()


fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(12, 5))
# before scaling
ax1.set_title('Before Scaling')
sns.kdeplot(X_train['Alcohol'], ax=ax1)
sns.kdeplot(X_train['Malic acid'], ax=ax1)
# after scaling
ax2.set_title('After Standard Scaling')
sns.kdeplot(X_train_scaled['Alcohol'], ax=ax2)
sns.kdeplot(X_train_scaled['Malic acid'], ax=ax2)
plt.show()


fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(12, 5))
# before scaling
ax1.set_title('Alcohol Distribution Before Scaling')
sns.kdeplot(X_train['Alcohol'], ax=ax1)
# after scaling
ax2.set_title('Alcohol Distribution After Standard Scaling')
sns.kdeplot(X_train_scaled['Alcohol'], ax=ax2)
plt.show()


fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(12, 5))
# before scaling
ax1.set_title('Malic acid Distribution Before Scaling')
sns.kdeplot(X_train['Malic acid'], ax=ax1)
# after scaling
ax2.set_title('Malic acid Distribution After Standard Scaling')
sns.kdeplot(X_train_scaled['Malic acid'], ax=ax2)
plt.show()

The distribution remains the same but gets stretched or shrunk to maintain the scale.

Difference between Standardization and Min Max Normalization:

Get Mr. Vivek Kumar’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Use Standardization when:

1. Dealing with algorithms that benefit from data centered around 0 like linear regression etc.
2. Features are normally distributed.

Use Min Max Normalization when:

1. When there is a bounded scale for features like we know the maximum and minimum value for the feature like marks from 0 and 100, but cannot use in salary.
2. When data doesn’t exhibit strong normal distribution like skewed distribution.
3. When working with algorithm which don’t work with negative values like Naive Bayes etc.

2. Robust Scaler

Robust Scaler is used when outliers are very useful (like in anomaly detection, fraud detection, etc.), and we don’t want to remove the outliers because they play a very crucial role in prediction. It is robust to outlier as it reduces the impact of outliers. It is used to scale features to a specific range, typically between -1 and 1. However, unlike the standard scaler, it is resistant to outliers that can skew the data distribution. This approach minimizes the influence of extreme values (outliers) on the scaling process.

Median and IQR in place of mean and standard deviation respectively are used.

1. Median Centering: The median of each feature will be shifted to zero. Since the robust scaler subtracts the median from each data point, the center of the scaled data for any feature will be its median, aligning it at 0 in the scaled dataset.
2. IQR Scaling: The IQR (the range between the 25th and 75th percentiles) becomes the scaling reference, set to 1. This means that the difference between the first quartile (Q1) and the third quartile (Q3) in the scaled data will be 1. Consequently, data points that were at Q1 and Q3 in the original data will be at -1 and 1, respectively, in the scaled data.
3. Outlier Influence Minimization: The scaling is less sensitive to outliers compared to methods like standardization or min-max scaling. Since the median and IQR are more robust against outliers, extreme values have less influence on the scaling process, ensuring that the core data is not skewed by anomalous points.
4. Data Distribution Shape: The overall shape of the data distribution is maintained. While the scale and location of the data change, the relative positioning of data points within the distribution remains constant.
5. No Fixed Range for Scaled Data: Unlike min-max scaling, the robust scaler does not confine the data to a fixed range. Scaled values can be greater than 1 or less than -1, especially for data points that lie outside the interquartile range.
6. Impact on Learning Algorithms: Scaling data with a robust scaler can improve the performance of machine learning algorithms sensitive to outliers. By ensuring that the feature’s core data is effectively normalized, models can focus on the more representative part of the data.
7. Reversibility: The transformation is reversible, provided that you retain the original median and IQR values. Knowing these parameters allows you to reconstruct the original data values from the scaled data.

url = "https://raw.githubusercontent.com/selva86/datasets/master/BostonHousing.csv"
df = pd.read_csv(url)

# Select a single feature for demonstration, for example, 'rm' (average number of rooms per dwelling)
feature_data = df['rm'].values

scaler = RobustScaler()
feature_scaled = scaler.fit_transform(feature_data.reshape(-1, 1)).flatten()


plt.figure(figsize=(12, 6))

plt.subplot(1, 2, 1)
plt.hist(feature_data, bins=20, color='blue', alpha=0.7)
plt.title('Original Data Distribution\n(Average Number of Rooms)')
plt.xlabel('Number of Rooms')
plt.ylabel('Frequency')

plt.subplot(1, 2, 2)
plt.hist(feature_scaled, bins=20, color='green', alpha=0.7)
plt.title('Robust Scaled Data Distribution\n(Average Number of Rooms)')
plt.xlabel('Scaled Number of Rooms')
plt.ylabel('Frequency')

plt.tight_layout()
plt.show()

3. Max Absolute Scaler

Max Absolute Scaler is used when dealing with a sparse matrix where 0 has meaning, and we want to preserve that meaning, as other methods won’t guarantee that 0 will keep its meaning.

• Range: The scaled data will be within the range [-1, 1]. If the original data contains both positive and negative values, the scaled data will span this entire range. If the original data is all non-negative or non-positive, the scaled data will lie within [0, 1] or [-1, 0], respectively.
• Preservation of Zero: If a value is 0 in the original data, it remains 0 in the scaled data, as scaling does not introduce a shift.
• Outlier Sensitivity: Like min-max scaling, max absolute scaling is sensitive to outliers since extreme values will dictate the scaling factor for the entire feature.

1. Sign Preservation: The scaling maintains the sign of each value. Positive values remain positive, and negative values stay negative, ensuring that the relative relationships in terms of direction (positive or negative) are preserved.
2. Zero Values: Zero values in the data remain unchanged. This characteristic is crucial for datasets where the zero point carries specific meaning or indicates the absence of a feature.
3. Relative Distances: The relative distances between values in each feature are preserved. While the scale changes, the proportional relationships between data points remain consistent.
4. No Centering or Shifting: The data is not centered around the mean or median; only the scale changes. The center of the distribution remains the same as in the original data.
5. Dimensional Consistency: Each feature is scaled independently, ensuring that the scaling does not distort the relationships between different features in multivariate data.
6. Sparse Data Suitability: The method is particularly suitable for sparse data, as it does not alter zero entries, maintaining the sparsity pattern of the dataset.
7. Scale Interpretation: Post-scaling, the interpretation of a unit change in the data is consistent across features, facilitating comparisons and analyses that rely on the magnitude of values.
8. Reversibility: The scaling transformation is reversible. Knowing the original maximum absolute value allows you to rescale the data back to its original range, preserving the integrity of the data for inverse transformations if needed.
9. Outlier Sensitivity: The scaling process is sensitive to outliers. Extreme values can dominate the scaling factor, affecting how the rest of the data is scaled.

np.random.seed(0)
X_toy = np.random.rand(100, 2) * 100

max_abs_scaler = MaxAbsScaler()
X_toy_scaled = max_abs_scaler.fit_transform(X_toy)

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 6))
# before
ax1.scatter(X_toy[:, 0], X_toy[:, 1], color='blue')
ax1.set_title('Original Data')
ax1.set_xlabel('Feature 1')
ax1.set_ylabel('Feature 2')
ax1.axhline(0, color='black', linewidth=0.5)
ax1.axvline(0, color='black', linewidth=0.5)
ax1.grid(True)
# after
ax2.scatter(X_toy_scaled[:, 0], X_toy_scaled[:, 1], color='red')
ax2.set_title('Data after MaxAbs Scaling')
ax2.set_xlabel('Feature 1')
ax2.set_ylabel('Feature 2')
ax2.axhline(0, color='black', linewidth=0.5)
ax2.axvline(0, color='black', linewidth=0.5)
ax2.grid(True)
plt.tight_layout()
plt.show()


plt.figure(figsize=(12, 6))
# before scaling
plt.subplot(1, 2, 1)
sns.kdeplot(X_toy[:, 0], fill=True, color="blue")
plt.title('PDF of Feature 1 Before MaxAbs Scaling')
plt.xlabel('Feature 1')
plt.ylabel('Density')
# after scaling
plt.subplot(1, 2, 2)
sns.kdeplot(X_toy_scaled[:, 0], fill=True, color="red")
plt.title('PDF of Feature 1 After MaxAbs Scaling')
plt.xlabel('Feature 1')
plt.ylabel('Density')
plt.tight_layout()
plt.show()

4. L1 and L2 Normalization

L1 and L2 Normalization are used when the importance of a row is greater than the importance of a column, or when vectors are involved. like recommendation system, embedding, etc. Column wise distribution of all features will change as our focus is on rows only.

Use-case:

1. L1 is used for high-dimensional and sparse features, like text preprocessing, etc.
2. Similarity and ranking based algorithms like cosine similarity, etc.
3. Image processing based applications
4. Neural network embeddings

# Generating a toy dataset
np.random.seed(0)
X_toy = np.random.rand(100, 2) * 100

X_l2_normalized = normalize(X_toy, norm='l2')
X_l1_normalized = normalize(X_toy, norm='l1')


# Plotting the original, L2 normalized, and L1 normalized data
fig, ax = plt.subplots(1, 3, figsize=(15, 5))
# Original Data
ax[0].scatter(X_toy[:, 0], X_toy[:, 1], color='blue')
ax[0].set_title('Original Data')
ax[0].set_xlabel('Feature 1')
ax[0].set_ylabel('Feature 2')
# L2 Normalized Data
ax[1].scatter(X_l2_normalized[:, 0], X_l2_normalized[:, 1], color='green')
ax[1].set_title('L2 Normalized Data')
ax[1].set_xlabel('Feature 1')
ax[1].set_ylabel('Feature 2')
# L1 Normalized Data
ax[2].scatter(X_l1_normalized[:, 0], X_l1_normalized[:, 1], color='red')
ax[2].set_title('L1 Normalized Data')
ax[2].set_xlabel('Feature 1')
ax[2].set_ylabel('Feature 2')
plt.tight_layout()
plt.show()

# Plotting the distribution of Feature 1 before and after L2 and L1 normalization
fig, ax = plt.subplots(1, 3, figsize=(15, 5))
# Original Data
ax[0].hist(X_toy[:, 0], bins=15, color='blue', alpha=0.7)
ax[0].set_title('Original Data Feature 1 Distribution')
ax[0].set_xlabel('Feature 1')
ax[0].set_ylabel('Count')
# L2 Normalized
ax[1].hist(X_l2_normalized[:, 0], bins=15, color='green', alpha=0.7)
ax[1].set_title('L2 Normalized Feature 1 Distribution')
ax[1].set_xlabel('Feature 1 (L2 Norm)')
ax[1].set_ylabel('Count')
# L1 Normalized
ax[2].hist(X_l1_normalized[:, 0], bins=15, color='red', alpha=0.7)
ax[2].set_title('L1 Normalized Feature 1 Distribution')
ax[2].set_xlabel('Feature 1 (L1 Norm)')
ax[2].set_ylabel('Count')
plt.tight_layout()
plt.show()

# Creating a new dataset with multiple points and one significant outlier
np.random.seed(0)
X_new = np.random.rand(10, 2)
X_new = np.vstack([X_new, [5, 5]])

X_new_l2_normalized = normalize(X_new, norm='l2')
X_new_l1_normalized = normalize(X_new, norm='l1')

fig, axes = plt.subplots(1, 3, figsize=(18, 6))
# Original data
axes[0].scatter(X_new[:-1, 0], X_new[:-1, 1], color='blue')
axes[0].scatter(X_new[-1, 0], X_new[-1, 1], color='orange', label='Outlier')
axes[0].set_title('Original Data with Outlier')
axes[0].set_xlim(-1, 6)
axes[0].set_ylim(-1, 6)
axes[0].set_xlabel('Feature 1')
axes[0].set_ylabel('Feature 2')
axes[0].axhline(0, color='black', linewidth=0.5)
axes[0].axvline(0, color='black', linewidth=0.5)
axes[0].legend()
# L2 Normalized
axes[1].scatter(X_new_l2_normalized[:-1, 0], X_new_l2_normalized[:-1, 1], color='green')
axes[1].scatter(X_new_l2_normalized[-1, 0], X_new_l2_normalized[-1, 1], color='orange', label='Outlier')
axes[1].set_title('L2 Normalized Data')
axes[1].set_xlim(-0.1, 1.1)
axes[1].set_ylim(-0.1, 1.1)
axes[1].set_xlabel('Feature 1 (L2 Norm)')
axes[1].set_ylabel('Feature 2 (L2 Norm)')
axes[1].axhline(0, color='black', linewidth=0.5)
axes[1].axvline(0, color='black', linewidth=0.5)
axes[1].legend()
# L1 Normalized
axes[2].scatter(X_new_l1_normalized[:-1, 0], X_new_l1_normalized[:-1, 1], color='red')
axes[2].scatter(X_new_l1_normalized[-1, 0], X_new_l1_normalized[-1, 1], color='orange', label='Outlier')
axes[2].set_title('L1 Normalized Data')
axes[2].set_xlim(-0.1, 1.1)
axes[2].set_ylim(-0.1, 1.1)
axes[2].set_xlabel('Feature 1 (L1 Norm)')
axes[2].set_ylabel('Feature 2 (L1 Norm)')
axes[2].axhline(0, color='black', linewidth=0.5)
axes[2].axvline(0, color='black', linewidth=0.5)
axes[2].legend()

plt.tight_layout()
plt.show()

➡️ Next Up: [Discretization/Binning Explained in Detail]

If you found value in this content, I invite you to stay connected for more insightful posts. Your time and interest are greatly appreciated. Thank you for reading!