Feature Transformations Explained

Unlocking Complex Transformations: The Power of Box-Cox and Yeo-Johnson Transformation

Photo by Jimmy Liu on Unsplash

⬅️ Previous: [Outlier Detection in Machine Learning Explained]

Feature Transformations

Feature transformation involves applying mathematical operations to modify the original numerical features/targets in a way that enhances their representation for machine learning models and conform to model assumptions, which in turn amplifies the model’s predictive power. This helps in improving model accuracy. It is very important for linear models and not important for tree-based models.

Advantages:

1. It can even out the variance.
2. To make the feature more normal.
3. It can reduce the skewness.
4. It can linearize the relationship between the feature and the target.
5. Reduce the impact of outliers.

Disadvantages:

1. Interpretation is lost.
2. Finding the best transformation is tricky.
3. Additional step in an already complex pipeline

Types of transformation:

1. Log Transformations: Log Transformations.
2. Power Transformations: Reciprocal, Square, Square Root Transformations, etc.
3. Complex Transformations: Box-Cox and Yeo-Johnson Transformations.

1. Log Transformation

Log Transformation compress the exponential scale of numberical feature of a moderate right skewed (skewness is between 1 to 3 or 4) dataset to a normal like distribution. Inverse of Log Transform is possible. Algorithms that benefit from Log Transform are Linear Models, ANOVA, Time Series Analysis, K-Means, PCA, Gaussian Naïve Bayes, Neural Networks etc.

NOTE: we can check for Normality using i. PDF ii. Skew iii. QQ Plot iv. Statistical test etc.

When to use?

1. When you have moderate right skewed dataset.
2. When your data contains outliers as it reduces impact of outliers. We can apply Log Transformation first before apply MinMax Scaler, thus increases the power of outlier handling of MinMax Scaler.
3. Reduces Heteroskedasticity.

When not to use?

1. With negative or zero values.
2. With normal or uniform distribution.
3. When interpretation is important.

Example:

from sklearn.metrics import accuracy_score, r2_score, mean_squared_error
from sklearn.preprocessing import PowerTransformer, FunctionTransformer
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.linear_model import LogisticRegression, LinearRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.compose import ColumnTransformer
import matplotlib.pyplot as plt
from scipy.stats import skew
import scipy.stats as stats
import seaborn as sns
import pandas as pd
import numpy as np

df = pd.read_csv('train.csv', usecols=['Age', 'Fare', 'Survived'])

# replacing missing values with mean
df['Age'].fillna(df['Age'].mean(), inplace=True)
df.head()

X = df.iloc[:, 1:3]
y = df.iloc[:, 0]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

plt.figure(figsize=(14, 4))
plt.subplot(121)
sns.distplot(X_train['Age'])
plt.title('Age PDF')
plt.subplot(122)
stats.probplot(X_train['Age'], dist="norm", plot=plt)
plt.title('Age QQ Plot')
plt.show()

plt.figure(figsize=(14, 4))
plt.subplot(121)
sns.distplot(X_train['Fare'])
plt.title('Fare PDF')
plt.subplot(122)
stats.probplot(X_train['Fare'], dist="norm", plot=plt)
plt.title('Fare QQ Plot')
plt.show()

Age moderately follow normal distribution.

Fare is moderately right skewed.

trf = ColumnTransformer([('log', FunctionTransformer(np.log1p), ['Fare'])], remainder='passthrough')

X_train_transformed = trf.fit_transform(X_train)
X_test_transformed = trf.transform(X_test)

clf = LogisticRegression()
clf.fit(X_train_transformed, y_train)
y_pred = clf.predict(X_test_transformed)
print(accuracy_score(y_test, y_pred)) # 0.6815642458100558

X_transformed = trf.fit_transform(X)
clf = LogisticRegression()
print(np.mean(cross_val_score(clf, X_transformed, y, scoring='accuracy', cv=10)))
# 0.678027465667915

plt.figure(figsize=(14,4))
plt.subplot(121)
stats.probplot(X_train['Fare'], dist="norm", plot=plt)
plt.title('Fare Before Log')
plt.subplot(122)
stats.probplot(X_train_transformed['Fare'], dist="norm", plot=plt)
plt.title('Fare After Log')
plt.show()

We can notice improvement in QQ plot and Fare is now follow more nomal like distribution than before.

2. Square Root Transformation

Square Root Transformation is a less extreme version of Log Transformation and works on mildly right-skewed (skewness is between 0 to 1) dataset. All properties of Log Transformation also apply here.

3. Square Transformation

Square Transformation is used with mildly left-skewed (skewness is between -1 to 0) dataset and non-linear relationships.

NOTE: To increase the strength of the transformation, we can increase the power to 3, 4, etc., from a square transformation.

Example of non-linear relationships:

# Because the highest power of X is 2, the true relationship is a downward-opening parabola.
np.random.seed(42)
X = np.random.uniform(0, 10, 200)
Y = -X**2 + 10*X + np.random.normal(0, 5, 200)
df = pd.DataFrame({'X': X, 'Y': Y})

df['X_squared'] = df['X']**2
df_sorted = df.sort_values(by='X')

plt.scatter(df_sorted['X'], df_sorted['Y'], color='blue', label='Actual Data', alpha=0.6)
plt.show()

plt.scatter(df_sorted['X_squared'], df_sorted['Y'], color='blue', label='Actual Data', alpha=0.6)
plt.show()

# Linear model using X
linear_model = LinearRegression()
linear_model.fit(df[['X']], df['Y'])
df['Linear_Prediction'] = linear_model.predict(df[['X']])

# Linear model using X^2
df['X_squared'] = df['X']**2
squared_model = LinearRegression()
squared_model.fit(df[['X', 'X_squared']], df['Y'])
df['Squared_Prediction'] = squared_model.predict(df[['X', 'X_squared']])

# Sort the dataframe by X values for better plotting
df_sorted = df.sort_values(by='X')

# Plot the data and the models with sorted values
plt.figure(figsize=(10, 6))
plt.scatter(df_sorted['X'], df_sorted['Y'], color='blue', label='Actual Data', alpha=0.6)
plt.plot(df_sorted['X'], df_sorted['Linear_Prediction'], color='red', label='Linear Prediction')
plt.plot(df_sorted['X'], df_sorted['Squared_Prediction'], color='green', label='Squared Prediction')
plt.title('Comparison of Linear and Squared Models (Corrected)')
plt.xlabel('X')
plt.ylabel('Y')
plt.legend()
plt.show()

Linear regression is linear with respect to its parameters (weights), not necessarily its features (inputs). By creating a new column for X², we are taking a 1D non-linear problem and projecting it into a 2D space where a linear algorithm can solve it.

When you pass only X into LinearRegression(), the algorithm is forced to find the best straight line. It searches for two numbers (a weight w1 and a bias b) to satisfy this equation Y = w1X + b.

A straight line cannot bend, so it draws the best average line through the curve, resulting in massive underfitting (the red line in your plot).

You explicitly calculate the square of every X value and store it in a new column. To you, X and X² are related. To the machine learning algorithm, they are completely independent variables. It doesn’t know that column 2 is the square of column 1; it just sees two distinct features: Feature1 and Feature2.

Now, you feed both columns to the LinearRegression() model. Because it has two features, the standard linear algorithm searches for three numbers (w1, w2, and b) to satisfy this equation Y = w1(Feature1) + w2(Feature2) + b.

Since Feature1 = X and Feature2 = X2, the model is actually solving:Y = w1X + w2X2 + b

Get Mr. Vivek Kumar’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

When you plot those predictions back against your original 1D X-axis, the result is a beautiful, curved green parabola that captures the non-linear relationship perfectly.

4. Reciprocal Transformation

Reciprocal Transformation is used when feature values are greater than 1 (otherwise, it will exponentially increase the result instead of reducing) and for strongly right-skewed (skewness is greater than 3 or 4) features. It can also be used to make feature linear from non-linear.

Example

df = pd.read_csv('https://raw.githubusercontent.com/selva86/datasets/master/BostonHousing.csv')
df.head()

# Plotting the distplots without any transformation
for col in df.columns[0:-1]:
    plt.figure(figsize=(14,4)) 
    plt.subplot(121)
    sns.kdeplot(df[col])
    txt = col + " " + str(df.skew()[col])
    plt.title(txt)
    plt.show()

Applying Log Transformation on Moderately Right-Skewed Data

# log on zn feature
fig, axs = plt.subplots(ncols=2, figsize=(12, 6))
sns.kdeplot(df['zn'], color='blue', ax=axs[0])
sns.kdeplot(np.log1p(df['zn']), color='red', ax=axs[1])
plt.tight_layout()
plt.show()

print('skew before log transform', skew(df['zn']))
print('skew after log transform', skew(np.log1p(df['zn'])))
# skew before log transform 2.219063057148425
# skew after log transform 1.1899105701640726

# log on chas
fig, axs = plt.subplots(ncols=2, figsize=(12, 6))
sns.kdeplot(df['chas'], color='blue', ax=axs[0])
sns.kdeplot(np.log1p(df['chas']), color='red', ax=axs[1])
plt.tight_layout()
plt.show()

print('skew before reciprocal transform', skew(df['chas']))
print('skew after reciprocal transform', skew(np.log1p(df['chas'])))
# skew before reciprocal transform 3.395799292642519
# skew after reciprocal transform 3.3957992926425202

Skewness has not decreased after applying log transformation on the chas. Because it only contains 0s and 1s (i.e. chas is a categorical feature), it does not have a “continuous” distribution that can be un-skewed. So, no need to apply any transformation on the categorical feature.

# log on dis
fig, axs = plt.subplots(ncols=2, figsize=(12, 6))
sns.kdeplot(df['dis'], color='blue', ax=axs[0])
sns.kdeplot(np.log1p(df['dis']), color='red', ax=axs[1])
plt.tight_layout()
plt.show()

print('skew before log transform', skew(df['dis']))
print('skew after log transform', skew(np.log1p(df['dis'])))
# skew before log transform 1.0087787565152246
# skew after log transform 0.33057774263430006

# log on rad
fig, axs = plt.subplots(ncols=2, figsize=(12, 6))
sns.kdeplot(df['rad'], color='blue', ax=axs[0])
sns.kdeplot(np.log1p(df['rad']), color='red', ax=axs[1])
plt.tight_layout()
plt.show()

print('skew before log transform', skew(df['rad']))
print('skew after log transform', skew(np.log1p(df['rad'])))
# skew before log transform 1.0018334924536951
# skew after log transform 0.5311793311965491

Applying Square Root on Mild Right Skewness

# sqrt on indus
fig, axs = plt.subplots(ncols=2, figsize=(12, 6))
sns.kdeplot(df['indus'], color='blue', ax=axs[0])
sns.kdeplot(np.sqrt(df['indus']), color='red', ax=axs[1])
plt.tight_layout()
plt.show()

print('skew before sqrt transform', skew(df['indus']))
print('skew after sqrt transform', skew(np.sqrt(df['indus'])))
# skew before log transform 0.29414627684418543
# skew after log transform -0.07239396333108224

It has reduced to slightly negative and that is ok.

# sqrt on nox
fig, axs = plt.subplots(ncols=2, figsize=(12, 6))
sns.kdeplot(df['nox'], color='blue', ax=axs[0])
sns.kdeplot(np.sqrt(df['nox']), color='red', ax=axs[1])
plt.tight_layout()
plt.show()

print('skew before sqrt transform', skew(df['nox']))
print('skew after sqrt transform', skew(np.sqrt(df['nox'])))
# skew before sqrt transform 0.7271441597740319
# skew after sqrt transform 0.5351964490014817

# sqrt on rm
fig, axs = plt.subplots(ncols=2, figsize=(12, 6))
sns.kdeplot(df['rm'], color='blue', ax=axs[0])
sns.kdeplot(np.sqrt(df['rm']), color='red', ax=axs[1])
plt.tight_layout()
plt.show()

print('skew before sqrt transform', skew(df['rm']))
print('skew after sqrt transform', skew(np.sqrt(df['rm'])))
# skew before sqrt transform 0.40241466802496245
# skew after sqrt transform 0.07249861410468098

# sqrt on tax
fig, axs = plt.subplots(ncols=2, figsize=(12, 6))
sns.kdeplot(df['tax'], color='blue', ax=axs[0])
sns.kdeplot(np.sqrt(df['tax']), color='red', ax=axs[1])
plt.tight_layout()
plt.show()

print('skew before sqrt transform', skew(df['tax']))
print('skew after sqrt transform', skew(np.sqrt(df['tax'])))
# skew before sqrt transform 0.6679682687237768
# skew after sqrt transform 0.5202453627239406

Square Root Transformation doesn’t perform well for Bimodal Distribution. Instead, split the data into subgroups, use mixture models, or switch to tree-based models.

# sqrt on lstat
fig, axs = plt.subplots(ncols=2, figsize=(12, 6))
sns.kdeplot(df['lstat'], color='blue', ax=axs[0])
sns.kdeplot(np.sqrt(df['lstat']), color='red', ax=axs[1])
plt.tight_layout()
plt.show()

print('skew before sqrt transform', skew(df['lstat']))
print('skew after sqrt transform', skew(np.sqrt(df['lstat'])))
# skew before sqrt transform 0.9037707431346133
# skew after sqrt transform 0.30647851994358943

Applying Reciprocal transformation on Strong Right Skewness

# reciprocal on crim
fig, axs = plt.subplots(ncols=2, figsize=(12, 6))
sns.kdeplot(df['crim'], color='blue', ax=axs[0])
sns.kdeplot(np.reciprocal(df['crim']), color='red', ax=axs[1])
plt.tight_layout()
plt.show()

print('skew before reciprocal transform', skew(df['crim']))
print('skew after reciprocal transform', skew(np.reciprocal(df['crim'])))
# skew before reciprocal transform 5.207652387859715
# skew after reciprocal transform 3.733474573321791

Applying Square transformation on Mild Left Skewness

# square on age
fig, axs = plt.subplots(ncols=2, figsize=(12, 6))
sns.kdeplot(df['age'], color='blue', ax=axs[0])
sns.kdeplot(np.square(df['age']), color='red', ax=axs[1])
plt.tight_layout()
plt.show()

print('skew before reciprocal transform', skew(df['age']))
print('skew after reciprocal transform', skew(np.square(df['age'])))
# skew before reciprocal transform -0.5971855948016143
# skew after reciprocal transform -0.18263633415807376

# square on ptratio
fig, axs = plt.subplots(ncols=2, figsize=(12, 6))
sns.kdeplot(df['ptratio'], color='blue', ax=axs[0])
sns.kdeplot(np.square(df['ptratio']), color='red', ax=axs[1])
plt.tight_layout()
plt.show()

print('skew before reciprocal transform', skew(df['ptratio']))
print('skew after reciprocal transform', skew(np.square(df['ptratio'])))
# skew before reciprocal transform -0.7999445320367449
# skew after reciprocal transform -0.6085086626298968

Applying Log Transformation on Moderately Left-Skewed Data

To deal with moderately left skewed (skewness is smaller than -1) dataset, we change the left skewness to right skewness by applying a negative sign. But, this will contain negative values and log transformation doesn't work with negative values. So, we shift towards positive x-axis by maximum value. Now, we apply log transformation on moderately left skewed dataset or reciprocal transformation for strong left skewed dataset.

# reflect log on b
# max(-df['b']) # 396.900000001
fig, axs = plt.subplots(ncols=2, figsize=(12, 6))
sns.kdeplot(df['b'], color='blue', ax=axs[0])
sns.kdeplot(np.log1p(-df['b'] + max(-df['b'])), color='red', ax=axs[1])
plt.tight_layout()
plt.show()

print('skew before reflect log transform', skew(df['b']))
print('skew after reflect log transform', skew(np.log1p(-df['b'] + max(-df['b']))))
# skew before reflect log transform -2.8817983452627716
# skew after reflect log transform 0.5995750622214898

Model training

X = df.drop('medv', axis=1)
y = df['medv']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)


def reflect_shift_log_transform(x):
    """
    Function to perform Reflect, shift and log transformation.
    """
    return np.log1p(-x + 396.900000001)

preprocessor = ColumnTransformer(
    transformers=[
        ('log', FunctionTransformer(np.log1p), ['zn', 'chas', 'dis', 'rad']),
        ('sqrt', FunctionTransformer(np.sqrt), ['indus', 'nox', 'rm', 'tax', 'lstat']),
        ('reciprocal', FunctionTransformer(np.reciprocal), ['crim']),
        ('square', FunctionTransformer(np.square), ['age', 'ptratio']),
        ('reflect_shift_log', FunctionTransformer(reflect_shift_log_transform), ['b']),
    ],
    remainder='passthrough'
)
X_train_trf = preprocessor.fit_transform(X_train)
X_test_trf = preprocessor.transform(X_test)


reg = LinearRegression()
reg.fit(X_train_trf, y_train)
y_pred = reg.predict(X_test_trf)

print('r2 score', r2_score(y_test, y_pred))
print('mse', mean_squared_error(y_test, y_pred))
# r2 score 0.7219859878030749
# mse 20.38781928598069

We have to check the plot, skewness, distribution and apply transformation on each column one by one. That is why we need Complex Transformations like Box-Cox and Yeo-Johnson Transformation.

Box-Cox Transformation

The Box-Cox transformation is a family of power transformations that are applied to data to stabilize variance, make the data more closely follow a normal distribution, or improve the skewness of both the right and left skewed data. It’s particularly useful when dealing with non-normal data that violates assumptions of many statistical tests and models. The parameter, λ is usually determined by maximizing the log-likelihood function or MLE (Maximum Likelihood Estimation), which effectively finds the best value that makes the data as normal as possible after the transformation.

When to use:

1. Handle Skewness
2. Handle non-normal data.
3. Heteroscedasticity
4. Can act as general case to many other transformations.

When not to use:

1. Negative and zero values are present in the data.
2. Interpretability is a concern.
3. Data is already normally distributed.
4. Don’t use on Categorical Data.

pt = PowerTransformer(method='box-cox')
X_train_transformed = pt.fit_transform(X_train+0.000001) # change 0 to 0.000001 for box-cox transformation
X_test_transformed = pt.transform(X_test+0.000001)

reg = LinearRegression()
reg.fit(X_train_transformed, y_train)
y_pred = reg.predict(X_test_transformed)

print('r2 score', r2_score(y_test, y_pred)) # r2 score 0.7321931094639539
print('mse', mean_squared_error(y_test, y_pred)) # mse 19.639292439410752

pd.DataFrame({'cols':X_train.columns, 'box_cox_lambdas':pt.lambdas_})

# Before and after comparison for Box-Cox Plot
X_train_transformed = pd.DataFrame(X_train_transformed, columns=X_train.columns)

for col in X_train_transformed.columns:
    plt.figure(figsize=(14, 4))

    plt.subplot(121)
    sns.kdeplot(X_train[col])
    txt = col + " -> " + str(X_train[col].skew())
    plt.title(txt)

    plt.subplot(122)
    sns.kdeplot(X_train_transformed[col])
    txt1 = col + " -> " + str(X_train_transformed[col].skew())
    plt.title(txt1)

    plt.show()

Yeo-Johnson Transformation

The Yeo-Johnson transformation is a modification of the Box-Cox transformation, designed to handle both positive and negative data, making it more versatile in practical applications. This transformation is particularly useful when dealing with real-world data that can contain both negative and positive values, ensuring that a wider range of datasets can be normalized for analysis.

pt1 = PowerTransformer() # default is Yeo-Johnson
X_train_transformed2 = pt1.fit_transform(X_train)
X_test_transformed2 = pt1.transform(X_test)

lr = LinearRegression()
lr.fit(X_train_transformed2, y_train)
y_pred3 = lr.predict(X_test_transformed2)
print(r2_score(y_test, y_pred3)) # 0.73149141

pd.DataFrame({'cols':X_train.columns, 'box_cox_lambdas':pt.lambdas_, 'Yeo_Johnson_lambdas':pt1.lambdas_})

# Before and after comparison for Box-Cox Plot
X_train_transformed2 = pd.DataFrame(X_train_transformed2 , columns=X_train.columns)

for col in X_train_transformed2.columns:
    plt.figure(figsize=(14,4))

    plt.subplot(121)
    sns.kdeplot(X_train[col])
    txt = col + " -> " + str(X_train[col].skew())
    plt.title(txt)

    plt.subplot(122)
    sns.kdeplot(X_train_transformed2[col])
    txt1 = col + " -> " + str(X_train_transformed2[col].skew())
    plt.title(txt1)

    plt.show()

➡️ Next Up: [Feature Scaling Explained: Min-Max, Robust and Standard Scalers]

If you found value in this content, I invite you to stay connected for more insightful posts. Your time and interest are greatly appreciated. Thank you for reading!