Feature Engineering in Machine Learning

Feature Engineering is one of the most crucial steps in building effective machine learning models. It involves transforming raw data into a machine-understandable format that helps models learn patterns efficiently and make accurate predictions.

The performance of any model depends heavily on the quality and relevance of features used during training. While there is no single “best” feature engineering method or pipeline, carefully analyzing the data helps identify which features are most important and how many should be included to solve a given problem effectively.

What is Feature Engineering?

Feature Engineering is the process of transforming, creating, or selecting variables (called features) from raw data to improve a model’s performance.

The Key Steps in Feature Engineering

i) Feature Transformation
ii) Feature Selection
iii) Feature Extraction

i) Feature Transformation

It is important to transform raw data into a format that provides higher-quality results.
If the data contains missing values, duplicates, or outliers, models will not perform well.

This process involves creating, modifying, or selecting features to enhance a model’s ability to learn patterns and make accurate predictions.
It helps reduce noise and improves generalization to unseen data.

Handling Missing Data

If there is any missing value in the data, there are different options to handle it. You can remove those rows, but this is not preferred if doing so also removes important data.
Another option is to fill (impute) the missing values.

If less than 5% of the data has missing values, it is acceptable to delete those rows.

The SimpleImputer (in scikit-learn) can fill missing values using different strategies such as:

• Mean Imputation — replaces missing values with the column’s average
• Median Imputation — replaces missing values with the median
• Mode Imputation — replaces missing values with the most frequent value
• Constant Value Imputation — fills missing values with a fixed number or string

ii) Encoding Categorical Variables

Since most machine learning algorithms require numerical input, categorical variables must be transformed into numerical form using encoding techniques.

1) Label Encoding

Label encoding assigns a unique integer to each category.

Example:
Red = 0
Green = 1
Blue = 2

Use Cases:

• Ordinal categorical variables: e.g., small=0, medium=1, large=2
• Binary categories: e.g., yes=1, no=0
• Tree-based models: Algorithms like Decision Tree, Random Forest, and Gradient Boosting can handle label-encoded nominal data effectively.

2) One-Hot Encoding

One-hot encoding is used for nominal categorical variables, such as colors red, blue, green.
It is recommended when the number of categories is small (e.g., less than 10).

Models that assume linearity or use distance metrics such as Linear Regression, Logistic Regression, or K-Nearest Neighbors (KNN) benefit most from one-hot encoding.

How It Works:
One-hot encoding creates new columns that indicate the presence or absence of each possible value in the original data.

For example, if a dataset has a “color” feature with three categories red, yellow, and green the one-hot encoded dataset will have three columns: red, yellow, and green.
Whenever the original color is red, we put a 1 in the red column.
If it’s yellow, we put a 1 in the yellow column, and so on.

Why Encoding Is Necessary

Machine learning models such as Linear Regression, Decision Trees, or Neural Networks typically operate on numerical data.
Categorical variables, being non-numeric, must be converted into numbers so that these algorithms can process them.

However, the encoding method must correctly represent the variable’s properties to avoid misleading the model.

iii) Scaling and Normalization

Feature scaling ensures that all numerical variables are on a similar scale, so that no feature dominates the others due to its range.

Common Methods Include:

• Standardization (Z-score)
• Min-Max Scaling
• Normalization

iv) Creating Interaction Features

Interaction features are created by combining two or more existing features to capture relationships between them.

Get shila’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Example:
Total_Price = Quantity × Unit_Price

Such features can help models discover deeper patterns in the data.

v) Binning and Discretization

This technique converts continuous numerical data into categorical features.
Binning creates categories (bins) from continuous values.
Smoothing bins is a form of local smoothing that helps reduce noise in the input data.

Example:
Ages can be grouped into:

• 0–18 (Children)
• 19–35 (Youth)
• 36–60 (Adults)

This simplifies models and reduces the effect of noise.

ii) Feature Selection

• > Feature selection is the process of selecting features of a dataset to use when training data in machine learning model. It is the most important process and very impactful in ml performance. As irrelevant features weaken a performance, while choosing the right features for a amodel makes it more accurate, more precise.

The benifits of feature selection include:

i) Better model performance

ii) Reduced overfitting

iii) Shorter training times

iii) Feature Extraction

-> Feature extraction is a technique that reduces the complexity of data to improve the performance and efficiency of machine learning (ML) algorithms. During the extraction process, unstructured data is converted into a more structured and usable format to enhance the data quality and model interpretability. Feature extraction is a subset of feature engineering. It is especially useful when working with large datasets that have many correlated or redundant features. It reduces noise and helps the model train faster without losing key patterns or relationships.

Model Validation

Once you have created a model, you need to evaluate how accurate it is.

For example,

if a house actually costs 15 crore but your model predicts 10 crore, then the error is 5 crore.
To summarize model quality, we use MAE (Mean Absolute Error).

If your model predicts that all green-colored houses have high prices, it may show high accuracy on the current data.
However, when new data has different patterns, accuracy may drop.

The best way to evaluate your model is to exclude some data from training and test the model’s accuracy on data it hasn’t seen before.
This is called validation data.

The scikit-learn library provides a function called train_test_split to divide data into two parts:

• Training data — used to fit the model
• Validation data — used to calculate mean_absolute_error

Cross-Validation

Cross-validation is a key technique in machine learning used to evaluate how well a model performs on unseen data.
It provides a more reliable estimate of a model’s performance compared to a single train-test split by systematically testing the model on different subsets of the data.

Cross-validation is a resampling method that splits a dataset into multiple parts to validate and assess a model’s performance.
The basic idea is to train the model on some portions of the data and test it on the remaining portion.

Benefits of Cross-Validation

1. Better Performance Estimate
A simple train-test split can give misleading results if the test set is unusually easy or hard.
Cross-validation averages performance across multiple splits, reducing bias and providing a more accurate measure of how the model will generalize.

2. Prevents Overfitting and Underfitting
Since data is split multiple times, cross-validation helps detect both overfitting and underfitting.

3. Maximizes Data Usage
It makes full use of limited data by training and validating on different subsets.

How Cross-Validation Works

The most common method is K-Fold Cross-Validation.

Steps:

1. Divide the dataset into K equal folds.
2. Use K–1 folds for training and 1 fold for testing.
3. Repeat the process K times, each time using a different fold as the test set.
4. Average all test scores to estimate model performance.

Conclusion

Feature Engineering is the foundation of every successful machine learning project.
Even the most advanced algorithm cannot perform well without well-prepared data.

By carefully handling missing values, encoding categorical variables, scaling numerical features, and validating models using cross-validation, we ensure our models become both accurate and generalizable.