Machine Learning Pipeline & Data Preprocessing: A Practical Guide

A beginner-friendly, code-first guide to the ML workflow, preprocessing techniques, model training, evaluation metrics, and data leakage.

Why Data Preparation Matters

Imagine building a model to predict house prices. Your dataset contains missing values, inconsistent formats, and features measured on different scales.

If these problems are ignored, even a powerful algorithm can produce unreliable predictions.

A successful machine learning project is not just about choosing the right algorithm. It is about preparing the data, evaluating the model correctly, and monitoring its performance in the real world.

In this guide, we’ll break down the machine learning pipeline using simple explanations, visual diagrams, and short Python examples.

1. Understanding the Machine Learning Pipeline

A machine learning pipeline is a sequence of steps that transforms raw data into a model capable of making predictions.

Think of it as a journey: we start with a real-world problem, prepare the data, train a model, evaluate its performance, and finally deploy it for real use.

The main stages are:

• Problem Definition: Identify what needs to be predicted and how success will be measured.
• Data Collection: Gather relevant data from files, databases, APIs, or other sources.
• Exploratory Data Analysis (EDA): Understand data patterns, missing values, and outliers.
• Data Preprocessing: Clean data, encode categories, and scale numerical features when needed.
• Feature Engineering: Create or improve features that help the model learn.
• Model Training: Train an algorithm using prepared training data.
• Model Evaluation: Measure performance using suitable metrics.
• Deployment: Make the model available through an application or API.
• Monitoring: Track performance and detect changes after deployment.

Figure 1: The end-to-end machine learning pipeline and its continuous feedback loop.

Key takeaway: A machine learning project is not just about training a model. Every stage contributes to the reliability of its predictions.

2. Exploratory Data Analysis (EDA)

Before training a machine learning model, we need to understand the data.

Exploratory Data Analysis (EDA) helps us discover patterns, identify data-quality problems, and understand relationships between features.

During EDA, we typically examine:

• Dataset structure: Rows, columns, and data types.
• Distributions: How numerical values are spread.
• Missing values: Which features contain incomplete data.
• Outliers: Values that differ significantly from the rest.
• Relationships: How features vary together.

EDA does not automatically fix data problems. It helps us identify them and decide what to do next.

Figure 2 — Common EDA techniques used to understand a dataset before modeling.

import matplotlib.pyplot as plt

# Check numerical distributions
df.hist(figsize=(10, 6))
plt.tight_layout()
plt.show()

Key takeaway: EDA helps us understand what the data contains, what needs attention, and which patterns may be useful for modeling. Always investigate the data before deciding how to preprocess it.

3. Handling Missing Values

Real-world datasets are rarely perfect. Missing values can occur because of incomplete forms, sensor failures, or errors during data collection.

If we ignore them, some algorithms may fail or produce unreliable results.

How do we handle missing values?

• Mean imputation: Replaces missing numerical values with the average. It can be sensitive to outliers.
• Median imputation: Uses the middle value and is often more suitable for skewed numerical data.
• Mode imputation: Replaces missing categorical values with the most frequent category.
• Deletion: Removes rows or columns when justified, but may discard useful information.

There is no single best method for every dataset. Choose based on the feature, distribution, amount of missing data, and why the values are missing.

Figure 3 — Common strategies for detecting and handling missing data.

from sklearn.impute import SimpleImputer

# Numerical columns
imputer = SimpleImputer(strategy="median")

X_train[["Age", "Income"]] = imputer.fit_transform(
    X_train[["Age", "Income"]]
)

X_test[["Age", "Income"]] = imputer.transform(
    X_test[["Age", "Income"]]
)

Key takeaway: Check missing values during EDA, select an appropriate imputation strategy, and fit the imputer on training data only to help prevent data leakage.

4. Feature Scaling: Min-Max Scaling vs. Standardization

Imagine a dataset containing Age (18–70) and Annual Income (10,000–200,000). Because these features have very different ranges, some machine learning algorithms may give greater influence to larger numerical values.

Feature scaling transforms numerical features to comparable scales.

Min-Max Scaling

Min-Max scaling usually transforms values into a range between 0 and 1.

It is useful when a fixed range is desired, but it can be sensitive to extreme values.

Standardization

Standardization transforms data to have a mean of approximately 0 and a standard deviation of 1.

It is commonly useful for algorithms such as Support Vector Machines and Logistic Regression.

Which should you choose? It depends on the algorithm and the data. Tree-based models, such as Random Forests, generally do not require feature scaling.

Figure 4 — Comparing Min-Max scaling and standardization, with guidance on when to use each.

from sklearn.preprocessing import MinMaxScaler, StandardScaler

minmax = MinMaxScaler()
standard = StandardScaler()

X_train_scaled = minmax.fit_transform(X_train)
X_test_scaled = minmax.transform(X_test)

Key takeaway: Scaling helps algorithms that are sensitive to feature magnitudes. Choose Min-Max scaling or standardization based on your data and model, and never fit the scaler on the test set.

5. Categorical Encoding: Converting Text into Numbers

Machine learning models generally work with numerical data. However, real-world datasets often contain categorical values such as city names, education levels, or product types.

Categorical encoding converts these categories into numerical representations that a model can process.

There are three common techniques:

A. One-Hot Encoding

One-Hot Encoding creates a separate binary column for each category. It is useful for unordered categories, such as city names.

Example: Hyderabad, Chennai, and Bangalore become separate columns containing 0 or 1.

B. Ordinal Encoding

Ordinal Encoding assigns numbers to categories that have a meaningful order.

Example: Low → 0, Medium → 1, High → 2.

The numbers represent the order of the categories, not necessarily equal distances between them.

C. Label Encoding

Label Encoding converts target class labels into numbers.

Example: Not Spam → 0, Spam → 1.

In Scikit-learn, LabelEncoder is commonly used for the target variable (y). For unordered input features (X), prefer One-Hot Encoding instead.

Figure 5 — Choosing the right encoding.

Key takeaway: Choose the encoding technique based on whether the categories are unordered, ordered, or target labels.

6. Outlier Detection and Treatment

An outlier is a data point that is unusually far from most other observations in a dataset.

For example, if most houses in a dataset cost between ₹20 lakh and ₹80 lakh, a house priced at ₹8 crore may be an outlier. It could be a data-entry error — or a genuine luxury property.

Outliers can affect statistical measures and some machine learning models. However, not every outlier is an error, so investigate it before deciding what to do.

Common Methods to Detect Outliers

1. Box Plot: Visually highlights values outside the typical range.

2. Interquartile Range (IQR): Uses the middle 50% of the data to identify unusually low or high values.

3. Z-Score: Measures how many standard deviations a value is from the mean. It is most useful when the data is reasonably close to a normal distribution.

Understanding the IQR Method

• (Q1): 25th percentile
• (Q3): 75th percentile
• (IQR = Q3 — Q1)

Calculate the boundaries:

• Lower bound = (Q1–1.5 \times IQR)
• Upper bound = (Q3 + 1.5 \times IQR)

Values outside these boundaries are flagged as potential outliers.

How Should We Handle Outliers?

Depending on the problem, you can correct data-entry errors, cap extreme values, transform skewed features, use robust models, or retain genuine observations.

Key takeaway: Detect outliers, understand why they exist, and choose a treatment based on domain knowledge — not just a formula.

Figure 6: Outlier detection and treatment using box plots, the IQR method, and Python. Outliers should be investigated before deciding whether to remove, cap, transform, or retain them.

Key takeaway: Detect outliers, understand why they exist, and choose a treatment based on domain knowledge — not just a formula.

7. Feature Selection and Class Imbalance

A. Feature Selection

Feature selection is the process of choosing the most useful input features for a machine learning model.

For example, when predicting house prices, features such as location, area, and number of bedrooms may be more useful than an unrelated property ID.

Why is it important?

• Reduces unnecessary or irrelevant features.
• Can reduce training time.
• May improve model performance and interpretability.
• Helps reduce overfitting in some situations.

Common techniques:

• Correlation: Identifies relationships between numerical features and the target.
• Statistical tests: Measures whether features are associated with the target.
• Recursive Feature Elimination (RFE): Repeatedly removes less useful features.
• Model-based selection: Uses feature importance or model coefficients.

Perform feature selection using training data only to avoid data leakage.

B. Class Imbalance

Class imbalance occurs when one class has significantly more examples than another.

Get Hemanthkumarchintakuntla’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Example: In a fraud detection dataset, 99% of transactions may be legitimate and only 1% fraudulent.

A model that predicts every transaction as legitimate could achieve 99% accuracy while failing to detect any fraud.

Common ways to handle it:

• Class weights: Give more importance to the minority class during training.
• Oversampling: Increase minority-class examples.
• Undersampling: Reduce majority-class examples.
• SMOTE: Generates synthetic minority-class examples.

Evaluate performance using precision, recall, F1-score, and the precision-recall curve — not accuracy alone.

Important: Apply resampling only to the training data, never before splitting the full dataset. Otherwise, information can leak into the test set.

Figure 7: Feature selection and class imbalance in machine learning, illustrating feature-selection techniques, class distribution, and approaches such as class weighting and SMOTE.

Key takeaway: Select relevant features and evaluate imbalanced datasets with metrics that reflect the real problem.

8. Train-Test Split and Model Training

A. Train-Test Split

A machine learning model must be evaluated on data it has not seen during training. To do this, we divide the dataset into separate subsets.

• Training set: Used to teach the model patterns in the data.
• Testing set: Used to evaluate how well the trained model performs on unseen data.

A common split is 80% for training and 20% for testing, although the appropriate ratio depends on the dataset and problem.

For classification tasks, stratify=y can help preserve class proportions in both subsets.

For time-series data, use a chronological split instead of randomly shuffling observations.

B. Model Training

Model training is the process of allowing an algorithm to learn patterns from the training data.

For example, a house-price prediction model can learn relationships between house size, number of bedrooms, and price.

The choice of algorithm depends on the problem:

• Regression: Predicts continuous values, such as house prices.
• Classification: Predicts categories, such as spam or not spam.

Start with a simple baseline model, then compare alternatives using appropriate evaluation metrics.

Important: Fit preprocessing steps using training data only. Keep the test set separate until the final evaluation to avoid data leakage.

Figure 8: Train-test splitting and model training in machine learning. The training set teaches the model, while the unseen test set helps evaluate its ability to generalize.

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error

# X = input features, y = target values
X_train, X_test, y_train, y_test = train_test_split(
    X, y,
    test_size=0.2,
    random_state=42
)

model = LinearRegression()
model.fit(X_train, y_train)

predictions = model.predict(X_test)

mae = mean_absolute_error(y_test, predictions)
print("Mean Absolute Error:", mae)

Key takeaway: A model should perform well not only on training data but also on unseen data.

9. Model Evaluation Metrics

Model evaluation helps us understand how well a machine learning model performs on unseen data. The right metric depends on whether the problem is regression or classification.

A. Classification Metrics

Classification predicts categories, such as spam or not spam.

1. Accuracy

The proportion of all predictions that are correct.

2. Precision

Of all the cases predicted as positive, how many were actually positive?

High precision is useful when false alarms are costly.

3. Recall

Of all the actual positive cases, how many did the model correctly identify?

High recall is important when missing a positive case is costly, such as failing to detect fraud.

4. F1-Score

The harmonic mean of precision and recall. It is useful when you need to balance both metrics.

5. Confusion Matrix

A summary of correct and incorrect predictions, showing true positives, true negatives, false positives, and false negatives.

B. Regression Metrics

Regression predicts numerical values, such as house prices.

1. MAE (Mean Absolute Error): The average absolute difference between actual and predicted values. It is easy to interpret in the target’s units.

2. MSE (Mean Squared Error): The average squared prediction error. Larger errors receive a greater penalty.

3. RMSE (Root Mean Squared Error): The square root of MSE, expressed in the target’s units.

4. R² Score: Indicates how well the model explains variation in the target relative to a baseline that predicts the mean. It can be negative when the model performs worse than that baseline.

Figure 9: Machine learning evaluation metrics, including the confusion matrix, accuracy, precision, recall, F1-score, MAE, and RMSE, with guidance on selecting appropriate metrics

Key takeaway: Do not rely on accuracy alone for imbalanced classification, and always choose metrics that match the real-world cost of prediction errors.

10. Data Transformation

Data transformation is the process of changing the format or distribution of data to make it more suitable for machine learning.

Some numerical features may be highly skewed or contain a wide range of values. Certain transformations can help models learn more effectively.

Common Types of Data Transformation

1. Log Transformation

Reduces right skew in positive numerical data by compressing large values.

2. Square Root Transformation

Can reduce moderate right skew in non-negative data.

3. Power Transformation

Techniques such as Box-Cox and Yeo-Johnson can help make data distributions more suitable for some models.

Python Example

import numpy as np
import pandas as pd

df = pd.DataFrame({
    "Income": [10000, 20000, 50000, 100000, 500000]
})
df["Log_Income"] = np.log1p(df["Income"])
print(df)

np.log1p() calculates the natural logarithm of one plus each value, and works with zero as well as positive values.

When Should We Use It?

• When numerical data is strongly skewed.
• When large values dominate a feature’s range.
• When a transformation is appropriate for the chosen algorithm and data.

Limitations and Common Mistakes

• A transformation does not guarantee better model performance.
• Some transformations have domain restrictions.
• Interpretability can become more difficult after transformation
• Fit learned transformations using training data only.

Key takeaway: Choose a transformation based on the data distribution and validate whether it improves performance.

11.Deployment and Monitoring

Deployment is the process of making a trained machine learning model available for real-world use. A model may be integrated into an application, exposed through an API, or deployed as a service.

Monitoring means tracking the model after deployment to ensure it continues to perform reliably. Teams can monitor prediction quality, data changes, model drift, latency, and errors.

For example, a house-price prediction model may become less accurate when market conditions change. Monitoring helps identify this change so the model can be evaluated and updated when necessary.

Key takeaway: A machine learning project does not end when a model is trained. Deployment makes it usable, and monitoring helps maintain its reliability.

Quick Revision: Machine Learning Workflow

• EDA: Understand the dataset before making preprocessing decisions.
• Missing values: Choose deletion or imputation based on the data.
• Feature scaling: Use Min-Max Scaling or Standardization when appropriate for the algorithm.
• Categorical encoding: Use One-Hot Encoding for unordered categories and Ordinal Encoding when categories have a meaningful order.
• Outliers: Investigate unusual values before removing or changing them.
• Feature selection: Retain useful features and avoid using test data to select them.
• Class imbalance: Consider class weights or resampling, and evaluate with suitable metrics.
• Train-test split: Evaluate the model on unseen data.
• Model evaluation: Choose metrics that match the problem and the cost of prediction errors.
• Data leakage: Fit preprocessing steps on training data only.
• Deployment and monitoring: Make the model usable and track its real-world performance.

Remember: Good machine learning depends not only on the algorithm but also on data quality, correct evaluation, and reliable deployment.

References and Further Reading

1. Scikit-learn — Preprocessing Data
2. Scikit-learn — Common Pitfalls and Recommended Practices
3. Scikit-learn — Model Evaluation
4. Scikit-learn — Pipelines and Composite Estimators
5. Imbalanced-learn — Over-sampling Methods
6. Google — Machine Learning Crash Course

Connect with Me

I’m learning and building projects in Machine Learning, Python, Data Science, and AI. Follow my journey and explore my work!

• LinkedIn: linkedin.com/Hemanth-kumar-ch
• GitHub: github.com/Hemanth-kumar-ch
• Medium: hemanthkumarchintakuntla
• Feel free to connect, explore my projects, and share your feedback!