Data Cleaning Techniques Using Python

Data scientists often say that 80% of their work is cleaning data. Real-world datasets are messy: they contain missing values, duplicate rows, inconsistent formatting, and outliers that distort results. If you feed dirty data into an analysis or a machine learning model, you get unreliable output. Python, with libraries like Pandas and NumPy, makes cleaning fast, flexible, and repeatable.

In this blog, we’ll walk through the most important data cleaning techniques with practical code examples that you can apply to almost any dataset.

Why Data Cleaning Matters

Clean data gives you several benefits:

• Accurate analysis: correct insights and fewer misleading conclusions
• Better model performance: machine learning models learn better from consistent data
• Time savings: fewer bugs and surprises later in the pipeline
• Trustworthy reports: stakeholders can rely on your numbers

Skipping this stage might feel faster at first, but it almost always costs more time later when results don’t make sense and you have to trace the problem back to the source.

Step 1: Load and Inspect the Data

Before cleaning anything, understand what you have. Start by loading your dataset and checking its structure, size, and basic statistics.

import pandas as pd
import numpy as np

df = pd.read_csv("sales_data.csv")

print(df.head())      # First 5 rows
print(df.info())      # Column types and non-null counts
print(df.describe())  # Summary statistics
print(df.shape)       # Rows and columnsdf = pd.read_csv("sales_data.csv")

The info() method is especially useful because it shows data types and non-null counts, which quickly reveal where values are missing. The describe() method helps you spot suspicious numbers, such as a negative age or an unrealistically high price.

Step 2: Handle Missing Values

Missing values are the most common problem in raw data. They appear as NaN in Pandas and can break calculations or cause errors in machine learning models. First, find out how many you have in each column:

print(df.isnull().sum())

Once you know where the gaps are, you can either drop the affected rows or fill them with a sensible value.

python

df = df.dropna(subset=["price"])                  # Drop rows missing price
df["age"] = df["age"].fillna(df["age"].median())  # Fill numeric with median
df["city"] = df["city"].fillna("Unknown")         # Fill categorical

Tip: Use the median instead of the mean when your data has outliers, since extreme values don’t pull the median. Dropping rows is fine when only a few are affected, but if many rows have gaps, filling is usually the better choice so you don’t lose valuable information.

Step 3: Remove Duplicate Records

Duplicate rows inflate counts and skew averages, especially when data is merged from multiple sources or exported more than once. A customer who appears twice can easily double your revenue numbers by mistake.

print(df.duplicated().sum())                              # Count duplicates
df = df.drop_duplicates()                                 # Remove exact duplicates
df = df.drop_duplicates(subset=["email"], keep="first")   # Based on a column

The keep="first" argument retains the first occurrence and removes the rest. Use the subset option when only certain columns, like an email or ID, define whether two rows are really the same record.

Step 4: Fix Data Types

Sometimes numbers are stored as text or dates are read as plain strings. Wrong data types break calculations, filtering, and sorting. For example, a price stored as text cannot be averaged, and dates stored as strings won’t sort in true chronological order.

python

df["price"] = pd.to_numeric(df["price"], errors="coerce")
df["order_date"] = pd.to_datetime(df["order_date"], errors="coerce")
df["category"] = df["category"].astype("category")

The errors="coerce" option turns invalid values into NaN instead of crashing, so you can handle them afterwards using the missing value techniques from Step 2. Converting repeated text columns to the category type also saves memory.

Step 5: Standardize Text Data

Inconsistent text is a hidden problem. Python treats “Mumbai”, “mumbai “, and “MUMBAI” as three different values, which leads to wrong group counts and broken joins. Standardizing text fixes this.

Get Devnarayancomp’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

python

df["city"] = df["city"].str.strip().str.lower()
df["country"] = df["country"].replace({
    "USA": "United States",
    "U.S.": "United States",
    "US": "United States"
})

Here, strip() removes extra spaces and lower() makes everything lowercase. The replace() method maps different spellings of the same value to a single standard label, which keeps your categories clean and reliable.

Step 6: Detect and Handle Outliers

Outliers are values that are far from the rest of the data. They may be genuine or they may be errors, so always investigate before removing them. A popular detection method is the IQR (Interquartile Range) technique:

Q1 = df["salary"].quantile(0.25)
Q3 = df["salary"].quantile(0.75)
IQR = Q3 - Q1

lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR

df = df[(df["salary"] >= lower) & (df["salary"] <= upper)]  # Remove
# df["salary"] = df["salary"].clip(lower, upper)            # Or cap

Capping, also called winsorizing, replaces extreme values with the boundary values instead of deleting rows. This is useful when you don’t want to lose data but still want to reduce the influence of extreme numbers.

Step 7: Clean Column Names

Clear, consistent column names make your code easier to read and maintain. Messy names with spaces or mixed capitalization are easy to mistype and cause avoidable errors.

df.columns = df.columns.str.strip().str.lower().str.replace(" ", "_")
df = df.rename(columns={"cust_nm": "customer_name"})
df = df.drop(columns=["unnamed:_0"])  # Remove unnecessary columns

After this step, every column follows the same lowercase, underscore-separated style, and unnecessary columns are gone.

Step 8: Validate the Cleaned Data

After cleaning, always verify the results rather than assuming they’re correct. A quick validation step can catch mistakes before they spread into your analysis.

print(df.isnull().sum())      # Missing values left?
print(df.duplicated().sum())  # Duplicates left?
print(df.dtypes)              # Are types correct?
assert df["price"].min() >= 0, "Negative prices found!"

Using assert statements is a simple way to build data quality checks into your workflow. If a rule is broken, your code stops immediately and tells you what went wrong.

A Reusable Cleaning Function

Wrapping your steps in a function makes the process repeatable for new datasets and saves you from rewriting the same code every time:

python

def clean_data(df):
    df = df.drop_duplicates()
    df.columns = df.columns.str.strip().str.lower().str.replace(" ", "_")
    df["city"] = df["city"].str.strip().str.lower()
    df["price"] = pd.to_numeric(df["price"], errors="coerce")
    df["price"] = df["price"].fillna(df["price"].median())
    df["order_date"] = pd.to_datetime(df["order_date"], errors="coerce")
    return df
clean_df = clean_data(pd.read_csv("sales_data.csv"))

Common Mistakes to Avoid

Even experienced beginners make a few predictable errors when cleaning data:

• Deleting too much data: dropping every row with a missing value can shrink your dataset badly
• Filling blindly: using the mean for skewed data can hide the real pattern
• Ignoring context: an outlier might be a genuine, important value, such as a very large order
• Skipping validation: assuming the cleaning worked without checking the result

Avoiding these mistakes will save you hours of debugging and make your results far more trustworthy.

Best Practices for Data Cleaning

1. Never modify the original file. Always work on a copy.
2. Document every step so your process is reproducible.
3. Understand the context before deleting or filling values.
4. Visualize your data with histograms and box plots to spot problems.
5. Automate repetitive cleaning with functions or pipelines.

Conclusion

Data cleaning is the foundation of every good data project. With Python and Pandas, you can handle missing values, remove duplicates, fix data types, standardize text, and detect outliers in just a few lines of code. The key is to inspect your data first, clean it step by step, and validate the results at the end.

Master these techniques and your analyses and models will be far more reliable. Start practicing on a real dataset, such as one from Kaggle, and you’ll quickly see how much difference clean data makes.

This blog is created as part of learning at Pillai College of Engineering (PCE). For more details, visit Pillai University Admissions | Best University in Mumbai