Lab 1. Plotting Graph On The Data. & Lab 2. Data Cleaning Techniques in Python
1. Plotting Graph On The Data
First We Import Python Libraries by assigning them their name — pd & plt.
- Random — provides a set of functions for generating random numbers.
- Pandas — for data manipulation and analysis.
- matplotlib —for creation of static, animated, and interactive visualizations in Python.
import random
import pandas as pd
import matplotlib.pyplot as pltproviding data of age, gender & income of 100 people.
then converting the data into dataframe of pandas.
data = {
'age': [random.randint(20, 60) for _ in range(100)],
'gender': [random.choice(['Male', 'Female']) for _ in range(100)],
'income': [random.randint(20000, 100000) for _ in range(100)]
}
#convert data to a pandas dataframe
df = pd.DataFrame(data)
df.to_csv('data.csv',index=False)Output Part
print df stands for dataframe by calling it directely.
Table of data. with column name — age, gender & income.
df
- we want histogram chart of age of data.
2. Providing the name of x label as ‘Age’ & the y label as ‘frequency’
and title as ‘Age Distribution’.
plt.hist(data['age'])
plt.xlabel('Age')
plt.ylabel('frequency')
plt.title('Age Distribution')
plt.show()
manipulating the colors of chart, label and text in matplotlib library.
#hist stands for histogram
plt.hist(data['age'],color= 'purple')
plt.xlabel('Age',color='blue')
plt.ylabel('Frequency',color='green')
plt.title('Age Distribution',color='red')
plt.show()
giving bin value of 10,
#hist stands for histogram
plt.hist(data['age'],color= 'purple',bins=20)
plt.xlabel('Age',color='blue')
plt.ylabel('Frequency',color='green')
plt.title('Age Distribution',color='red')
plt.show()giving error in unique() attribute.
plt.bar(data['gender'].unique(), data[gender].value_counts())
plt.xlabel('Gender')
plt.ylabel('Count')
plt.title('Gender Comparison')
plt.show()output
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
/tmp/ipython-input-3366002232.py in <cell line: 0>()
----> 1 plt.bar(data['gender'].unique(), data[gender].value_counts())
2 plt.xlabel('Gender')
3 plt.ylabel('Count')
4 plt.title('Gender Comparison')
5 plt.show()
AttributeError: 'list' object has no attribute 'unique'2. Data Cleaning Techniques in Python
Most tutorial datasets are artificially polished — no gaps, no wrong formats, no surprises.
However, real data collected from users, systems, or sensors almost always contains:
- Missing values
- Text mixed with numbers
- Incomplete time records
If this data is fed directly into a model, the results become unreliable. That’s why data preprocessing is a non-negotiable step before analysis or model training.
To handle this, we rely on Python’s most powerful data libraries.
NumPy
NumPy is designed for numerical computation.
It handles arrays and mathematical operations efficiently, making it ideal for large-scale data calculations.
Pandas
Pandas provides a structured, table-like format for data.
It simplifies reading files, handling missing values, filtering rows, and modifying columns.
Scikit-learn
Scikit-learn is a machine learning library that also offers excellent preprocessing utilities.
It allows data to be scaled, normalized, and encoded so models can learn effectively.
Get Om Kumar Sahu’s stories in your inbox
Join Medium for free to get updates from this writer.
For this to work, We are going to import these libraries
import pandas as pd
import numpy as np
from sklearn.preprocessing import MinMaxScaler, StandardScaler, RobustScaler, LabelEncoderIn Python the library for Scikit-learn is sklearn for short and we are importing these functions for Pre-Processing.
Understanding the Preprocessing Tools
MinMaxScaler
- Rescales values between 0 and 1
- Ensures uniformity across numerical columns
StandardScaler
- Centers data around zero
- Useful when models depend on distance calculations
RobustScaler
- Scales data while ignoring outliers
- Effective when extreme values exist
LabelEncoder
- Converts text labels into numbers
- Makes categorical data usable for models
Creating a Sample Dataset with Errors
To simulate real-world conditions, we intentionally create a dataset with missing values.
data = {
'age': [35,41,23,32,28,36,45,39,44,29],
'income': ['70000','90000','50000','60000','','75000','100000','80000','95000','55000'],
'date':['2020-01-01','2020-01-02','2020-01-03','2020-01-04','2020-01-05','2020-01-06','2020-01-07','','2020-01-09','2020-01-10'],
'marital_status':['married','single','married','single','married','single','married','single','married','single'],
'gender':['female','male','male','female','female','male','female','male','male','female']
}
df = pd.DataFrame(data)
df.to_csv('data.csv', index=False)This dataset includes five attributes:
- Age
- Income
- Date
- Marital Status
- Gender
Some fields are deliberately left blank to replicate data quality issues.
Then its getting structured by Panda(pd) and is stored in df variable.
df is then use to store the data in a data.csv file.
df = pd.read_csv('data.csv')
print(df)This code will print the actual data on the screen in a tabular format.
age income date marital_status gender
0 34 50000.0 2020-01-01 married Male
1 47 96000.0 2020-01-02 married Female
2 21 53000.0 2020-01-03 single Male
3 50 NaN 2020-01-04 married Female
4 24 49000.0 2020-01-05 married Male
5 36 75000.0 2020-01-06 single Male
6 33 100000.0 2020-01-07 single Female
7 47 80000.0 NaN married Male
8 33 95000.0 2020-01-09 single Female
9 22 55000.0 2020-01-10 single MaleIdentifying Missing Data
Before fixing issues, we must locate them.
- We are going to detect missing data using pandas isnull() funtion in the data.
- We are going to use different methods to fix it, some are these : —
1.Use the Mean, Median or Mode for the numerical Data.
2.Forward or backward-fill for Time data.
3.Deleting the Row or Column if beyond repair.
First Lets Find the missing data
print(df.columns)
print(df.isnull().sum())
print(df)This will output this : —
Index(['age', 'income', 'date', 'marital_status', 'gender'], dtype='object')
age 0
income 1
date 1
marital_status 0
gender 0
dtype: int64
age income date marital_status gender
0 34 50000.0 2020-01-01 married Male
1 47 96000.0 2020-01-02 married Female
2 21 53000.0 2020-01-03 single Male
3 50 NaN 2020-01-04 married Female
4 24 49000.0 2020-01-05 married Male
5 36 75000.0 2020-01-06 single Male
6 33 100000.0 2020-01-07 single Female
7 47 80000.0 NaN married Male
8 33 95000.0 2020-01-09 single Female
9 22 55000.0 2020-01-10 single MaleFilling the Missing Data
df['income'].fillna(df['income'].mean(), inplace=True)
print(df)The first command is essentially just replacing Null value with the mean value of the data in income column and so this will be the output.
age income date marital_status gender
0 46 50000.000000 2020-01-01 single Female
1 38 96000.000000 2020-01-02 single Male
2 39 53000.000000 2020-01-03 single Female
3 46 72555.555556 2020-01-04 married Female
4 23 49000.000000 2020-01-05 single Female
5 49 75000.000000 2020-01-06 married Male
6 29 100000.000000 2020-01-07 single Male
7 28 80000.000000 NaN single Female
8 25 95000.000000 2020-01-09 single Female
9 41 55000.000000 2020-01-10 married MaleUsing Forward Fill Method
df['date'].fillna(method='ffill', inplace=True)
print(df)This ffill method work on Columns with Time data, in this case its Date , this function just copy the values from the previous entry.
age income date marital_status gender
0 46 50000.000000 2020-01-01 single Female
1 38 96000.000000 2020-01-02 single Male
2 39 53000.000000 2020-01-03 single Female
3 46 72555.555556 2020-01-04 married Female
4 23 49000.000000 2020-01-05 single Female
5 49 75000.000000 2020-01-06 married Male
6 29 100000.000000 2020-01-07 single Male
7 28 80000.000000 2020-01-07 single Female
8 25 95000.000000 2020-01-09 single Female
9 41 55000.000000 2020-01-10 married MaleWe can also drop rows with missing data using dropna() function.
print("Before")
print(df)
df.dropna(inplace=True)
print("After")
print(df)Before
age income date marital_status gender
0 46 50000.0 2020-01-01 single Female
1 38 96000.0 2020-01-02 single Male
2 39 53000.0 2020-01-03 single Female
3 46 NaN 2020-01-04 married Female
4 23 49000.0 2020-01-05 single Female
5 49 75000.0 2020-01-06 married Male
6 29 100000.0 2020-01-07 single Male
7 28 80000.0 NaN single Female
8 25 95000.0 2020-01-09 single Female
9 41 55000.0 2020-01-10 married Male
After
age income date marital_status gender
0 46 50000.0 2020-01-01 single Female
1 38 96000.0 2020-01-02 single Male
2 39 53000.0 2020-01-03 single Female
4 23 49000.0 2020-01-05 single Female
5 49 75000.0 2020-01-06 married Male
6 29 100000.0 2020-01-07 single Male
8 25 95000.0 2020-01-09 single Female
9 41 55000.0 2020-01-10 married MaleWe can Remove unwanted Column using drop() function
df1 = df.drop('gender', axis=1) //this will remove the gender column
print(df1)age income date marital_status
0 27 50000.0 2020-01-01 married
1 24 96000.0 2020-01-02 married
2 40 53000.0 2020-01-03 married
4 50 49000.0 2020-01-05 married
5 46 75000.0 2020-01-06 married
6 32 100000.0 2020-01-07 married
8 42 95000.0 2020-01-09 single
9 35 55000.0 2020-01-10 single#the element and the column in gender are removed or droppedTo insert Column we can use assign() function.
Im gonna demonstrate how to add a column with the age cubed.
df1 = df1.assign(age_cubed=lambda x: x['age']**3) # assign the age^3 col
print(df1)age income date marital_status age_cubed
0 27 50000.0 2020-01-01 married 19683
1 24 96000.0 2020-01-02 married 13824
2 40 53000.0 2020-01-03 married 64000
4 50 49000.0 2020-01-05 married 125000
5 46 75000.0 2020-01-06 married 97336
6 32 100000.0 2020-01-07 married 32768
8 42 95000.0 2020-01-09 single 74088
9 35 55000.0 2020-01-10 single 42875We can rename the column heading using rename() funciton.
print("Before")
print(df1)
df1 = df1.rename(columns={'income': 'annual_income'})
print("After")
print(df1)Before
age income date marital_status
0 27 50000.0 2020-01-01 married
1 24 96000.0 2020-01-02 married
2 40 53000.0 2020-01-03 married
4 50 49000.0 2020-01-05 married
5 46 75000.0 2020-01-06 married
6 32 100000.0 2020-01-07 married
8 42 95000.0 2020-01-09 single
9 35 55000.0 2020-01-10 single
After
age annual_income date marital_status
0 27 50000.0 2020-01-01 married
1 24 96000.0 2020-01-02 married
2 40 53000.0 2020-01-03 married
4 50 49000.0 2020-01-05 married
5 46 75000.0 2020-01-06 married
6 32 100000.0 2020-01-07 married
8 42 95000.0 2020-01-09 single
9 35 55000.0 2020-01-10 single# The heading of the income changed to annual_incomeFEATURE SCALING
Normalizing the numerical values in the dataset to a common scale to avoid biased results. Starting with Minmaxscaler().
scaler = MinMaxScaler()
df1['age'] = scaler.fit_transform(df1[['age']])
print(df1)This will make the age column fit under a range for better understanding of the model for the datasets.
age annual_income date marital_status
0 0.115385 50000.0 2020-01-01 married
1 0.000000 96000.0 2020-01-02 married
2 0.615385 53000.0 2020-01-03 married
4 1.000000 49000.0 2020-01-05 married
5 0.846154 75000.0 2020-01-06 married
6 0.307692 100000.0 2020-01-07 married
8 0.692308 95000.0 2020-01-09 single
9 0.423077 55000.0 2020-01-10 singleNow Standardscaler() will make the average 0 and range the values accordingly.
scaler = StandardScaler()
df1[['annual_income', 'age_squared']] = scaler.fit_transform(df1[['annual_income', 'age_squared']])
print(df1)# The output
age annual_income date marital_status age_cubed age_cubbed
0 0.115385 -1.025141 2020-01-01 married 0.001536 -0.848822
1 0.000000 1.155506 2020-01-02 married 0.000000 -0.853428
2 0.615385 -0.882925 2020-01-03 married 0.233045 -0.154684
4 1.000000 -1.072547 2020-01-05 married 1.000000 2.144893
5 0.846154 0.159993 2020-01-06 married 0.605826 0.963033
6 0.307692 1.345128 2020-01-07 married 0.029131 -0.766085
8 0.692308 1.108101 2020-01-09 single 0.331816 0.141463
9 0.423077 -0.788114 2020-01-10 single 0.075728 -0.626370Converting Categorical values into numerical values
For better understanding for the model.
Using get_dummies() will make N columns ranging in the values in the age column.
df1 = pd.get_dummies(df, columns=['age'])
print(df1) income date marital_status gender age_24 age_26 age_27 \
0 50000.0 2020-01-01 married Female False False True
1 96000.0 2020-01-02 married Female True False False
2 53000.0 2020-01-03 married Female False False False
3 NaN 2020-01-04 married Female False True False
4 49000.0 2020-01-05 married Male False False False
5 75000.0 2020-01-06 married Male False False False
6 100000.0 2020-01-07 married Female False False False
7 80000.0 NaN single Male False False False
8 95000.0 2020-01-09 single Male False False False
9 55000.0 2020-01-10 single Male False False False age_28 age_32 age_35 age_40 age_42 age_46 age_50
0 False False False False False False False
1 False False False False False False False
2 False False False True False False False
3 False False False False False False False
4 False False False False False False True
5 False False False False False True False
6 False True False False False False False
7 True False False False False False False
8 False False False False True False False
9 False False True False False False FalseLabel Encoding is a function that will give words a number for better understanding and calculation for model.
For Example:-
Red
Blue
Green
after Label encoding
Red → 0
Blue → 1
Green → 2le = LabelEncoder()
df[['marital_status', 'gender']] = df[['marital_status', 'gender']].apply(le.fit_transform)
print(df)Here we gave marital_status and gender a number like 0 for female or 1 for male.
age income date marital_status gender
0 27 50000.0 2020-01-01 0 0
1 24 96000.0 2020-01-02 0 0
2 40 53000.0 2020-01-03 0 0
3 26 NaN 2020-01-04 0 0
4 50 49000.0 2020-01-05 0 1
5 46 75000.0 2020-01-06 0 1
6 32 100000.0 2020-01-07 0 0
7 28 80000.0 NaN 1 1
8 42 95000.0 2020-01-09 1 1
9 35 55000.0 2020-01-10 1 1Final Thoughts
Cleaning and preprocessing data is not optional — it defines the quality of your results.
By handling missing values, scaling features, and encoding categories, your dataset becomes consistent, interpretable, and ready for machine learning models.
Raw data is chaos.
Preprocessing is the discipline that turns it into insight.
