Air Quality Data-Preprocessing

Preprocessing the tabular data

About Dataset

The dataset contains air quality data and AQI (Air Quality Index) at hourly and daily level of various stations across multiple cities in India. This project deals with exploration of the data and modelling to predict the classes namely : Good, Satisfactory, moderate, poor, very poor or severe cases of the air quality. The dataset is obtained from Kaggle here. The data has been made publicly available by the Central Pollution Control Board: https://cpcb.nic.in/ which is the official portal of Government of India.

Import and load the data

Import necessary packages: Here I have imported packages needed for preprocessing and modelling

import pandas as pd
import os 
import zipfile 
import matplotlib.pyplot as plt
import seaborn as sns
import plotly.graph_objs as go
import numpy as np
from plotly.offline import init_notebook_mode,iplot 
import missingno as msno  
from sklearn.impute import KNNImputer   
from sklearn import preprocessing        
import pylab
import scipy.stats as stats      
from scipy.special import boxcox1p
import pylab
import scipy.stats as stats                                                  
%matplotlib inline               
# Transformation and modelling packages
from sklearn.model_selection import train_test_split , KFold                   
from sklearn.metrics import accuracy_score                                        
from sklearn. preprocessing import StandardScaler 
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB
from sklearn.naive_bayes import BernoulliNB
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier 
from xgboost import XGBClassifier
from collections import Counter
from sklearn.model_selection import GridSearchCV ,RandomizedSearchCV
from sklearn.metrics import classification_report                             
from sklearn.metrics import confusion_matrix 
from sklearn import metrics                        
from sklearn.model_selection import cross_val_score 
from sklearn.svm import SVC

The dataset upon loading and observation, it’s important and easy to process the data with smaller size.

• The data types are changed while loading.
• String data is converted into categorical values, float64 to float32 etc. * *
• This process place vital role while handling dataset with larger size.

dtypes1 = {
    'City':'category',
    'Datetime' : 'category', 
    'PM2.5' : 'float32', 
    'PM10' : 'float32', 
    'NO' : 'float32', 
    'NO2' : 'float32', 
    'NOx' : 'float32', 
    'NH3' : 'float32',
    'CO' : 'float32', 
    'SO2' : 'float32', 
    'O3' : 'float32', 
    'Benzene' : 'float32', 
    'Toluene' : 'float32', 
    'Xylene' : 'float32', 
    'AQI' : 'float32', 
    'AQI_Bucket': 'category'
} 

Loop through the zip files and extract files.

path = '/content/drive/MyDrive/Projects/air-quality-data-in-india.zip'          

with zipfile.ZipFile(path, 'r') as f:
  for name in f.namelist():
    if name=='city_hour.csv':
      df_c = pd.read_csv(f.open(name), dtype=dtypes1)

Save the files for in csv format for direct access. Load the csv files using pandas ‘read_csv’

# Save the dtyp changed csv files
fp1 = '/content/drive/MyDrive/Projects/city_hour.csv'
df_c.to_csv(fp1, index=False)

df_c = pd.read_csv('/content/drive/MyDrive/Projects/city_hour.csv',dtype = dtypes1)  

We are ready with our data! Let’s explore them!

Get Pooja’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The function ‘basic_exploration’ explorates the routine exploration done on any tabular data. Functionizing makes the task easier and saves a lot of time.

def space():
  print(" ")
  print("---------------------------------------------------------------")
  print(" ") 

def basic_exploration(steps):
  for i in steps:
    print(i)
    space()

steps = [df.shape, df.duplicated().any(), df.isnull().sum()]
basic_exploration(steps) 

(2589083, 16)
 
---------------------------------------------------------------
 
False
 
---------------------------------------------------------------
 
StationId           0
Datetime            0
PM2.5          647689
PM10          1119252
NO             553711
NO2            528973
NOx            490808
NH3           1236618
CO             499302
SO2            742737
O3             725973
Benzene        861579
Toluene       1042366
Xylene        2075104
AQI            570190
AQI_Bucket     570190
dtype: int64
 
--------------------------------------------------------------

The dataset has numerous missing values.

df_c.head(3)

Data Preprocessing

Convert to datetime

df_c['Datetime'] = pd.to_datetime(df_c['Datetime'])

df_c['AQI_Bucket'].unique() 

Label encode the ‘AQI_Bucket’ which is the target column for our classification model. Label Encoder assign integers to the classes.

le = preprocessing.LabelEncoder()                               
df_c['AQI_Bucket'] = le.fit_transform(df_c['AQI_Bucket'])   

Here we apply inverse transform function to the label encoder inorder to store the mappings done by the encoder.

mappings = {
    'nan': 6,
    'Poor': 2,
    'Moderate': 1,
    'Very Poor': 5,
    'Severe': 4,
    'Satisfactory': 3,
    'Good': 0
} 

Pattern of Missing Values

‘missingno’ is the python library which has functions for analysing the pattern of the missing values. According to the missing value pattern suitable imputation can be decided

msno.heatmap(df_c)

• From the above observation of distribution of missing values there is significant common pattern between the data columns. We observe that NO and NO2 are strongly correlated. As this is reading of air polluting gases, it can also be the case where industrial pollutants emit certain type of gases and vehicle pollutants have different set of gases. Hence there is correlation with missing values.

Imputation

• We apply forward imputation to the given dataset. Linear interpolation is an imputation technique that assumes a linear relationship between data points and utilizes non-missing values from adjacent data points to compute a value for a missing data point.

Detailed guide for choice of imputation techniques can be found here here.

• Choose the columns to interpolate the missing data.
• Apply bfill to initial values with ‘Nan’ as there isn’t any value prior to them to interpolate.

cols = ['PM2.5','PM10','NO','NO2', 'NOx','NH3','CO', 'SO2', 'O3', 'Benzene', 'Toluene', 'Xylene', 'AQI', 'AQI_Bucket'] 

df_c_imp = df_c[cols]
df_c_imp.interpolate(limit_direction='forward', inplace=True); 

df_c_1 = df_c[['City', 'Datetime']]
df_c_2 = pd.concat([df_c_1, df_c_imp], axis=1, join='inner')
df_c_2['AQI'].fillna(method='bfill', inplace=True)  

Extract the range of AQI_Bucket labels, ie, based on AQI (Air Quality Index) corresponding classes are assigned in AQI_bucket. Hence knowing the range of AQI is important to assign the corresponding classes to ‘Nan’ values in target label(AQI_Bucket). Here we:

• Write conditions
• Extract the upper limit of the range
• Use the upper limit to add the mappings

modc = df_c['AQI_Bucket'] == 1
satc = df_c['AQI_Bucket'] == 3
vpc = df_c['AQI_Bucket'] == 5             
prc = df_c['AQI_Bucket'] == 2
gdc = df_c['AQI_Bucket'] == 0                                                               
src = df_c['AQI_Bucket'] == 4 

severec = np.max(df_c[src]['AQI'])
very_poorc = np.max(df_c[vpc]['AQI'])
satisfactoryc = np.max(df_c[satc]['AQI'])
poorc = np.max(df_c[prc]['AQI'])
moderatec = np.max(df_c[modc]['AQI'])
goodc = np.max(df_c[gdc]['AQI'])    

print('maximum values for:')
print("severe {}\nvery poor {}\npoor {}\nmoderate {}\ngood{}\nsatisfactory {} ".format(seve

sr_cond = df_c_2['AQI'] > 400
vp_cond = df_c_2['AQI'] <=400
pr_cond = df_c_2['AQI'] <= 300
md_cond = df_c_2['AQI'] <= 200
st_cond = df_c_2['AQI'] <= 100
gd_cond = df_c_2['AQI'] <= 50

df_c_2.loc[sr_cond, 'AQI_Bucket'] = 4
df_c_2.loc[vp_cond, 'AQI_Bucket'] = 5
df_c_2.loc[pr_cond, 'AQI_Bucket'] = 2
df_c_2.loc[md_cond, 'AQI_Bucket'] = 1
df_c_2.loc[st_cond, 'AQI_Bucket'] = 3
df_c_2.loc[gd_cond, 'AQI_Bucket'] = 0

Check for Null and remove null

df_c_2.isnull().sum() 

df_c_2['PM2.5'] = df_c_2['PM2.5'].fillna(0)
df_c_2['O3'] = df_c_2['O3'].fillna(0)
df_c_2['NH3'] = df_c_2['NH3'].fillna(0)
df_c_2['PM10'] = df_c_2['PM10'].fillna(0)
df_c_2.isnull().sum() 

Save the preprocessed data!.

fp_c = '/content/drive/MyDrive/Projects/imp_city_hour.csv'
df_c_2.to_csv(fp_c, index=False)     

Exploratory Data Analysis

Distribution of classes in the Data

Countplot is a univariate plot ie. single variable under consideration, showing the number of data points representing the categorical variables.

def countplot(x_val):     
  plt.figure(figsize = (8, 6))          
  sns.countplot(x = x_val, palette = 'ocean_r');
  plt.xlabel("Good -0 ,Moderate-1 ,Poor-2 ,Satisfactory-3 ,Severe - 4, Very Poor -5")
countplot(df_ch['AQI_Bucket'])

• Most cities fall nder moderate and satisfactory range.
• There are few representation for good air quality and very poor air quality.

Cities vs AQI Indexes

"""Select city, AQI from df_ch ORDER BY AQI DESC"""
# do average of every city aqi value,                                                                                        
city_vs_AQI = df_ch[['City', 'AQI']].groupby(['City']).median().sort_values('AQI', ascending=False).reset_index()

def barplot(a,b, title, x_label, y_label, palet):
  plt.figure(figsize=(12,8))
  sns.barplot(x = a, y = b, palette =palet, order=a)
  plt.xticks(rotation = 90)
  plt.title(title, fontsize = 20)
  plt.xlabel(x_label, fontsize = 18)
  plt.ylabel(y_label, fontsize = 18)

barplot(city_vs_AQI['City'], city_vs_AQI['AQI'], "AQI Of Cities","Cities", "AQI","YlOrBr_r")

Ahmedabad has AQI in the range of 300 followed by Delhi. 300 indicates poor quality.

Gases and top 3 cities with maximum of that particular gas

Different cities are extracted which has maximum amount of gas type. Different gases have different cities in top 3.

cols = ['PM2.5','PM10','NO','NO2', 'NOx','NH3','CO', 'SO2', 'O3', 'Benzene', 'Toluene', 'Xylene', 'AQI', 'AQI_Bucket'] 

# Analyze the states with highest particular gases 
dff = []
for i in cols: 
  data = df_ch[[i, 'City']].groupby(['City']).median().sort_values(i, ascending = False).iloc[0:3].reset_index()
  dff.append(data) 

for i in range(len(cols)):
  print(dff[i])
  print("--------------------------")

…

Datetime Vs AQI

plt.plot(df_ch['Datetime'], df_ch['AQI'])

We observe that AQI is rising to severe in year 2018–2019 and slightly reduced in 2020 due to lockdown.

Analyzing Pollution in Delhi

Select data values corresponding to city= Delhi. Scatterplot is used for representation.

dli = df_ch[df_ch.City == 'Delhi'].reset_index() 

sns.set_style("whitegrid")

fig, axes = plt.subplots(nrows = 12,
                         ncols = 1,
                         figsize = (16,20)) 
axes[0].set_title('CO')
sns.scatterplot(x=dli['Datetime'], y =dli['CO'],data = df_ch,ax = axes[0], alpha = 1 )
axes[1].set_title('SO2')
sns.scatterplot(x=dli['Datetime'], y =dli['SO2'],data = df_ch, ax = axes[1], alpha = 1)

axes[2].set_title('PM2.5')
sns.scatterplot(x=dli['Datetime'], y =dli['PM2.5'],data = df_ch,ax = axes[2], alpha = 1 )
axes[3].set_title('NO2')
sns.scatterplot(x=dli['Datetime'], y =dli['NO2'],data = df_ch, ax = axes[3], alpha = 1)

axes[4].set_title('NH3')
sns.scatterplot(x=dli['Datetime'], y =dli['NH3'],data = df_ch,ax = axes[4], alpha = 1 )
axes[5].set_title('Benzene')
sns.scatterplot(x=dli['Datetime'], y =dli['Benzene'],data = df_ch, ax = axes[5], alpha = 1)

axes[6].set_title('Toluene')
sns.scatterplot(x=dli['Datetime'], y =dli['Toluene'],data = df_ch,ax = axes[6], alpha = 1 )
axes[7].set_title('O3')
sns.scatterplot(x=dli['Datetime'], y =dli['O3'],data = df_ch, ax = axes[7], alpha = 1)

axes[8].set_title('PM10')
sns.scatterplot(x=dli['Datetime'], y =dli['PM10'],data = df_ch,ax = axes[8], alpha = 1 )
axes[9].set_title('NOx')
sns.scatterplot(x=dli['Datetime'], y =dli['NOx'],data = df_ch, ax = axes[9], alpha = 1)

axes[10].set_title('Xylene')
sns.scatterplot(x=dli['Datetime'], y =dli['Xylene'],data = df_ch,ax = axes[10], alpha = 1 )
axes[11].set_title('NO')
sns.scatterplot(x=dli['Datetime'], y =dli['NO'],data = df_ch, ax = axes[11], alpha = 1)

We observe different pattern of density of gases from 2015 to 2020.

• There is rise of toluene
• Particulate matters, NOx NO2, have similar distributions.

Do explore further! In the next part we are upto statistical analysis.