Encoding Categorical Features
Introduction
In machine learning projects, one important part is feature engineering. It is very common to see categorical features in a dataset. However, our machine learning algorithm can only read numerical values. It is essential to encoding categorical features into numerical values.
Here we will cover three different ways of encoding categorical features:
1. LabelEncoder and OneHotEncoder
2. DictVectorizer
3. Pandas get_dummies
For your convenience, the complete code can be found in my github.
Data Set
The data set we use here is from UCI Machine Learning Repository. It is used to predict whether a patient has kidney disease using various blood indicators as features. We use pandas to read the data in.
# load data
df = pd.read_csv(‘datasets/chronic_kidney_disease.csv’, header=None,
names=[‘age’, ‘bp’, ‘sg’, ‘al’, ‘su’, ‘rbc’, ‘pc’, ‘pcc’, ‘ba’, ‘bgr’, ‘bu’, ‘sc’, ‘sod’, ‘pot’,
‘hemo’, ‘pcv’, ‘wc’, ‘rc’, ‘htn’, ‘dm’, ‘cad’, ‘appet’, ‘pe’, ‘ane’, ‘class’])# head of df
df.head(10)
We can see from the table, we have several categorical features, such as ‘rbc’(red blood cells), ‘pc’(pus cell), ‘pcc’(pus cell clumps) and so on. It is obvious the dataset contains missing values, but since this is out scope of the topic in this blog, we’ll not cover how to do it here. But filling missing values is essential and should be done before encoding categorical features. You can refer to my github to see how to perform filling missing values. The completed dataset that ready for categorical encoding is shown as follows.
LabelEncoder & OneHotEncoder
The labelEncoder and OneHotEncoder only works on categorical features. We need first to extract the categorial featuers using boolean mask.
# Categorical boolean mask
categorical_feature_mask = X.dtypes==object# filter categorical columns using mask and turn it into a list
categorical_cols = X.columns[categorical_feature_mask].tolist()
LabelEncoder converts each class under specified feature to a numerical value. Let’s go through the steps to see how to do it.
Instantiate a LabelEncoder object:
# import labelencoder
from sklearn.preprocessing import LabelEncoder# instantiate labelencoder object
le = LabelEncoder()
Apply LabelEncoder on each of the categorical columns:
# apply le on categorical feature columns
X[categorical_cols] = X[categorical_cols].apply(lambda col: le.fit_transform(col))X[categorical_cols].head(10)
Note that the output of LabelEncoder is still a dataframe. The results is shown as following:
As we can see, all the categorical feature columns are binary class. But if the categorical feature is multi class, LabelEncoder will return different values for different classes. See the following example, the ‘Neighborhood’ feature has as many as 24 classes.
In this case, using LabelEncoder only is not a good choice, since it brings in a natural ordering for different classes. For example, under ‘Neighborhood’ feature, class_a has value 5 but class_b has value 24, is class_b ‘greater’ than class_a? The answer is obviously no. Thus allowing model learning this result will lead to poor performance. Therefore, for dataframe containing multi class features, a further step of OneHotEncoder is needed. Let’s see the steps to do it.
Get Yang Liu’s stories in your inbox
Join Medium for free to get updates from this writer.
Instantiate OneHotEncoder object:
# import OneHotEncoder
from sklearn.preprocessing import OneHotEncoder# instantiate OneHotEncoder
ohe = OneHotEncoder(categorical_features = categorical_feature_mask, sparse=False )
# categorical_features = boolean mask for categorical columns
# sparse = False output an array not sparse matrix
We need to specify categorical feature using its mask inside OneHotEncoder. The sparse=False argument outputs a non-sparse matrix.
Apply OneHotEncoder on DataFrame:
# apply OneHotEncoder on categorical feature columns
X_ohe = ohe.fit_transform(X) # It returns an numpy arrayNote that the output is a numpy array, not a dataframe. For each class under a categorical feature, a new column is created for it. For example, there are 20 columns created for the ten binary class categorical features.
DictVectorizer
As we can see, the LabelEncoder and OneHotEncoder usually need to be used together as two steps procedure. An more convenient way is using DictVectorizer which can achieve these two steps all at once.
First, we need to convert the dataframe into a dictionary. This can be achieved by Pandas to_dict method.
# turn X into dict
X_dict = X.to_dict(orient='records') # turn each row as key-value pairs# show X_dict
X_dict
The orient='records' is required to turn the data frame into a {column:value} format. The result is a list of dictionaries, among which each dictionary represent one sample. Note that, in this case we don’t need to extract the categorical features, we can convert the whole dataframe into a dict. This is one advantage compared to LabelEncoder and OneHotEncoder.
Now we instantiate a DictVectorizer:
# DictVectorizer
from sklearn.feature_extraction import DictVectorizer# instantiate a Dictvectorizer object for X
dv_X = DictVectorizer(sparse=False)
# sparse = False makes the output is not a sparse matrix
The sparse=False makes the output to be a non-sparse matrix.
DictVectorizer fit and transform on the converted dict:
# apply dv_X on X_dict
X_encoded = dv_X.fit_transform(X_dict)# show X_encoded
X_encoded
The result is a numpy array:
Each row represents a sample and each column represents a feature. If we want to know what feature for each column, we can check the vocabulary of this DictVectorizer:
# vocabulary
vocab = dv_X.vocabulary_# show vocab
vocab
Get Dummies
Pandas get_dummies method is a very straight forward one step procedure to get the dummy variables for categorical features. The advantage is you can directly apply it on the dataframe and the algorithm inside will recognize the categorical features and perform get dummies operation on it. Here is how to do it:
# Get dummies
X = pd.get_dummies(X, prefix_sep='_', drop_first=True)# X head
X.head()
The prefix_sep='_' makes each class has a unique name separated by the delimiter. The drop_first=True drops one column from the resulted dummy features. The purpose is to avoid multicollinearity. Here is the results:
Conclusion
LabelEncoder and OneHotEncoder is usually need to be used together as a two steps method to encode categorical features. LabelEncoder outputs a dataframe type while OneHotEncoder outputs a numpy array. OneHotEncoder has the option to output a sparse matrix. DictVectorizer is a one step method to encode and support sparse matrix output. Pandas get dummies method is so far the most straight forward and easiest way to encode categorical features. The output will remain dataframe type.
As my point of view, the first choice method will be pandas get dummies. But if the number of categorical features are huge, DictVectorizer will be a good choice as it supports sparse matrix output.
