Text Classification

Using K Nearest Neighbor

Brief

K nearest neighbor (k-NN) algorithm is a non parametric supervised machine learning algorithm used for classification and regression. For the context of text classification, the object is classified based on vote among the k nearest neighbors. The class with the maximum neighbor vote is assigned as the class of the object. A vote is provided to k closest neighbor which is calculated using the distance measures such as Euclidean distance, Minkowski, et cetera.

k-NN is a type of instance-based learning, or lazy learning, where the generalization of the training dataset is deferred until evaluation is done. This makes training the classifier faster but testing slower and costlier (in terms of time and memory).

Steps

1. Get labeled data : The labeled data consists of features and labels. Features are the characteristics or the property of the object whereas labels are the class of the object with those features.
2. Convert labeled data to encoded data : Usually computations are based on numerical form so we convert the data to numeric form by encoding them.
3. Create feature set : Creating a set of features by packing the features.
4. Split the data for train and test : The data are split training and testing. Usually, 80% for training and 20% for testing but can select based on need.
5. Train the classifier : Training the classifier with the training data by specifying the value of k. Use k =3 for binary classification, i.e. two labels classification. If used k =1 then it is simply a nearest neighbor classifier.
6. Test the classifier : Testing the classifier with the testing data.
7. Evaluate : Evaluating the classifier using confusion matrix and its evaluation metrics i.e. accuracy, precision, recall, et cetera.

Implementation

Requirements

Python, Scikit-learn, Numpy

Code

Get Ashin Shakya’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Get labeled data

# First Feature
weather =['Sunny','Sunny','Overcast','Rainy','Rainy','Rainy','Overcast','Sunny','Sunny','Rainy','Sunny','Overcast','Overcast','Rainy']
# Second Feature
temp =['Hot','Hot','Hot','Mild','Cool','Cool','Cool','Mild','Cool','Mild','Mild','Mild','Hot','Mild']
# Label or target variable
play =['No','No','Yes','Yes','Yes','No','Yes','No','Yes','Yes','Yes','Yes','Yes','No']

Convert labeled data to encoded data

from sklearn import preprocessing
le = preprocessing.LabelEncoder()

weather_encoded = le.fit_transform(weather)
print('Weather Encoded',weather_encoded)

temp_encoded = le.fit_transform(temp)
print('Temperature Encoded',temp_encoded)

label=le.fit_transform(play)
print('Label Encoded',label)

Create feature set

features = list(zip(weather_encoded,temp_encoded))
print('Features',features)

Split the data for train and test

from sklearn.model_selection import train_test_split
# Splitting train : test to 70 : 30 ratio
X_train, X_test, y_train, y_test = train_test_split(features,label,test_size=0.3)

Train the classifier

from sklearn.neighbors import KNeighborsClassifier
# Applying k = 3, default Minkowski distance metrics
model = KNeighborsClassifier(n_neighbors=3)
# Training the classifier
model.fit(X_train,y_train)

Test the classifier

# Testing the classifier
y_pred = model.predict(X_test)
print('Predicted',y_pred)
print('Actual data',y_test)

Evaluate

from sklearn.metrics import accuracy_score
accuracy = accuracy_score(y_test, y_pred)
print('Accuracy',accuracy)

Full Code Implementation

Get the full code here.

Advantages

a. Simple and easy to implement.
b. Training phase is faster due to lazy learning.
c. Suitable for multi class problem.
d. Works better for continuously changing data due to instance based learning.

Disadvantages

a. Testing phase is slower and costlier in terms of time and memory.
b. Highly sensitive to outliers.
c. Curse of dimensionality i.e. if the number of dimensions (features) increases the required number of data must increases exponentially.