K-Nearest Neighbors (KNN): Machine Learning Through Similarity

Introduction

Machine learning can sometimes look complicated, with complex algorithms and mathematical calculations.

But what if we could make a prediction simply by looking at similar examples around us?

That is exactly the idea behind K-Nearest Neighbors (KNN).

KNN is a simple machine learning algorithm that predicts the class or value of a new data point by looking at the most similar data points already available in the dataset.

It is also known as a Lazy Learning algorithm because it doesn’t build a traditional model during training. Instead, it stores the training data and performs most of the work when a prediction is required.

But why is it called K-Nearest Neighbors?

What exactly does it do?

How does it find the nearest neighbors?

Let’s understand it with a simple example.

How K-Nearest Neighbors Works

Imagine we have a graph containing two types of data points:

• Class A
• Class B

Now, we receive a new data point and want to determine whether it belongs to Class A or Class B.

KNN follows a few simple steps.

Step 1: Find the Distance

First, KNN calculates how far the new data point is from all the existing data points.

The basic idea is:

The closer two data points are, the more similar they are likely to be.

For example, imagine we want to classify a customer.

If a new customer has a similar age, income, and purchasing behavior to another customer, KNN considers them neighbors.

Step 2: Sort the Data Points

After calculating the distances, KNN arranges the data points from:

Closest → Farthest

Now we know which data points are the nearest to our new data point.

Step 3: Choose K

The K in KNN represents the number of neighbors we want to consider.

For example:

• K = 1 → Look at the closest 1 data point
• K = 3 → Look at the closest 3 data points
• K = 5 → Look at the closest 5 data points

Choosing the right K is important.

If K is too small, the prediction can be heavily influenced by a single unusual data point.

If K is too large, the algorithm may consider data points that are not actually very similar.

Usually, we try different values of K and choose the one that performs best on our data.

Step 4: Check the Neighbors

Let’s say we choose:

K = 5

Our five nearest neighbors are:

• Class A
• Class A
• Class B
• Class A
• Class B

Now we simply count them.

Class A → 3

Class B → 2

Class A has the majority.

Step 5: Make the Prediction

Since most of the nearest neighbors belong to Class A, KNN predicts that the new data point belongs to Class A.

So the complete process is:

New Data → Find Distance → Sort → Select K Neighbors → Check Majority → Prediction

That’s KNN!

Understanding Similarity and Distance

The most important concept in KNN is similarity.

But computers don’t understand similarity the way humans do.

So KNN uses distance to measure how similar two data points are.

Think about two people.

If two people have similar:

• Age
• Income
• Spending habits
• Location

we might consider them similar.

Similarly, KNN looks at the values of different features and determines which observations are closest to each other.

There are different ways to calculate distance.

The commonly used ones include:

• Euclidean Distance
• Manhattan Distance
• Minkowski Distance

You don’t always need to calculate these manually. Machine learning libraries such as Scikit-learn can handle the calculations for you.

The important idea to remember is:

Smaller distance generally means greater similarity.

KNN for Classification

KNN can be used for classification problems.

Classification means predicting a category or class.

For example:

• Spam or Not Spam
• Yes or No
• Disease or No Disease
• Cat or Dog
• Customer Will Churn or Won’t Churn

KNN looks at the nearest neighbors and chooses the class that appears most frequently.

Example

Suppose we want to determine whether a new email is spam.

We choose K = 5.

Among the five nearest emails:

• 4 are Spam
• 1 is Not Spam

Since most of the neighbors are Spam, KNN predicts:

Spam

Simple, right?

KNN for Regression

KNN can also be used for regression.

Regression means predicting a numerical value.

For example:

• House price
• Salary
• Temperature
• Sales
• Customer spending

Instead of selecting the most common class, KNN looks at the values of the nearest neighbors and typically takes their average.

Example

Suppose we want to predict the price of a house.

The three most similar houses have prices of:

• ₹50 lakh
• ₹55 lakh
• ₹60 lakh

KNN can use these nearby values to estimate the price of the new house.

The prediction would be around ₹55 lakh.

Get Vanganithindev’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

So:

Classification → Majority vote

Regression → Average of nearby values

Why Feature Scaling Is Important

There is one important thing we need to remember when using KNN:

Feature Scaling

Suppose our dataset contains:

Age: 20–60

Salary: ₹2 lakh–₹20 lakh

Salary contains much larger numbers than age.

Because KNN calculates distance, the salary feature could have a much greater influence on the result.

This can lead to incorrect predictions.

Therefore, we usually scale the features before applying KNN.

Common techniques include:

• Standardization
• Min-Max Scaling

The goal is to bring the features into comparable ranges so that one feature doesn’t unfairly dominate the distance calculation.

How Do We Choose the Right K?

Choosing K is an important part of using KNN.

Small K

A small K means the algorithm looks at only a few neighbors.

This can make the model sensitive to noise.

For example, if K = 1 and the closest point happens to be an unusual data point, the entire prediction could be affected.

Large K

A large K means the algorithm considers many neighbors.

This can make the model more stable, but it may also include data points that aren’t very similar to the new observation.

Therefore, we need to find a balance.

A common approach is to test different K values and use cross-validation to determine which one performs best.

Advantages of KNN

KNN has several advantages.

1. Easy to Understand

The basic idea is very simple:

Find similar data points and use them to make a prediction.

2. Easy to Implement

KNN can be implemented with relatively little code.

3. Works for Classification and Regression

The same basic algorithm can be used for both types of problems.

4. No Complex Training

KNN doesn’t spend a lot of time building a traditional model during training.

5. Can Handle Non-Linear Patterns

KNN doesn’t assume that the data must follow a particular straight-line relationship.

Limitations of KNN

KNN also has some disadvantages.

1. Slow for Large Datasets

When a new prediction is made, KNN may need to compare the new data point with many existing data points.

With a very large dataset, this can become slow.

2. Sensitive to Feature Scaling

If features have very different ranges, the distance calculation can become misleading.

3. Sensitive to Irrelevant Features

Unnecessary features can affect the distance calculation and reduce the quality of predictions.

4. Struggles With Too Many Features

KNN can become less effective when a dataset has a very large number of features.

This problem is commonly associated with the Curse of Dimensionality.

5. Choosing K Is Important

A poor choice of K can lead to poor predictions.

Real-World Applications of KNN

Even though KNN is a simple algorithm, its idea of finding similar examples can be useful in many areas.

1. Recommendation Systems

KNN can find users or products that are similar.

For example, if you have purchased several products, KNN can help identify other products that similar customers purchased.

2. Healthcare

KNN can help classify patients based on similarities in their characteristics and historical records.

3. Image Recognition

Images can be converted into numerical features, and KNN can compare a new image with previously labeled images.

4. Customer Analysis

Businesses can identify customers with similar purchasing behavior.

This can help with customer segmentation and targeted marketing.

5. Fraud Detection

KNN can compare a new transaction with previously observed transactions and identify whether it resembles suspicious activity.

6. Missing Value Imputation

KNN can also help fill missing values.

Instead of simply replacing a missing value with the mean or median, we can find similar records and use their values to estimate the missing information.

Why Is KNN Called a Lazy Learner?

Now we can understand why KNN is called a Lazy Learner.

During training, KNN doesn’t try to build a complicated mathematical model.

It mainly stores the available training data.

When a new data point arrives, it starts doing the actual work:

New Data Point

↓

Calculate Distances

↓

Find Nearest Neighbors

↓

Select K

↓

Make Prediction

So KNN is called “lazy” because most of its computation happens when we make predictions rather than during training.

A Simple Real-Life Example

Imagine you move into a new neighborhood.

You want to know whether people in your area prefer Tea or Coffee.

Instead of asking everyone in the neighborhood, you ask the five people who live closest to you.

You discover:

• Person 1 → Coffee
• Person 2 → Coffee
• Person 3 → Tea
• Person 4 → Coffee
• Person 5 → Tea

Three people prefer Coffee, while two prefer Tea.

Based on your nearest neighbors, you might predict:

Coffee

That is essentially the intuition behind KNN.

Look at the closest examples and use them to make a prediction.

Conclusion

K-Nearest Neighbors is one of the easiest machine learning algorithms to understand because its logic is similar to how humans often make decisions.

When we encounter something new, we often compare it with things we already know.

KNN does something similar.

It finds the nearest and most similar data points and uses them to predict the outcome.

For classification, it uses the majority class among the neighbors.

For regression, it uses the values of nearby observations to estimate a numerical result.

Although KNN has limitations, especially with large and high-dimensional datasets, it remains an excellent algorithm for beginners because it teaches one of the most important concepts in machine learning:

Similar data points can help us predict the outcome of a new data point.

And that simple idea is the heart of K-Nearest Neighbors.