Outlier
Outlier
Outliers are those data points that are significantly different from the rest of the dataset.
An outlier can distract the model.
For eg:
mean of age: 30, 45,50, 80, 3000. Here 3000 is an outlier.
mean = 641 age which is impossible. This is how the outlier effects model.
In this way, the performance of the model is reduced.
Sometimes outlier is also useful, especially for anomalous data like (email classification)
How to detect Outlier
1. Normal Distribution
(mean+3SD) >
(mean-3SD) <
2. Screwed Distribution
min(Q1–1.5*IQR) <
Max(Q3+1.5*IQR) >
3. Other Distribution
Using Percentile
How to treat Outliers?
1. Trimming:
If outliers are more than the data seems too thin. It will be much faster.
2. Capping:
It will make a limit between two end boundaries.
if max = 80 and outliers=90,85. Then outliers become 80
if min=5 and outliers=3,2,0. Then outliers become 5
Get Nabin Adhikari’s stories in your inbox
Join Medium for free to get updates from this writer.
Techniques for Outlier Detection and Removal
- Z-Score Treatment
Z-score treatment, also known as standardization or normalization, is a method for transforming numerical data so that it has a mean of 0 and a standard deviation of 1. This is done by subtracting the mean from each value and dividing the result by the standard deviation.
2. IQR-based Filtering
IQR-based filtering, also known as interquartile range (IQR) filtering, is a method for identifying and removing outliers in a dataset. It is based on the idea that most of the values in a dataset should be within the range of the first quartile (Q1) to the third quartile (Q3).
To perform IQR filtering, you first need to calculate the IQR of the data by subtracting the first quartile from the third quartile. Then, you can identify and remove the outliers by applying the following criteria:
- Values that are less than Q1–1.5 * IQR are considered lower outliers.
- Values that are greater than Q3 + 1.5 * IQR are considered upper outliers.
3. Percentile
A percentile is a measure of the relative standing of a value in a dataset. It represents the value below which a certain percentage of the values in the dataset fall.
For example, if the 50th percentile of a dataset is 10, that means that 50% of the values in the dataset are less than or equal to 10. The 50th percentile is also known as the median.
Percentiles can be useful for understanding the distribution of values in a dataset and for identifying values that are unusually high or low. They can also be used to perform winsorization, which is a method for handling outliers in a dataset.
To calculate percentiles in Python, you can use the quantile function from the pandas library. This function takes a Pandas Series or DataFrame and a percentile as input, and it returns the value at the specified percentile.
Here is an example of how you might use the quantile function to calculate the 50th percentile (median) of a Pandas Series:
Copy codeimport pandas as pd# Load the data
df = pd.read_csv('data.csv')# Calculate the 50th percentile (median)
median = df['value'].quantile(0.5)
In this example, the quantile function will return the median of the 'value' column in the DataFrame.
It is important to note that percentiles are sensitive to the size of the dataset, and they may not always accurately reflect the distribution of the values. In addition, the interpretation of percentiles may depend on the context of the analysis.
4. Winsorization(percentile after Capping)
Winsorization, also known as “capping” or “trimming,” is a method for handling outliers in a dataset. It involves replacing extreme values in the dataset with less extreme values, in order to reduce the influence of outliers on the statistical properties of the data.
There are two main types of winsorization: single-sided winsorization and double-sided winsorization. Single-sided winsorization replaces only the values that are above or below a certain percentile with the value at that percentile. Double-sided winsorization replaces both the highest and lowest values with the values at a certain percentile.
