Future Engineering-Outliers (aykırı değer)
We refer to the values that significantly deviate from the general trend in the data as outliers.
So, what is this general trend?
As we can see from the graph, the points where values are densely clustered around certain ranges are called the general trend. On the contrary, the points that deviate significantly from these clusters are considered outliers. Outliers have a considerable impact, especially on linear problems. However, their effects are quite low in tree-based methods.
To determine outlier values, a critical step is to set a threshold value. In other words, what is the maximum value that can be taken and accepted? We consider values above and below this threshold as outliers. The most commonly used method for identifying outliers is the box plot method.
Let’s discuss based on the image.
According to this method:
The line at the center of the box represents the median value. The area inside the box is where the majority of the values lie. Imagine sorting the variables from smallest to largest. The middle value in this sorted list is the IQR value, which is the difference between Q3 and Q1. For the upper bound, we subtract 1.5 times the IQR value from Q1. For the lower bound, we add 1.5 times the IQR value to Q3. Values above and below these bounds are considered outliers.
DETECTING OUTLIERS
def detecting_outliers(data_drame, col_name, q1=0.25, q2=0.75):
quartile1 = dataframe[col_name].quantile(q1)
quartile3 = dataframe[col_name].quantile(q3)
iqr= quartile3 — quartile1
up_limit= quartile3 + 1.5 * iqr
low_limit=quartile1–1.5*iqr
return low_limit, up_limit
Get Rabia Budak’s stories in your inbox
Join Medium for free to get updates from this writer.
def has_outliers(dataframe, col_name):
low_limit, up_limit= detecting_outliers(dataframe, col_name):
if dataframe[(dataframe[col_name]> up_limit) | (dataframe[col_name]<low_limit)].any(axis=None):
return true
else:
return false
After functionally transforming the dataset, how should I select numerical variables?
Let there be a function that provides the names of categorical, numerical, and categorical but cardinal variables in the dataset.
NOTE: Categorical variables should include numerically represented categorical variables as well. Can you give an example of a categorical variable? For instance, gender is a categorical variable. Categorical variables have as many unique values as there are observations in the dataset.
def grap_col_names(dataframe, cat_th=10, car_th=20): # Estimates may vary depending on the dataset.
cat_cols=[col for col in dataframe.columns if dtaframe[col].dtypes == “O”]
num_but_cat= [col for col in dataframe.columns if dataframe[col].nunique() < cat_th and dataframe[col].dtypes != “O”]
num_but_car = [col for col in dataframe.columns if dataframe[col].nunique() > car_th and dataframe[col].dtypes == “O”]
cat_cols = cat_cols + num_but_cat
cat_cols = [col for col in cat_cols if col not in cat_but_car]
num_cols = [col for col in dataframe.columns if dataframe[col].dtypes != “O”]
num_cols = [ col for col in num_cols if col not in num_but_cat]
return cat_cols, num_cols, cat_but_car
