Probability Analysis
Introduction of Probability Project by Pacmann Academy
Background
One of the roles of a Data Analyst is the ability to extract simple insights or information from data. Sometimes, simply exploring the dataset can be more helpful for business decision-making than using complex models. Through this project, you will be asked to analyze the variables in the data you are interested in using basic probability principles. You may also include data visualizations when necessary. Provide a summary of the analysis you perform so that it can be a valuable tool for business decision-makers among the relevant stakeholders.
The data used in this project is health insurance data sourced from Kaggle. The dataset contains information on age, sex, BMI (Body Mass Index), the number of children, whether the individual is a smoker or not, their region of origin, and the amount of the insurance bill.
In this project, you are required to analyze the variables that are related to health insurance bills and provide insights into the characteristics of users who have high health insurance bills. This information can help the insurance company determine premium values for their policyholders.
Data Preparation
First, we will check the data based on null values, duplicates, data types, and unique values.
From the image above, the data no longer contains null and duplicate values. Next, we will check for outliers.
Handling Outliers
Outliers is detected. Then, we will do an approach to handle our outliers since it isn’t removing values (since it may be useful information).
capping comes in two approach:
- using IQR upper and lower bound
- using fixed quantile
for this task, we will use fixed quantile aproach.
Fixed quantile approach values as outliers if they are less than or more than the values for 97% of all the records in your dataset. In such cases you can use the quantile() method of the Pandas dataframe and pass it thresholds for lower and upper limits.
Get Saprilian Hidayat’s stories in your inbox
Join Medium for free to get updates from this writer.
For instance, a threshold value of 0.03 for the quantile() method will find the values that are less than 97% of all the records in the dataset. Where as a threshold value of 97% returns values that are greater than 97% of the records.
import numpy as np
import pandas as pd
# Function to remove outliers using capping method
def remove_outliers_capping(df, outlier_columns, lower_quantile=0.03, upper_quantile=0.97):
# Create a deep copy of the original dataframe
df_capped = df.copy()
for col in outlier_columns:
# Define lower and upper limits for capping
lwr_limit = df_capped[col].quantile(lower_quantile)
upr_limit = df_capped[col].quantile(upper_quantile)
print(str(col).upper())
print("lwr:", lwr_limit, "upr:", upr_limit)
print()
# Apply capping to the outliers
df_capped[col] = np.where(df_capped[col] > upr_limit, upr_limit,
np.where(df_capped[col] < lwr_limit, lwr_limit, df_capped[col]))
return df_cappedWe back to check the outliers after using fix quantile method.
Data Analysis
Case 1 — Descriptive Statistic Analysis
Let’s start this analysis process with the most basic step, which is summarizing the characteristics based on the data, such as finding the mean and data spread.
Based on the above analysis, the average age of patients is 39 years, with the average age of female and male smokers being 38 years. The average bill for smokers is significantly higher compared to non-smokers.
Case 2 — Variabel Discreat Analysis
Furthermore, to deepen the analysis, you can identify opportunities where specific conditions may have the potential to influence certain variable values.
In the above data, it contains people from the Southwest, Southeast, Northwest, and Northeast regions with a ratio of 27%, 24%, 24%, and 24% respectively. When comparing between women and men, the average bill for men is significantly higher at $9,434.76, and there is a higher tendency for men to smoke, at 58%.
Case 3 — Variabel Kontinue Analysis
Variables in the data we have may not all be discrete. To understand the potential for continuous variable conditions in the data, we can perform an analysis on those variables.
From the analysis above, it was found that individuals with a BMI above or below 25 receive health insurance bills above $16.7k, and smokers with a BMI above or below 25 also receive health insurance bills above $16.7k.
Case 4 — Variabel Correlation Variable
After addressing the more likely conditions from the previous steps, we can also explore the relationships between these conditions and the variables of interest. Correlation analysis will be necessary here.
Among the variables in the data, age and BMI have the highest correlation, which is 30% and 20%, respectively, compared to the other variables.
Case 5 — Hypothesis Testing
The final step is to determine whether there is enough statistical evidence to support the claims or hypotheses in our data.
From the above analysis, we can answer several hypotheses:
1. Health insurance bills for smokers are higher than those for non-smokers.
2. The proportion of male smokers is greater than that of female smokers.
3. Health insurance bills for individuals with a BMI above 25 are higher than those with a BMI below 25.
4. Health insurance bills for males are higher than those for females.
5. The likelihood of a male being a smoker is higher than that of a female.
