A Deep Dive into Employee Attrition Patterns and Insights

Illustrations by Getty Images

Introduction

Employee attrition is the event when an employee leaves an organization for any reason and is not replaced for a long time, or not ever. It often results in a decrease in the size of an organization’s or department’s workforce because positions aren’t refilled when employees leave.

Attrition is unavoidable factors such as employee resignation for personal or professional reasons. For example, a car factory that used to need people to put on wheels and install windshields on the assembly line. But now, machines and robots do those jobs, so they don’t need as many assembly line workers. That is typical reason for employee attrition.

When attrition crosses a particular threshold, it becomes a cause for concern. Thats why, understanding the underlying factors contributing to employee attrition is crucial for organizations seeking to optimize their workforce, reduce operational costs, and implement effective retention strategies in an ever-evolving job landscape.

Data Understanding

To conduct a comprehensive analysis of employee attrition, I leveraged the rich and extensive IBM HR Employee Attrition and Performance dataset which provides a wealth of information about employee profiles, job roles, attrition outcomes, and etc.

Dataset: IBM HR Analytics Employee Attrition and Performance

This dataset consists of 1470 employee records, 35 columns, 25 numerical features, and 10 categorical features.

Exploratory Data Analysis (EDA)

1. Statistics Analysis

Descriptive Statistics Informative Table

In the first section, I explore the demographic and tenure aspects of organization’s workforce. The organization boasts a diverse age distribution, with an average employee age of 37 years. Employees typically spend an average of 7 years with the company before considering attrition.

Moving forward, I examine critical numerical features such as salary, job satisfaction, performance ratings, and work-life balance scores. The organization has a wide income spectrum, with an average salary of around 6,500 USD and a range spanning from 1,009 USD to 19,999 USD. Job satisfaction averages 2.7, while performance ratings average 3.15, and work-life balance scores average 2.76.

Additionally, I analyze workload, promotion cycles, and training opportunities, with employees dedicating 162 hours per month to their roles on average, experiencing promotions roughly every 2.2 years, and attending training sessions approximately every 2.8 years.

2. Univariate Analysis

In the given period, approximately 16.12% of the total workforce chose to leave the company, indicating a significant attrition rate. This warrants further investigation as high attrition rates can lead to increased recruitment and training costs, reduced morale among remaining employees, and potential loss of valuable talent.

The majority of employees are married, fall within the youth and adult age groups, and hold bachelor’s degrees. Understanding the demographics of your workforce is crucial for tailoring HR policies and employee engagement strategies. For instance, married employees might have different needs than single employees, and age can affect career development and mentorship programs.

The departments with the highest employee counts are R&D, and the most common job role is Sales Executive. Moreover, an alarming 28.30% of employees are working overtime, signifying a substantial portion of the workforce regularly working beyond standard hours. High overtime rates can result in burnout, decreased job satisfaction, and potential health issues for employees. It’s essential to investigate the reasons behind this high rate and consider workload distribution, process optimization, and employee well-being initiatives to mitigate burnout and improve work-life balance.

3. Bivariate Analysis

Employee Satisfaction Group

In this bivariate analysis focusing on employee satisfaction, I uncovered a striking insight: employees who report high levels of satisfaction, along with high involvement in their work and a better work-life balance, are more likely to experience attrition.

This revelation challenges conventional assumptions about employee attrition. Traditionally, it is expected that employees with higher satisfaction, engagement, and work-life balance would be less likely to leave an organization. However, our analysis suggests otherwise. This implies that the factors contributing to attrition are complex and multifaceted. While highly satisfied and engaged employees may indeed have positive sentiments towards their work, it is possible that other, possibly external, factors are motivating their decision to leave.

Employee Background Group

Next, bivariate analysis focusing on employee background group, I have identified specific demographic and educational characteristics that are associated with a higher likelihood of attrition. Employees who are male, approximately 40 years old, single, hold a bachelor’s degree, and come from a background in life sciences are more likely to experience attrition.

This analysis sheds light on potential patterns within the workforce that contribute to attrition. While these characteristics alone may not directly cause attrition, they serve as indicators of potential risk factors that warrant deeper exploration.

Employee History Group I

Next, bivariate analysis focusing on employee history group, several noteworthy patterns have emerged. Firstly, employees with long tenures in terms of total working years, years in their current role, and years at the same company are more likely to experience attrition, although it’s worth mentioning that most employees with total working experience exceeding 40 years have chosen to stay.

Employee History Group II

Additionally, employees with long-distance commutes to their workplace are more likely to leave, while those with higher incomes are more likely to stay. Furthermore, we observed that employees with the job role of laboratory technician are at a higher risk of attrition compared to other roles within the company. Moreover, employees at the junior job level and those with stock option values of 0 are more likely to experience attrition.

Lastly, it is noteworthy that employees with a history of overtime experience and a record of excellent performance are also more likely to experience attrition. These insights reveal the complexity of attrition factors and emphasize the importance of tailored retention strategies, including addressing work-life balance, compensation, job roles, and job levels, to ensure employee satisfaction and retention throughout the organization.

Features Correlation

Correlation Heatmap

My analysis revealed a strong correlation between employees’ total working years and career-related factors. This highlights the pivotal role of career experiences in job satisfaction and attrition. To leverage this insight, the organization should customize career development and mentorship programs as employees progress through their careers, ultimately boosting engagement, satisfaction, and reducing attrition.

Regression Model

I employed logistic regression analysis as a robust statistical tool to delve deeper into the factors contributing to employee attrition. This method allows me to identify and quantify the influence of various independent variables, such as demographic characteristics, job-related factors, and performance metrics, on the likelihood of attrition. By examining the odds ratios and statistical significance of these variables, I gain valuable insights into which factors are most strongly associated with attrition.

Get Taufik Budi Wibowo’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Build Logistic Regression Model

I began my analysis by conducting feature engineering and feature selection. These processes involved transforming the data and selecting the most relevant variables to build an effective model for predicting employee attrition.

#Encode categorical features
encode_cat_col = data.select_dtypes(exclude='number')

for col in encode_cat_col:
   data[col] = LabelEncoder().fit_transform(data[col])

#Select the features based on their correlation strength
correlation_matrix = data.corr()
correlation_threshold = 0.16  # Set your correlation threshold
attrition_correlations = correlation_matrix['attrition']

selected_features = attrition_correlations[abs(attrition_correlations) > correlation_threshold].drop('attrition').index.tolist()
print("Selected features based on correlation:", selected_features)

As I delved into the code, it became evident that a correlation threshold of 0.16 was meticulously chosen to determine the features included in my model. This threshold signifies that the selected features must exhibit a correlation with attrition above this critical value. And I got the features are marital status, over time, total working years, years in current role.

To comprehensively assess the significance of these results, I do print to the coefficient, so I can analyze it in a mathematical way.

P(Attrition) = logit^-1 (-2.07 + 0.64 marital_status + 1.44 over_time - 0.057 total_working_years - 0.089 years_in_current_role)

Here are key insights derived from the coefficient analysis,

For the intercept, I do ‘.expit’ to reverse log the intercept result. The intercept represents the probability of attrition when all predictors are set to zero. In this case (assuming this will be making sense) when all other factors are held constant, the log-odds of attrition are -2.07. Or we can say that my model predicts a probability of attrition of approximately 11.2%.

There’s a positive coefficient for marital status, suggesting that being married is associated with a higher likelihood of attrition. Specifically, for every unit increase in the marital status feature, the log-odds of attrition increase by 0.63, holding all other features constant. For the context of the probability scale, I divide it by 4: 0.63/4 = 0.1575. Therefore, being married, compared to being single (assuming all else is equal), corresponds to an approximate 15.75% increase in the probability of attrition.

The positive coefficient for overtime indicates that employees who work overtime are more likely to experience attrition. The log-odds of attrition increase by 1.44, holding other factors constant. For the context of probability, I divide it by 4: 1.44/4 = 0.36. So, when comparing employees who work overtime to those who don’t, assuming all factors are constant, the probability of attrition increases by approximately 36%.

The coefficient for total working years is negative, suggesting that as the total working years increase, the log-odds of attrition decrease slightly. However, the effect size is relatively small, with a decrease of 0.05 in log-odds for each year of working.

Similarly, to total working years, the negative coefficient for years in the current role implies that as the number of years in the current role increases, the log-odds of attrition decrease. This suggests that employees who stay in their current role for longer periods are less likely to leave.

Logistic Regression Model Evaluation

At the evaluation stage of the analysis, I sought to measure the performance of our model. It is important to evaluate the model to ensure its effectiveness in accurately predicting employee attrition. In my modeling process, I got the ‘Current Function Value’ of 0.379320 serves as a critical metric, representing the performance of my model with respect to the chosen optimization criterion. This value indicates how well my model aligns with its objective, with lower values generally signifying better performance or I can say it has small losses.

Log Score Each Model

In the Leave-One-Out (LOO) section, I observed that the performance of the LOO model was slightly lower than the previous logistic regression model, with a difference of 6 units. While this indicates that the LOO estimation model is not as well aligned with the data as the previous model, this deviation is still within the acceptable range. This suggests that the LOO model, while not a perfect model, is still a reasonable choice for attrition prediction.

Prediction Probability

Further analysis involved calculating the average log score of my model predictions across LOO iterations, resulting in a score of 0.809 or 0.81 when rounded. A higher average log score generally indicates better prediction performance. In this context, my score of 0.81 suggests that the predicted probabilities closely align with the actual outcomes, achieving an accuracy rate of 81%. This powerful insight underscores the model’s ability to provide valuable insights into attrition risk. It also emphasizes the potential for enhancing predictive accuracy through ongoing model refinement and data-driven interventions. By leveraging these insights, the organization can fine-tune its attrition prediction strategies and make more informed decisions to retain valuable talent effectively.

Conclusion

1. The organization has a diverse age group but needs to address wide income disparities, varying job satisfaction, and performance ratings to enhance employee retention and experience.
2. A significant attrition rate of approximately 16.12% requires a thorough investigation and proactive measures to reduce talent loss, while recognizing key employee demographics is crucial for effective HR policies.
3. Employee satisfaction doesn’t always prevent attrition, emphasizing the complexity of attrition factors, including demographics, education, and historical characteristics.
4. Recognizing the significant correlation between employees’ total working years and career-related factors underscores the importance of tailoring career development and mentorship programs to enhance engagement, satisfaction, and ultimately reduce attrition within the organization.
5. The logistic regression analysis indicates that marital status and overtime work significantly increase the likelihood of attrition, with married employees having approximately a 15.75% higher probability of attrition and those working overtime experiencing a roughly 36% increased risk, while total working years and years in the current role have relatively minor effects on reducing attrition, suggesting the need for focused retention strategies addressing these key factors.
6. the evaluation of our attrition prediction model demonstrates its effectiveness, with a ‘Current Function Value’ of 0.379320 indicating good alignment with the optimization criterion and a Leave-One-Out (LOO) model, although slightly less aligned, still within an acceptable range. The average log score of 0.81 underscores the model’s ability to accurately predict attrition with an 81% accuracy rate.

Recommendation

1. To enhance employee retention and satisfaction, it’s essential to address income disparities and tailor compensation structures to bridge gaps. Additionally, conduct regular surveys and feedback sessions to better understand and address varying job satisfaction levels among employees, ensuring that their concerns are actively acknowledged and resolved.
2. To reduce the high attrition rate of 16.12%, initiate a comprehensive investigation into the root causes of attrition. This should be followed by the development and implementation of proactive HR policies that take into account the diversity of employee demographics, emphasizing inclusivity and equal opportunities for career growth and development.
3. Acknowledge that employee satisfaction alone may not prevent attrition due to the multifaceted nature of attrition factors, including demographics, education, and historical characteristics. Therefore, create a holistic approach to address these factors, ensuring that HR strategies consider the unique needs and circumstances of individual employees.
4. Recognize the correlation between total working years and career-related factors as an opportunity to tailor career development and mentorship programs. By doing so, it can enhance employee engagement and satisfaction, ultimately reducing attrition within the organization.
5. Given the logistic regression analysis findings, implement specific measures to support married employees and mitigate their 15.75% higher attrition risk. Similarly, manage and minimize overtime work to reduce the approximately 36% increased risk associated with it. These targeted efforts will significantly contribute to attrition reduction.

Given the pressing project deadline and my current limited proficiency in regression models also knowledge about HR world, I am committed to dedicating additional time to enhance my knowledge and refine my skills in this domain. My ultimate objective is to develop a more precise predictive model in future endeavors.

References

• Pacmann Data Academy: https://pacmann.io
• AIHR | Employee Attrition
• Predict Employee Attrition by Yenlingktw
• HR Analytics by IronMan2483
• Employee Performance Analysis by AkhsayDusad

GitHub Link: Employee Attrition Analysis by Taufik Budi Wibowo

If there is any discussion, suggestions, or critiques, please feel free to chat through LinkedIn: Taufik Budi Wibowo.