A Project on Reject Inference

Summer Internship Project Documentation

For my summer internship, I had been tasked to work on a project from Banking Industry. The concept to be implemented, precisely, was reject inference.

Here is a chronological account of how I proceeded with the project :

1. The Problem

The most crucial step, is to first understand where you are headed with the problem in hand. On the same note, I had dataset of 10,001 applicants (who applied for a credit card) and were accepted and rejected basis a pre-determined score card.

The next thing to ponder over was, once we filter out these two branches, there were further splits available for both the parties on lines of whether they defaulted or if they did not.

In an ideal world and situation, the target variable should not be present for the rejected lot, since we never observed that applicants behavior's to flag them basis their default or no default status.

But one of the assumptions to validate the presence of this data in my case, is perhaps, the target variable for the rejected was obtained from the CIBIL scores or something likewise.

Hence, here is a visual representation of a flowchart to better illustrate the problem in hand. (Note that the values in the diagram are dummy values).

Use Case with Business Application

And lastly to boil down the objective in a few crisp lines:

While maintaining the same acceptance rate (appx 50%) , can I reduce the default rate (appx 15%), wherein I reshuffle the non-default cases from the rejected lot with those who defaulted in the accepted lot ??

2. Understanding the Data

The first and foremost thing was to understand the wide variety of datasets handed over to me.

In total there are 10 data sets. For simplicity We will divide them into the following:

Datasets grouped into broad categories

After analyzing the data with a hawk’s eye, I realized that only the following 4 datasets were important and unique, the others were a redundant product of these 4 and hence unnecessary.

Hence, we now focus our attention on:

1. pd_appl_abt (10001)

(probability_of_default_application_analytical_base_table)

2. beh_pd_abt (10001)

(behaviour_probability_of_default_analytical_base_table

3. ccf_abt (5252)

(ccf_analytical_base_table)

4. Defaults (10001)

Here the numbers in the brackets denote the number of rows present in the datasets.

The next thing I observed was, the target variable sitting in the applications and behavior dataset had similar mapping, which meant for same account, the target value accounted for a same objective.

After further deduction, I finally came to a conclusion that the target variable helped detect if an applicant defaulted or actually did the otherwise, i.e. , 1 denoted default and 0 denoted non-default.

Actually, honestly, this discovery was made after I was handed over a flowchart that looked somewhat like this:

The Flowchart

So from this flowchart, I had an epiphany that the ccf_abt (5252) dataset sitting in my favorite 4 datasets actually stood for those applicants who were approved.

Attagirl, now obviously the ones who were rejected were these 5252 reduced from the universe of 10,001.

So now, I have 4749 applicants who were rejected, and 5252 applicants who were accepted , which naturally implied 47.48% were rejected. This implied 52.53% was the actual acceptance rate.

Also, the current default rate for the entire 10,001 universe in 15.00%.

3. The Methodology

So now I have all the 4 datasets with me.

The data was split broadly in 2 sets : accepted (approved) and rejected.

From here ahead, I cleaned both the datasets by being mindful of the following:

1. Create a new column — status which denotes value 0 for accepted (approved) and 1 for rejected. This column is of importance in the concluding part of the codes.
2. Impute the NA values (mean for numeric and mode for categorical).
3. Convert the categorical variables to numeric.

Then, my next agenda was to build a ML algorithm on the approved dataset where I wanted my model to predict whether an applicant defaulted or not.

I fit a decision tree as it is one of the only few models that explains visually the reason as to why a model made a prediction that it did.

Another catch while fitting the model was , I constrained the min_samples_leaf to around 25 as to avoid overfitting (I will refer to this part again ahead).

One of the errors I made at the initial step was to include the behavioral variables along with my application variables, which indeed was a very foolish move to make, since it’s high correlation with the target variable gave me a somewhat 99% accuracy 😮!

Get Shraddha Kodavade’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Here is a decision tree that points out this huge blunder I had made and want to you not fall for:

99% accuracy trap!!!

Hence, I just focused on building an algorithm on a simple and sweet dataset that only consisted variables one would contribute during the application process.

Examples of variables you may ask? Gender, Marital Status, Number of Years Employed, Annual Income etc. are some of the demographics which are available when one applies for such facilities.

4. Checking for the Model Accuracy

Remember I was talking about overfitting the decision tree? Well, I wanted to plot the Gain and Lift charts for my model, but the predict_proba() was only predicting the probabilities as 0 and 1 for my X_test ! This clearly is a sign of overfitting and hence to minimize and loosen up the fit I used min_samples_leaf.

Due to this tweak in the code my the accuracy the model fetched was 85.9% (calculated on basis of confusion matrix).

Further, I plotted the Gain and Lift Chart which looked like this (for accepted / approved dataset) :

Gain Chart

Lift Chart

5. Inferring the rejected lot:

Now that the model was capturing a pattern from the accepted dataset, the next obvious thing was to predict the default and no default on the rejected dataset.

One of the brownie points of this dataset was, as I have already pointed out at the start of this article, is the presence of default and no default flags for the rejected dataset.

Therefore, I first predict the labels for rejected label (by removing the target variable column), and then combine the accepted dataset with the rejected dataset that now has a column on inferred defaults into one dataset.

Also the confusion matrix for what the model predicted for rejected lot and what the actual default situation is gave an accuracy of 84.5%.

Onto this combined dataset, I again implement a decision tree algorithm to see how my model performs, in an attempt to remove the bias of the algorithm created due to focusing only on the accepted dataset.

This too produced around 92.5% accuracy.

It is over here, that I noticed, the default rate for rejected is 8.3%.

Also I realised the number of defaults in rejected was too low. Hence, blaming the class imbalance in the initial phase of training itself, I tweaked my code by using class_weights = {0: 1, 1: 2} to give more importance to correctly classifiying the defaults in rejected. With this manipulation, the default rate for rejected was now 10.35%

To be honest, this is heartbreaking since the default rate for the rejected lot > = default rate for accepted lot to due obvious reasons.

6. Time to RESHUFFLE

Oh now comes the most awaited part, the very part where I know if I have met my objective. The re-shuffling part of this is something I have done on my own, none of literatures recommended it. It’s just to see for the same universe how things would have worked by using the following approach.

I combined the X_train and X_test to just X and similarly combined Y_train and Y_test to Y (from the model built on the combined dataset).

Then I predicted the probability of default for the entirety of X (which has 10,001 rows) and arranged them in descending order.

What I did next was split this sorted data to rejected and accepted (on basis of status column mentioned earlier).

Logically I am interested in retrieving the lot with low probability of default from rejected lot and shuffle them back to accepted, while also removing the one with high probability of default from accepted lot and sending it back to the rejected lot.

Hence I did the same, I checked the dimensions of the four data splits that is acc_def_0, acc_def_1, rej_def_0, rej_def_1.

I took the 30 % of the minimum of the rows of acc_def_1 and rej_def_0 (which was 777*0.3 in my case), and by shuffle I mean changed the value of status column (for rejected going to accepted it changed from 1 to 0 and vice versa).

Lastly I combined all the 4 splits (along with the data that was left of while picking the 777 rows).

The default rate of this new dataset came down to 10.2%!

Alas, I can say while maintaining the same acceptance rate, the default rate reduced from 14.75% to 10.2%. This is the particular dataset of sccepted lots only. For the combined dataset, the rate dropped from around 15% to approximately 12%.

Okay now I know this reshuffling was bound to drop the default rate, but let just celebrate this moment shall we?

That’s not to say there are many things still to be checked and rectified in order to ensure consistency while maintaining accuracy.

7. Score Card Development

Now, I am attempting something new here, I wanted to understand how scorecard models work.

I will try to keep this part concise. I relied on logistic regression to model my entire dataset to fetch the log(odds). Accordingly, log(odds) is essential in calculating the scores by using the following formula:

Credit Score=Offset + Factor*log(odds)

To get the value of Factor we use the following equation:

Factor=Probability of doubling odds (PDO) / ln(2)

The method of calculating PDO and Offset is another topic.

Since my acceptance rate was 52% initially and since I want to maintain the same rate, I chose the value a little higher than the median as my threshold to segregate the accepted from the rejected lot.

I also mapped the distribution of default onto it to understand how many defaulters of mine are getting eliminated with this approach.

This approach did not help me much since the values did not deviate impressively for default rate but nevertheless, this approach was also accounted for.

Since you have reached here, check out the codes for the same on the following link :

https://github.com/shraddhacodes/Reject_Inference

Credit Card Default- Reject Inference Project

More content at PlainEnglish.io.

Sign up for our free weekly newsletter. Follow us on Twitter, LinkedIn, YouTube, and Discord.