<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Praveen Kotha on Medium]]></title>
        <description><![CDATA[Stories by Praveen Kotha on Medium]]></description>
        <link>https://medium.com/@praveenkotha?source=rss-681493d7f9d4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*YKh7TMmddVvC0Ej6.</url>
            <title>Stories by Praveen Kotha on Medium</title>
            <link>https://medium.com/@praveenkotha?source=rss-681493d7f9d4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 07:29:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@praveenkotha/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Home Credit Default Risk- End to End Machine learning project.]]></title>
            <link>https://medium.com/@praveenkotha/home-credit-default-risk-end-to-end-machine-learning-project-1871f52e3ef2?source=rss-681493d7f9d4------2</link>
            <guid isPermaLink="false">https://medium.com/p/1871f52e3ef2</guid>
            <category><![CDATA[loans]]></category>
            <category><![CDATA[plotly]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[homecredit]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Praveen Kotha]]></dc:creator>
            <pubDate>Sun, 05 May 2019 07:55:41 GMT</pubDate>
            <atom:updated>2019-09-27T02:01:10.582Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>“Computers are able to see, hear and learn. Welcome to the future”</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*4vQya2amGNjdEXfeAdts_A.jpeg" /></figure><h3>Introduction &amp; Overview of the problem:</h3><p>An existential problem for any Loan providers today is to find out the Loan applicants who are very likely to repay the loan. This way companies can avoid losses and incur huge profits.</p><p>Home Credit offers easy, simple and fast loans for a range of Home Appliances, Mobile Phones, Laptops, Two Wheeler&#39;s , and varied personal needs. Home Credit comes up with a Kaggle challenge to find out the loan applicants who is capable of repaying a loan, given the applicant data, all credits data from Credit Bureau, previous applications data from Home Credit and some more data.</p><h4><strong>Problem statement</strong>:</h4><p>Building a model to predict how capable each applicant is of repaying a loan, so that sanctioning loan only for the applicants who are likely to repay the loan.</p><h4>Real World / Business Objectives and Constraints:</h4><p>Before starting any problem, it is better to lay the constraints. So that we can change the modelling process based on the constraints.</p><p>1. No strict latency constraints.</p><p>Given the loan application data, we don’t have to predict whether the applicant is going to repay loan in milli seconds or seconds. We can take couple of minutes to predict. Even 1 hour should be fine. Since we don’t have strict latency constraint, we can use Ensemble models like Random forest, Xgboost etc.</p><p>2. Predict the probability of capability of each applicant of repaying a loan.</p><p>Suppose there are two applicants A, B whose probability values of repaying loan is 0.6 and 0.9, since the general threshold each machine learning model considers is 0.5. Both the applicants are labeled as ‘Repaying the loan’. But applicant B is more likely to repay the loan when compared to applicant A. If we predict the probabilities we can set the threshold like 0.8, 0.9 based on the business requirements.</p><p>3. The cost of a mis-classification is very high.</p><p>Let’s say, for an applicant A our model labelled as ‘Repaying the loan’. So the organization sanctioned the loan for that applicant. But the applicant for some reason is not able to pay the loan. This type of scenarios is loss to the organization. Hence we should come up with a model which can reduce the mis-classifications as much as possible.</p><p>4. Interpretability is partially important.</p><p>As long as our model predicts well on test data, we don’t need to care much about interpretability. Since our problem is not related to medical domain(Interpretability is important- cancer detection), it is fine if we can give some form of interpretability like feature importances.</p><h4>Performance Metric:</h4><p>In this problem, the data is imbalanced. So we can’t use accuracy as a error metric. When data is imbalanced we can use Log loss, F1-score and AUC. Here we are sticking to AUC which can handle imbalanced datasets.</p><ul><li>Area Under Curve (AUC)</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FOAl6eAyP-yo%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DOAl6eAyP-yo&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FOAl6eAyP-yo%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/aab8db2875729c77bc244704bd6f9547/href">https://medium.com/media/aab8db2875729c77bc244704bd6f9547/href</a></iframe><p>An ROC curve is the most commonly used way to visualize the performance of a binary classifier, and AUC is (arguably) the best way to summarize its performance in a single number.</p><ul><li>Confusion Matrix (To get an overview of complete predictions)</li></ul><h4>Data Description and Overview:</h4><p>The data is provided by Home Credit, a service dedicated to provided lines of credit (loans) to the unbanked population.</p><p>There are 7 different sources of data:</p><ul><li><strong>application_train/application_test</strong>: The main training data with information about each loan application at Home Credit. Every loan has its own row and is identified by the feature SK_ID_CURR. The training application data comes with the TARGET indicating 0: the loan was repaid or 1: the loan was not repaid. Here we will use only the Training data.</li><li><strong>bureau</strong>: In this dataset it consists of data concerning client’s previous credits from other financial institutions. Each previous credit has its own row in bureau, but one loan in the application data can have multiple previous credits.</li><li><strong>bureau_balance</strong>: It consists of monthly data about the previous credits in bureau. Each row is one month of a previous credit, and a single previous credit can have multiple rows, one for each month of the credit length.</li><li><strong>previous_application</strong>:The data of previous applications for loans at Home Credit of clients who have loans in the application data. Each current loan in the application data can have multiple previous loans. Each previous application has one row and is identified by the feature SK_ID_PREV.</li><li><strong>POS_CASH_BALANCE</strong>: It consists of monthly data about previous point of sale or cash loans clients have had with Home Credit. Each row is one month of a previous point of sale or cash loan, and a single previous loan can have many rows.</li><li><strong>credit_card_balance</strong>:The monthly data about previous credit cards clients have had with Home Credit. Each row is one month of a credit card balance, and a single credit card can have many rows.</li><li><strong>installments_payment</strong>:The data of payment history for previous loans at Home Credit. There is one row for every made payment and one row for every missed payment.</li></ul><p>The below diagram shows how the data is related:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s6uRljTMIgyBCD1ok4gfrg.png" /><figcaption>Data Overview</figcaption></figure><h4>Mapping to an Machine Learning Problem:</h4><p>For the given applicant data we need to predict if they are capable to repay the loan or not. Since we have two classes in the target label, we will pose this problem as a <strong>Binary classification</strong> problem.</p><p><strong>Data Source:<br></strong>We can get the complete data required for this problem from the link : <a href="https://www.kaggle.com/c/home-credit-default-risk/data">https://www.kaggle.com/c/home-credit-default-risk/data</a></p><h3>Exploratory Data Analysis:</h3><p>At first, import the necessary packages.</p><pre>import pandas as pd<br>import sklearn<br>import numpy as np<br>import matplotlib.pyplot as plt<br>import os<br>import warnings<br>import seaborn as sns</pre><pre>from sklearn.preprocessing import OneHotEncoder<br>from sklearn.impute import SimpleImputer<br>from sklearn.pipeline import Pipeline<br>from sklearn.compose import ColumnTransformer<br>from sklearn.preprocessing import StandardScaler<br>from sklearn.svm import LinearSVC<br>from sklearn.metrics import roc_auc_score<br>from sklearn.linear_model import LogisticRegression<br>from sklearn.metrics import roc_auc_score<br>from sklearn.calibration import CalibratedClassifierCV<br>from sklearn.metrics import confusion_matrix<br>from sklearn.ensemble import RandomForestClassifier<br>from sklearn.metrics import accuracy_score<br>from sklearn.linear_model import SGDClassifier</pre><pre>import plotly.offline as py<br>import plotly.graph_objs as go</pre><pre>from plotly.offline import init_notebook_mode, iplot<br>from sklearn.model_selection import train_test_split<br>init_notebook_mode(connected=True)</pre><pre>import cufflinks as cf<br>cf.go_offline()</pre><pre>import pickle<br>import gc<br>import lightgbm as lgb</pre><pre>warnings.filterwarnings(&#39;ignore&#39;)<br>%matplotlib inline</pre><ul><li><strong>Load the data from given csv file into a pandas dataframe.</strong></li></ul><pre>print(&#39;Reading the data....&#39;, end=&#39;&#39;)<br>application = pd.read_csv(&#39;application_train.csv&#39;)<br>print(&#39;done!!!&#39;)<br>print(&#39;The shape of data:&#39;,application.shape)<br>print(&#39;First 5 rows of data:&#39;)<br>application.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UZ9KFcim8E8X-4-v5bQUgQ.png" /></figure><p>We are using ‘application_train.csv’ file :<br>1. This dataset consists of 307511 rows and 122 columns.<br>2. Each row has unique id ‘SK_ID_CURR’ and the output label is in the ‘TARGET’ column.<br>3. TARGET indicating 0: the loan was repaid or 1: the loan was not repaid.<br>4. The description of each column can be found in the file ‘HomeCredit_columns_description.csv’</p><ul><li><strong>Let us check for missing values in each column.</strong></li></ul><pre>count = application.isnull().sum().sort_values(ascending=False)<br>percentage = ((application.isnull().sum()/len(application)*100)).sort_values(ascending=False)</pre><pre>missing_application = pd.concat([count, percentage], axis=1, keys=[&#39;Count&#39;,&#39;Percentage&#39;])<br>print(&#39;Count and percentage of missing values for top 20 columns:&#39;)<br>missing_application.head(20)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vHIjhKaYamG1wvfL7jGRLw.png" /></figure><p><em>Observations</em>:<br>1. There are lot of missing values in each column.<br>2. We need to somehow handle these missing values, we will see how to handle later in the case study.</p><ul><li><strong>Let’s check for duplicate data:</strong></li></ul><pre>columns_without_id = [col for col in application.columns if col!=&#39;SK_ID_CURR&#39;]</pre><pre>#Checking for duplicates in the data.<br>application[application.duplicated(subset = columns_without_id, keep=False)]</pre><pre>print(&#39;The no of duplicates in the data:&#39;,application[application.duplicated(subset = columns_without_id, keep=False)]<br>      .shape[0])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KWtTLowKLd9fSvBBVLkSag.png" /></figure><ul><li><strong>Let’s check the distribution of data points among output class.</strong></li></ul><pre>cf.set_config_file(theme=&#39;polar&#39;)<br>contract_val = application[&#39;NAME_CONTRACT_TYPE&#39;].value_counts()<br>contract_df = pd.DataFrame({&#39;labels&#39;: contract_val.index,<br>                   &#39;values&#39;: contract_val.values<br>                  })<br>contract_df.iplot(kind=&#39;pie&#39;,labels=&#39;labels&#39;,values=&#39;values&#39;, title=&#39;Types of Loan&#39;, hole = 0.6)</pre><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7EPraveenKotha%2F61%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7EPraveenKotha%2F61.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/3a20a3a6d9651e74d1336bbf1dd0c6d9/href">https://medium.com/media/3a20a3a6d9651e74d1336bbf1dd0c6d9/href</a></iframe><p><em>Observations</em><strong>:</strong><br>1. The data is imbalanced (91.9%(Loan repayed-0) and 8.07%(Loan not repayed-1)) and we need to handle this problem.</p><ul><li><strong>Distribution of AMT_INCOME_TOTAL.</strong></li></ul><pre>application[application[&#39;AMT_INCOME_TOTAL&#39;] &lt; 2000000][&#39;AMT_INCOME_TOTAL&#39;].iplot(kind=&#39;histogram&#39;, bins=100,<br>   xTitle = &#39;Total Income&#39;, yTitle =&#39;Count of applicants&#39;,<br>             title=&#39;Distribution of AMT_INCOME_TOTAL&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6H_d_QbfTZI2sPuPX6sY_A.png" /></figure><pre>(application[application[&#39;AMT_INCOME_TOTAL&#39;] &gt; 1000000][&#39;TARGET&#39;].value_counts())/len(application[application[&#39;AMT_INCOME_TOTAL&#39;] &gt; 1000000])*100</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gIeZXkGCgRG5Juilw3daAQ.png" /></figure><p><em>Observations</em>:<br>1. The distribution is right skewed and there are extreme values, we can apply log distribution.<br>2. People with high income(&gt;1000000) are likely to repay the loan.</p><ul><li><strong>Types of loan available.</strong></li></ul><pre>cf.set_config_file(theme=&#39;polar&#39;)<br>contract_val = application[&#39;NAME_CONTRACT_TYPE&#39;].value_counts()<br>contract_df = pd.DataFrame({&#39;labels&#39;: contract_val.index,<br>                   &#39;values&#39;: contract_val.values<br>                  })<br>contract_df.iplot(kind=&#39;pie&#39;,labels=&#39;labels&#39;,values=&#39;values&#39;, title=&#39;Types of Loan&#39;, hole = 0.6)</pre><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7EPraveenKotha%2F63%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7EPraveenKotha%2F63.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/00e92e483054e570548e8d4bf199390a/href">https://medium.com/media/00e92e483054e570548e8d4bf199390a/href</a></iframe><p><em>Observations</em>:<br>1. Many people are willing to take cash loan than revolving loan (<a href="https://www.investopedia.com/terms/r/revolving-loan-facility.asp">https://www.investopedia.com/terms/r/revolving-loan-facility.asp</a>).</p><ul><li><strong>Distribution of AMT_CREDIT</strong></li></ul><pre>application[&#39;AMT_CREDIT&#39;].iplot(kind=&#39;histogram&#39;, bins=100,<br>            xTitle = &#39;Credit Amount&#39;,yTitle =&#39;Count of applicants&#39;,<br>            title=&#39;Distribution of AMT_CREDIT&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bFYJBq38qF93-3_gQIvK-w.png" /></figure><pre>np.log(application[&#39;AMT_CREDIT&#39;]).iplot(kind=&#39;histogram&#39;, bins=100,<br>        xTitle = &#39;log(Credit Amount)&#39;,yTitle =&#39;Count of applicants&#39;,<br>        title=&#39;Distribution of log(AMT_CREDIT)&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V-8N-U7dNwT6v2pnAAvn7g.png" /></figure><p><em>Observations:<br></em>1. People who are taking credit for large amount are very likely to repay the loan.<br>2. Originally the distribution is right skewed, we used log transformation to make it normal distributed.</p><ul><li><strong>Distribution of Name of type of the Suite in terms of loan is repayed or not.</strong></li></ul><pre>cf.set_config_file(theme=&#39;polar&#39;)<br>suite_val = (application[&#39;NAME_TYPE_SUITE&#39;].value_counts()/len(application))*100<br>suite_val.iplot(kind=&#39;bar&#39;, xTitle = &#39;Name of type of the Suite&#39;,<br>             yTitle=&#39;Count of applicants in %&#39;,<br>             title=&#39;Who accompanied client when applying for the  application in % &#39;)</pre><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7EPraveenKotha%2F65%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7EPraveenKotha%2F65.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/7a52026fb500069916592df710e2905b/href">https://medium.com/media/7a52026fb500069916592df710e2905b/href</a></iframe><pre>suite_val = application[&#39;NAME_TYPE_SUITE&#39;].value_counts()</pre><pre>suite_val_y0 = []<br>suite_val_y1 = []<br>for val in suite_val.index:<br>    suite_val_y1.append(np.sum(application[&#39;TARGET&#39;][application[&#39;NAME_TYPE_SUITE&#39;]==val] == 1))<br>    suite_val_y0.append(np.sum(application[&#39;TARGET&#39;][application[&#39;NAME_TYPE_SUITE&#39;]==val] == 0))</pre><pre>data = [go.Bar(x = suite_val.index, y = ((suite_val_y1 / suite_val.sum()) * 100), name=&#39;Yes&#39; ),<br>        go.Bar(x = suite_val.index, y = ((suite_val_y0 / suite_val.sum()) * 100), name=&#39;No&#39; )]</pre><pre>layout = go.Layout(<br>    title = &quot;Who accompanied client when applying for the  application in terms of loan is repayed or not in %&quot;,<br>    xaxis=dict(<br>        title=&#39;Name of type of the Suite&#39;,<br>       ),<br>    yaxis=dict(<br>        title=&#39;Count of applicants in %&#39;,<br>        )<br>)</pre><pre>fig = go.Figure(data = data, layout=layout) <br>fig.layout.template = &#39;plotly_dark&#39;</pre><pre>py.iplot(fig)</pre><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7EPraveenKotha%2F67%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7EPraveenKotha%2F67.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/abf90ea067fb8e815d97f0327b4b424b/href">https://medium.com/media/abf90ea067fb8e815d97f0327b4b424b/href</a></iframe><ul><li><strong>Distribution of Income sources of Applicants in terms of loan is repayed or not.</strong></li></ul><pre>income_val = application[&#39;NAME_INCOME_TYPE&#39;].value_counts()</pre><pre>income_val_y0 = []<br>income_val_y1 = []<br>for val in income_val.index:<br>    income_val_y1.append(np.sum(application[&#39;TARGET&#39;][application[&#39;NAME_INCOME_TYPE&#39;]==val] == 1))<br>    income_val_y0.append(np.sum(application[&#39;TARGET&#39;][application[&#39;NAME_INCOME_TYPE&#39;]==val] == 0))</pre><pre>data = [go.Bar(x = income_val.index, y = ((income_val_y1 / income_val.sum()) * 100), name=&#39;Yes&#39; ),<br>        go.Bar(x = income_val.index, y = ((income_val_y0 / income_val.sum()) * 100), name=&#39;No&#39; )]</pre><pre>layout = go.Layout(<br>    title = &quot;Income sources of Applicants in terms of loan is repayed or not  in %&quot;,<br>    xaxis=dict(<br>        title=&#39;Income source&#39;,<br>       ),<br>    yaxis=dict(<br>        title=&#39;Count of applicants in %&#39;,<br>        )<br>)</pre><pre>fig = go.Figure(data = data, layout=layout) <br>fig.layout.template = &#39;plotly_dark&#39;</pre><pre>py.iplot(fig)</pre><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7EPraveenKotha%2F69%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7EPraveenKotha%2F69.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/a02a2d585fc88c4d7a92cbdb9acaf441/href">https://medium.com/media/a02a2d585fc88c4d7a92cbdb9acaf441/href</a></iframe><p><em>Observations:<br></em>1. All the Students and Businessman are repaying loan.(Open the chart and hover over the plot to observe).</p><ul><li><strong>Distribution of Education of Applicants in terms of loan is repayed or not.</strong></li></ul><pre>education_val = application[&#39;NAME_EDUCATION_TYPE&#39;].value_counts()</pre><pre>education_val_y0 = []<br>education_val_y1 = []<br>for val in education_val.index:<br>    education_val_y1.append(np.sum(application[&#39;TARGET&#39;][application[&#39;NAME_EDUCATION_TYPE&#39;]==val] == 1))<br>    education_val_y0.append(np.sum(application[&#39;TARGET&#39;][application[&#39;NAME_EDUCATION_TYPE&#39;]==val] == 0))</pre><pre>data = [go.Bar(x = education_val.index, y = ((education_val_y1 / education_val.sum()) * 100), name=&#39;Yes&#39; ),<br>        go.Bar(x = education_val.index, y = ((education_val_y0 / education_val.sum()) * 100), name=&#39;No&#39; )]</pre><pre>layout = go.Layout(<br>    title = &quot;Education sources of Applicants in terms of loan is repayed or not  in %&quot;,<br>    xaxis=dict(<br>        title=&#39;Education of Applicants&#39;,<br>       ),<br>    yaxis=dict(<br>        title=&#39;Count of applicants in %&#39;,<br>        )<br>)</pre><pre>fig = go.Figure(data = data, layout=layout) <br>fig.layout.template = &#39;plotly_dark&#39;</pre><pre>py.iplot(fig)</pre><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7EPraveenKotha%2F71%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7EPraveenKotha%2F71.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/b7f11a6b251c2b0d4e130e976991283a/href">https://medium.com/media/b7f11a6b251c2b0d4e130e976991283a/href</a></iframe><p><em>Observations:<br></em>1. People with Academic Degree are more likely to repay the loan(Out of 164, only 3 applicants are not able to repay)</p><ul><li><strong>Distribution of Family status of Applicants in terms of loan is repayed or not.</strong></li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7EPraveenKotha%2F73%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7EPraveenKotha%2F73.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/7096b90aac453f8e0fdc44aea0f1d1a4/href">https://medium.com/media/7096b90aac453f8e0fdc44aea0f1d1a4/href</a></iframe><p><em>Observations</em>:<br>1. Widows are more likely to repay the loan when compared to appliants with the other family statuses.</p><ul><li><strong>Distribution of Housing type of Applicants in terms of loan is repayed or not.</strong></li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7EPraveenKotha%2F75%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7EPraveenKotha%2F75.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/f949fad6da13b70bd6b86477f2b6c9a8/href">https://medium.com/media/f949fad6da13b70bd6b86477f2b6c9a8/href</a></iframe><ul><li><strong>Distribution of Clients Age</strong></li></ul><pre>cf.set_config_file(theme=&#39;pearl&#39;)<br>(application[&#39;DAYS_BIRTH&#39;]/(-365)).iplot(kind=&#39;histogram&#39;, <br>             xTitle = &#39;Age&#39;, bins=50,<br>             yTitle=&#39;Count of type of applicants in %&#39;,<br>             title=&#39;Distribution of Clients Age&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*acSoYa_V_PS5MJHFRUFFVw.png" /></figure><ul><li><strong>Distribution of years before the application the person started current employment.</strong></li></ul><pre>cf.set_config_file(theme=&#39;pearl&#39;)<br>(application[&#39;DAYS_EMPLOYED&#39;]).iplot(kind=&#39;histogram&#39;, <br>             xTitle = &#39;Days&#39;,bins=50,<br>             yTitle=&#39;Count of applicants in %&#39;,<br>             title=&#39;Days before the application the person started current employment&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dAUDQZ03kcrRxqbJ1CwZJw.png" /></figure><p><em>Observations:<br></em>1. The data looks strange(we have -1000.66 years(-365243 days) of employment which is impossible) looks like there is data entry error.</p><pre>error = application[application[&#39;DAYS_EMPLOYED&#39;] == 365243]<br>print(&#39;The no of errors are :&#39;, len(error))<br>(error[&#39;TARGET&#39;].value_counts()/len(error))*100</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1_fgGicnB128J3IoKzrcTQ.png" /></figure><pre># Create an error flag column<br>application[&#39;DAYS_EMPLOYED_ERROR&#39;] = application[&quot;DAYS_EMPLOYED&quot;] == 365243</pre><pre># Replace the error values with nan<br>application[&#39;DAYS_EMPLOYED&#39;].replace({365243: np.nan}, inplace = True)</pre><p>Created a seperate column ‘DAYS_EMPLOYED_ERROR’, which flags the error.</p><pre>cf.set_config_file(theme=&#39;pearl&#39;)<br>(application[&#39;DAYS_EMPLOYED&#39;]/(-365)).iplot(kind=&#39;histogram&#39;, xTitle = &#39;Years of Employment&#39;,bins=50,<br>             yTitle=&#39;Count of applicants in %&#39;,<br>             title=&#39;Years before the application the person started current employment&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BalhQDnkqH3b9jGBAvDIew.png" /></figure><pre>application[application[&#39;DAYS_EMPLOYED&#39;]&gt;(-365*2)][&#39;TARGET&#39;].value_counts()/sum(application[&#39;DAYS_EMPLOYED&#39;]&gt;(-365*2))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w0HYUpsxfwl-dqv6R06zdw.png" /></figure><p><em>Observations</em><strong>:<br></strong>1. The applicants with less than 2 years of employment are less likely to repay the loan.</p><h3>Data Preparation:</h3><p><strong>Feature Engineering of Application data:</strong></p><pre># Flag to represent when Total income is greater than Credit<br>application[&#39;INCOME_GT_CREDIT_FLAG&#39;] = application[&#39;AMT_INCOME_TOTAL&#39;] &gt; application[&#39;AMT_CREDIT&#39;]</pre><pre># Column to represent Credit Income Percent<br>application[&#39;CREDIT_INCOME_PERCENT&#39;] = application[&#39;AMT_CREDIT&#39;] / application[&#39;AMT_INCOME_TOTAL&#39;]</pre><pre># Column to represent Annuity Income percent<br>application[&#39;ANNUITY_INCOME_PERCENT&#39;] = application[&#39;AMT_ANNUITY&#39;] / application[&#39;AMT_INCOME_TOTAL&#39;]</pre><pre># Column to represent Credit Term<br>application[&#39;CREDIT_TERM&#39;] = application[&#39;AMT_CREDIT&#39;] / application[&#39;AMT_ANNUITY&#39;] </pre><pre># Column to represent Days Employed percent in his life<br>application[&#39;DAYS_EMPLOYED_PERCENT&#39;] = application[&#39;DAYS_EMPLOYED&#39;] / application[&#39;DAYS_BIRTH&#39;]</pre><pre># Shape of Application data<br>print(&#39;The shape of application data:&quot;,application.shape)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_JsnfTJ-tPOcogrd28sKOQ.png" /></figure><p><strong>Using Bureau Data:</strong></p><pre>print(&#39;Reading the data....&#39;, end=&#39;&#39;)<br>bureau = pd.read_csv(&#39;bureau.csv&#39;)<br>print(&#39;done!!!&#39;)<br>print(&#39;The shape of data:&#39;,bureau.shape)<br>print(&#39;First 5 rows of data:&#39;)<br>bureau.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mw-DGNBvg6Wuh3Fn3GYKDA.png" /></figure><p><strong>Joining Bureau data to Application data:</strong></p><pre># Combining numerical features<br>grp = bureau.drop([&#39;SK_ID_BUREAU&#39;], axis = 1).groupby(by=[&#39;SK_ID_CURR&#39;]).mean().reset_index()<br>grp.columns = [&#39;BUREAU_&#39;+column if column !=&#39;SK_ID_CURR&#39; else column for column in grp.columns]<br>application_bureau = application.merge(grp, on=&#39;SK_ID_CURR&#39;, how=&#39;left&#39;)<br>application_bureau.update(application_bureau[grp.columns].fillna(0))</pre><pre># Combining categorical features<br>bureau_categorical = pd.get_dummies(bureau.select_dtypes(&#39;object&#39;))<br>bureau_categorical[&#39;SK_ID_CURR&#39;] = bureau[&#39;SK_ID_CURR&#39;]</pre><pre>grp = bureau_categorical.groupby(by = [&#39;SK_ID_CURR&#39;]).mean().reset_index()<br>grp.columns = [&#39;BUREAU_&#39;+column if column !=&#39;SK_ID_CURR&#39; else column for column in grp.columns]<br>application_bureau = application_bureau.merge(grp, on=&#39;SK_ID_CURR&#39;, how=&#39;left&#39;)<br>application_bureau.update(application_bureau[grp.columns].fillna(0))</pre><pre># Shape of application and bureau data combined<br>print(&#39;The shape application and bureau data combined:&#39;,application_bureau.shape)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nelLZFOwHYCZ_9HL8b1ByQ.png" /></figure><p><strong>Feature Engineering of Bureau Data:</strong></p><pre># Number of past loans per customer<br>grp = bureau.groupby(by = [&#39;SK_ID_CURR&#39;])[&#39;SK_ID_BUREAU&#39;].count().reset_index().rename(columns = {&#39;SK_ID_BUREAU&#39;: &#39;BUREAU_LOAN_COUNT&#39;})</pre><pre>application_bureau = application_bureau.merge(grp, on=&#39;SK_ID_CURR&#39;, how=&#39;left&#39;)<br>application_bureau[&#39;BUREAU_LOAN_COUNT&#39;] = application_bureau[&#39;BUREAU_LOAN_COUNT&#39;].fillna(0)</pre><pre># Number of types of past loans per customer <br>grp = bureau[[&#39;SK_ID_CURR&#39;, &#39;CREDIT_TYPE&#39;]].groupby(by = [&#39;SK_ID_CURR&#39;])[&#39;CREDIT_TYPE&#39;].nunique().reset_index().rename(columns={&#39;CREDIT_TYPE&#39;: &#39;BUREAU_LOAN_TYPES&#39;})</pre><pre>application_bureau = application_bureau.merge(grp, on=&#39;SK_ID_CURR&#39;, how=&#39;left&#39;)<br>application_bureau[&#39;BUREAU_LOAN_TYPES&#39;] = application_bureau[&#39;BUREAU_LOAN_TYPES&#39;].fillna(0)</pre><pre># Debt over credit ratio <br>bureau[&#39;AMT_CREDIT_SUM&#39;] = bureau[&#39;AMT_CREDIT_SUM&#39;].fillna(0)<br>bureau[&#39;AMT_CREDIT_SUM_DEBT&#39;] = bureau[&#39;AMT_CREDIT_SUM_DEBT&#39;].fillna(0)</pre><pre>grp1 = bureau[[&#39;SK_ID_CURR&#39;,&#39;AMT_CREDIT_SUM&#39;]].groupby(by=[&#39;SK_ID_CURR&#39;])[&#39;AMT_CREDIT_SUM&#39;].sum().reset_index().rename(columns={&#39;AMT_CREDIT_SUM&#39;: &#39;TOTAL_CREDIT_SUM&#39;})</pre><pre>grp2 = bureau[[&#39;SK_ID_CURR&#39;,&#39;AMT_CREDIT_SUM_DEBT&#39;]].groupby(by=[&#39;SK_ID_CURR&#39;])[&#39;AMT_CREDIT_SUM_DEBT&#39;].sum().reset_index().rename(columns={&#39;AMT_CREDIT_SUM_DEBT&#39;:&#39;TOTAL_CREDIT_SUM_DEBT&#39;})</pre><pre>grp1[&#39;DEBT_CREDIT_RATIO&#39;] = grp2[&#39;TOTAL_CREDIT_SUM_DEBT&#39;]/grp1[&#39;TOTAL_CREDIT_SUM&#39;]</pre><pre>del grp1[&#39;TOTAL_CREDIT_SUM&#39;]</pre><pre>application_bureau = application_bureau.merge(grp1, on=&#39;SK_ID_CURR&#39;, how=&#39;left&#39;)<br>application_bureau[&#39;DEBT_CREDIT_RATIO&#39;] = application_bureau[&#39;DEBT_CREDIT_RATIO&#39;].fillna(0)<br>application_bureau[&#39;DEBT_CREDIT_RATIO&#39;] = application_bureau.replace([np.inf, -np.inf], 0)<br>application_bureau[&#39;DEBT_CREDIT_RATIO&#39;] = pd.to_numeric(application_bureau[&#39;DEBT_CREDIT_RATIO&#39;], downcast=&#39;float&#39;)</pre><pre># Overdue over debt ratio<br>bureau[&#39;AMT_CREDIT_SUM_OVERDUE&#39;] = bureau[&#39;AMT_CREDIT_SUM_OVERDUE&#39;].fillna(0)<br>bureau[&#39;AMT_CREDIT_SUM_DEBT&#39;] = bureau[&#39;AMT_CREDIT_SUM_DEBT&#39;].fillna(0)</pre><pre>grp1 = bureau[[&#39;SK_ID_CURR&#39;,&#39;AMT_CREDIT_SUM_OVERDUE&#39;]].groupby(by=[&#39;SK_ID_CURR&#39;])[&#39;AMT_CREDIT_SUM_OVERDUE&#39;].sum().reset_index().rename(columns={&#39;AMT_CREDIT_SUM_OVERDUE&#39;: &#39;TOTAL_CUSTOMER_OVERDUE&#39;})</pre><pre>grp2 = bureau[[&#39;SK_ID_CURR&#39;,&#39;AMT_CREDIT_SUM_DEBT&#39;]].groupby(by=[&#39;SK_ID_CURR&#39;])[&#39;AMT_CREDIT_SUM_DEBT&#39;].sum().reset_index().rename(columns={&#39;AMT_CREDIT_SUM_DEBT&#39;:&#39;TOTAL_CUSTOMER_DEBT&#39;})</pre><pre>grp1[&#39;OVERDUE_DEBT_RATIO&#39;] = grp1[&#39;TOTAL_CUSTOMER_OVERDUE&#39;]/grp2[&#39;TOTAL_CUSTOMER_DEBT&#39;]</pre><pre>del grp1[&#39;TOTAL_CUSTOMER_OVERDUE&#39;]</pre><pre>application_bureau = application_bureau.merge(grp1, on=&#39;SK_ID_CURR&#39;, how=&#39;left&#39;)<br>application_bureau[&#39;OVERDUE_DEBT_RATIO&#39;] = application_bureau[&#39;OVERDUE_DEBT_RATIO&#39;].fillna(0)<br>application_bureau[&#39;OVERDUE_DEBT_RATIO&#39;] = application_bureau.replace([np.inf, -np.inf], 0)</pre><pre>application_bureau[&#39;OVERDUE_DEBT_RATIO&#39;] = pd.to_numeric(application_bureau[&#39;OVERDUE_DEBT_RATIO&#39;], downcast=&#39;float&#39;)</pre><p><strong>Using Previous Application Data:</strong></p><pre>print(&#39;Reading the data....&#39;, end=&#39;&#39;)<br>previous_applicaton = pd.read_csv(&#39;previous_application.csv&#39;)<br>print(&#39;done!!!&#39;)<br>print(&#39;The shape of data:&#39;,previous_applicaton.shape)<br>print(&#39;First 5 rows of data:&#39;)<br>previous_applicaton.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3ipgqWn2-eyBedAldkCUrw.png" /></figure><p><strong>Joining Previous Application data to Application Bureau data:</strong></p><pre># Number of previous applications per customer<br>grp = previous_applicaton[[&#39;SK_ID_CURR&#39;,&#39;SK_ID_PREV&#39;]].groupby(by=[&#39;SK_ID_CURR&#39;])[&#39;SK_ID_PREV&#39;].count().reset_index().rename(columns={&#39;SK_ID_PREV&#39;:&#39;PREV_APP_COUNT&#39;})<br>application_bureau_prev = application_bureau.merge(grp, on =[&#39;SK_ID_CURR&#39;], how = &#39;left&#39;)<br>application_bureau_prev[&#39;PREV_APP_COUNT&#39;] = application_bureau_prev[&#39;PREV_APP_COUNT&#39;].fillna(0)</pre><pre># Combining numerical features<br>grp = previous_applicaton.drop(&#39;SK_ID_PREV&#39;, axis =1).groupby(by=[&#39;SK_ID_CURR&#39;]).mean().reset_index()<br>prev_columns = [&#39;PREV_&#39;+column if column != &#39;SK_ID_CURR&#39; else column for column in grp.columns ]<br>grp.columns = prev_columns<br>application_bureau_prev = application_bureau_prev.merge(grp, on =[&#39;SK_ID_CURR&#39;], how = &#39;left&#39;)<br>application_bureau_prev.update(application_bureau_prev[grp.columns].fillna(0))</pre><pre># Combining categorical features<br>prev_categorical = pd.get_dummies(previous_applicaton.select_dtypes(&#39;object&#39;))<br>prev_categorical[&#39;SK_ID_CURR&#39;] = previous_applicaton[&#39;SK_ID_CURR&#39;]<br>prev_categorical.head()</pre><pre>grp = prev_categorical.groupby(&#39;SK_ID_CURR&#39;).mean().reset_index()<br>grp.columns = [&#39;PREV_&#39;+column if column != &#39;SK_ID_CURR&#39; else column for column in grp.columns]</pre><pre>application_bureau_prev = application_bureau_prev.merge(grp, on=[&#39;SK_ID_CURR&#39;], how=&#39;left&#39;)<br>application_bureau_prev.update(application_bureau_prev[grp.columns].fillna(0))</pre><p><strong>Using POS_CASH_balance data:</strong></p><pre>print(&#39;Reading the data....&#39;, end=&#39;&#39;)<br>pos_cash = pd.read_csv(&#39;POS_CASH_balance.csv&#39;)<br>print(&#39;done!!!&#39;)<br>print(&#39;The shape of data:&#39;,pos_cash.shape)<br>print(&#39;First 5 rows of data:&#39;)<br>pos_cash.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VtqfHJxZ3MWiLo4nRuGXYA.png" /></figure><p><strong>Joining POS_CASH_balance data to application_bureau_prev_data:</strong></p><pre># Combining numerical features<br>grp = pos_cash.drop(&#39;SK_ID_PREV&#39;, axis =1).groupby(by=[&#39;SK_ID_CURR&#39;]).mean().reset_index()<br>prev_columns = [&#39;POS_&#39;+column if column != &#39;SK_ID_CURR&#39; else column for column in grp.columns ]<br>grp.columns = prev_columns<br>application_bureau_prev = application_bureau_prev.merge(grp, on =[&#39;SK_ID_CURR&#39;], how = &#39;left&#39;)<br>application_bureau_prev.update(application_bureau_prev[grp.columns].fillna(0))</pre><pre># Combining categorical features<br>pos_cash_categorical = pd.get_dummies(pos_cash.select_dtypes(&#39;object&#39;))<br>pos_cash_categorical[&#39;SK_ID_CURR&#39;] = pos_cash[&#39;SK_ID_CURR&#39;]</pre><pre>grp = pos_cash_categorical.groupby(&#39;SK_ID_CURR&#39;).mean().reset_index()<br>grp.columns = [&#39;POS_&#39;+column if column != &#39;SK_ID_CURR&#39; else column for column in grp.columns]</pre><pre>application_bureau_prev = application_bureau_prev.merge(grp, on=[&#39;SK_ID_CURR&#39;], how=&#39;left&#39;)<br>application_bureau_prev.update(application_bureau_prev[grp.columns].fillna(0))</pre><p><strong>Using installments_payments data:</strong></p><pre>print(&#39;Reading the data....&#39;, end=&#39;&#39;)<br>insta_payments = pd.read_csv(&#39;installments_payments.csv&#39;)<br>print(&#39;done!!!&#39;)<br>print(&#39;The shape of data:&#39;,insta_payments.shape)<br>print(&#39;First 5 rows of data:&#39;)<br>insta_payments.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LYMfK3XqCN-mhuwsNZaWdQ.png" /></figure><p><strong>Joining Installments Payments data to application_bureau_prev_data:</strong></p><pre># Combining numerical features and there are no categorical features in this dataset<br>grp = insta_payments.drop(&#39;SK_ID_PREV&#39;, axis =1).groupby(by=[&#39;SK_ID_CURR&#39;]).mean().reset_index()<br>prev_columns = [&#39;INSTA_&#39;+column if column != &#39;SK_ID_CURR&#39; else column for column in grp.columns ]<br>grp.columns = prev_columns<br>application_bureau_prev = application_bureau_prev.merge(grp, on =[&#39;SK_ID_CURR&#39;], how = &#39;left&#39;)<br>application_bureau_prev.update(application_bureau_prev[grp.columns].fillna(0))</pre><p><strong>Using Credit card balance data:</strong></p><pre>print(&#39;Reading the data....&#39;, end=&#39;&#39;)<br>credit_card = pd.read_csv(&#39;credit_card_balance.csv&#39;)<br>print(&#39;done!!!&#39;)<br>print(&#39;The shape of data:&#39;,credit_card.shape)<br>print(&#39;First 5 rows of data:&#39;)<br>credit_card.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iJw_6sTrjMAy5PiDvt_QDw.png" /></figure><p><strong>Joining Credit card balance data to application_bureau_prev data:</strong></p><pre># Combining numerical features<br>grp = credit_card.drop(&#39;SK_ID_PREV&#39;, axis =1).groupby(by=[&#39;SK_ID_CURR&#39;]).mean().reset_index()<br>prev_columns = [&#39;CREDIT_&#39;+column if column != &#39;SK_ID_CURR&#39; else column for column in grp.columns ]<br>grp.columns = prev_columns<br>application_bureau_prev = application_bureau_prev.merge(grp, on =[&#39;SK_ID_CURR&#39;], how = &#39;left&#39;)<br>application_bureau_prev.update(application_bureau_prev[grp.columns].fillna(0))</pre><pre># Combining categorical features<br>credit_categorical = pd.get_dummies(credit_card.select_dtypes(&#39;object&#39;))<br>credit_categorical[&#39;SK_ID_CURR&#39;] = credit_card[&#39;SK_ID_CURR&#39;]</pre><pre>grp = credit_categorical.groupby(&#39;SK_ID_CURR&#39;).mean().reset_index()<br>grp.columns = [&#39;CREDIT_&#39;+column if column != &#39;SK_ID_CURR&#39; else column for column in grp.columns]</pre><pre>application_bureau_prev = application_bureau_prev.merge(grp, on=[&#39;SK_ID_CURR&#39;], how=&#39;left&#39;)<br>application_bureau_prev.update(application_bureau_prev[grp.columns].fillna(0))</pre><p>Shape of final prepared data: (307511, 377)</p><p><strong>Dividing final data into train, valid and test datasets:</strong></p><pre>y = application_bureau_prev.pop(&#39;TARGET&#39;).values</pre><pre>X_train, X_temp, y_train, y_temp = train_test_split(application_bureau_prev.drop([&#39;SK_ID_CURR&#39;],axis=1), y, stratify = y, test_size=0.3, random_state=42)<br>X_val, X_test, y_val, y_test = train_test_split(X_temp, y_temp, stratify = y_temp, test_size=0.5, random_state=42)</pre><pre>print(&#39;Shape of X_train:&#39;,X_train.shape)<br>print(&#39;Shape of X_val:&#39;,X_val.shape)<br>print(&#39;Shape of X_test:&#39;,X_test.shape)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7BnN0y9jl2S4g__lnTW2Vw.png" /></figure><p><strong>Featurizing the data:</strong></p><pre># Seperation of columns into numeric and categorical columns<br>types = np.array([dt for dt in X_train.dtypes])</pre><pre>all_columns = X_train.columns.values<br>is_num = types != &#39;object&#39;</pre><pre>num_cols = all_columns[is_num]<br>cat_cols = all_columns[~is_num]</pre><pre># Featurization of numeric data<br>imputer_num = SimpleImputer(strategy=&#39;median&#39;)<br>X_train_num = imputer_num.fit_transform(X_train[num_cols])<br>X_val_num = imputer_num.transform(X_val[num_cols])<br>X_test_num = imputer_num.transform(X_test[num_cols])</pre><pre>scaler_num = StandardScaler()<br>X_train_num1 = scaler_num.fit_transform(X_train_num)<br>X_val_num1 = scaler_num.transform(X_val_num)<br>X_test_num1 = scaler_num.transform(X_test_num)</pre><pre>X_train_num_final = pd.DataFrame(X_train_num1, columns=num_cols)<br>X_val_num_final = pd.DataFrame(X_val_num1, columns=num_cols)<br>X_test_num_final = pd.DataFrame(X_test_num1, columns=num_cols)</pre><pre># Featurization of categorical data<br>imputer_cat = SimpleImputer(strategy=&#39;constant&#39;, fill_value=&#39;MISSING&#39;)<br>X_train_cat = imputer_cat.fit_transform(X_train[cat_cols])<br>X_val_cat = imputer_cat.transform(X_val[cat_cols])<br>X_test_cat = imputer_cat.transform(X_test[cat_cols])</pre><pre>X_train_cat1= pd.DataFrame(X_train_cat, columns=cat_cols)<br>X_val_cat1= pd.DataFrame(X_val_cat, columns=cat_cols)<br>X_test_cat1= pd.DataFrame(X_test_cat, columns=cat_cols)</pre><pre>ohe = OneHotEncoder(sparse=False,handle_unknown=&#39;ignore&#39;)<br>X_train_cat2 = ohe.fit_transform(X_train_cat1)<br>X_val_cat2 = ohe.transform(X_val_cat1)<br>X_test_cat2 = ohe.transform(X_test_cat1)</pre><pre>cat_cols_ohe = list(ohe.get_feature_names(input_features=cat_cols))</pre><pre>X_train_cat_final = pd.DataFrame(X_train_cat2, columns = cat_cols_ohe)<br>X_val_cat_final = pd.DataFrame(X_val_cat2, columns = cat_cols_ohe)<br>X_test_cat_final = pd.DataFrame(X_test_cat2, columns = cat_cols_ohe)</pre><pre># Final complete data<br>X_train_final = pd.concat([X_train_num_final,X_train_cat_final], axis = 1)<br>X_val_final = pd.concat([X_val_num_final,X_val_cat_final], axis = 1)<br>X_test_final = pd.concat([X_test_num_final,X_test_cat_final], axis = 1)<br>print(X_train_final.shape)<br>print(X_val_final.shape)<br>print(X_test_final.shape)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SoqJOXlFkplsGkTk-DzbIw.png" /></figure><p><strong>Saving the files for future use:</strong></p><pre># Saving the Dataframes into CSV files for future use<br>X_train_final.to_csv(&#39;X_train_final.csv&#39;)<br>X_val_final.to_csv(&#39;X_val_final.csv&#39;)<br>X_test_final.to_csv(&#39;X_test_final.csv&#39;)</pre><pre># Saving the numpy arrays into text files for future use<br>np.savetxt(&#39;y.txt&#39;, y)<br>np.savetxt(&#39;y_train.txt&#39;, y_train)<br>np.savetxt(&#39;y_val.txt&#39;, y_val)<br>np.savetxt(&#39;y_test.txt&#39;, y_test)</pre><h4><strong>Selection of features:</strong></h4><pre>model_sk = lgb.LGBMClassifier(boosting_type=&#39;gbdt&#39;, max_depth=7, learning_rate=0.01, n_estimators= 2000, <br>                 class_weight=&#39;balanced&#39;, subsample=0.9, colsample_bytree= 0.8, n_jobs=-1)</pre><pre>train_features, valid_features, train_y, valid_y = train_test_split(X_train_final, y_train, test_size = 0.15, random_state = 42)</pre><pre>model_sk.fit(train_features, train_y, early_stopping_rounds=100, eval_set = [(valid_features, valid_y)], eval_metric = &#39;auc&#39;, verbose = 200)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UIRZKmrDWs-PSs2smGJlrg.png" /><figcaption>Training LGBM</figcaption></figure><pre>feature_imp = pd.DataFrame(sorted(zip(model_sk.feature_importances_, X_train_final.columns)), columns=[&#39;Value&#39;,&#39;Feature&#39;])<br>features_df = feature_imp.sort_values(by=&quot;Value&quot;, ascending=False)<br>selected_features = list(features_df[features_df[&#39;Value&#39;]&gt;=50][&#39;Feature&#39;])</pre><pre># Saving the selected features into pickle file<br>with open(&#39;select_features.txt&#39;,&#39;wb&#39;) as fp:<br>    pickle.dump(selected_features, fp)</pre><pre>print(&#39;The no. of features selected:&#39;,len(selected_features))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7OpmjZQxmJzagqAy1NTwXQ.png" /></figure><pre># Feature importance Plot<br>data1 = features_df.head(20)<br>data = [go.Bar(x =data1.sort_values(by=&#39;Value&#39;)[&#39;Value&#39;] , y = data1.sort_values(by=&#39;Value&#39;)[&#39;Feature&#39;], orientation = &#39;h&#39;,<br>              marker = dict(<br>        color = &#39;rgba(43, 13, 150, 0.6)&#39;,<br>        line = dict(<br>            color = &#39;rgba(43, 13, 150, 1.0)&#39;,<br>            width = 1.5)<br>    )) ]</pre><pre>layout = go.Layout(<br>    autosize=False,<br>    width=1300,<br>    height=700,<br>    title = &quot;Top 20 important features&quot;,<br>    xaxis=dict(<br>        title=&#39;Importance value&#39;<br>        ),<br>    yaxis=dict(<br>        automargin=True<br>        ),<br>    bargap=0.4<br>    )</pre><pre>fig = go.Figure(data = data, layout=layout)<br>fig.layout.template = &#39;seaborn&#39;</pre><pre>py.iplot(fig)</pre><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7EPraveenKotha%2F77%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7EPraveenKotha%2F77.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/efb78f8dd614b62d79ce9a189199acdd/href">https://medium.com/media/efb78f8dd614b62d79ce9a189199acdd/href</a></iframe><h3>Machine Learning Models:</h3><p>I have tried Logistic Regression, Random Forest and LightGBM machine learning models.</p><p>Reusable functions for plotting Confusion matrix and CV plot.</p><pre>def plot_confusion_matrix(test_y, predicted_y):<br>    # Confusion matrix<br>    C = confusion_matrix(test_y, predicted_y)<br>    <br>    # Recall matrix<br>    A = (((C.T)/(C.sum(axis=1))).T)<br>    <br>    # Precision matrix<br>    B = (C/C.sum(axis=0))<br>    <br>    plt.figure(figsize=(20,4))<br>    <br>    labels = [&#39;Re-paid(0)&#39;,&#39;Not Re-paid(1)&#39;]<br>    cmap=sns.light_palette(&quot;purple&quot;)<br>    plt.subplot(1,3,1)<br>    sns.heatmap(C, annot=True, cmap=cmap,fmt=&quot;d&quot;, xticklabels = labels, yticklabels=labels)<br>    plt.xlabel(&#39;Predicted Class&#39;)<br>    plt.ylabel(&#39;Orignal Class&#39;)<br>    plt.title(&#39;Confusion matrix&#39;)<br>    <br>    plt.subplot(1,3,2)<br>    sns.heatmap(A, annot=True, cmap=cmap, xticklabels = labels, yticklabels=labels)<br>    plt.xlabel(&#39;Predicted Class&#39;)<br>    plt.ylabel(&#39;Orignal Class&#39;)<br>    plt.title(&#39;Recall matrix&#39;)<br>    <br>    plt.subplot(1,3,3)<br>    sns.heatmap(B, annot=True, cmap=cmap, xticklabels = labels, yticklabels=labels)<br>    plt.xlabel(&#39;Predicted Class&#39;)<br>    plt.ylabel(&#39;Orignal Class&#39;)<br>    plt.title(&#39;Precision matrix&#39;)<br>    <br>    plt.show()<br></pre><pre>def cv_plot(alpha, cv_auc):<br>    <br>    fig, ax = plt.subplots()<br>    ax.plot(np.log10(alpha), cv_auc,c=&#39;g&#39;)<br>    for i, txt in enumerate(np.round(cv_auc,3)):<br>        ax.annotate((alpha[i],str(txt)), (np.log10(alpha[i]),cv_auc[i]))<br>    plt.grid()<br>    plt.xticks(np.log10(alpha))<br>    plt.title(&quot;Cross Validation Error for each alpha&quot;)<br>    plt.xlabel(&quot;Alpha i&#39;s&quot;)<br>    plt.ylabel(&quot;Error measure&quot;)<br>    plt.show()</pre><h4><strong>Logistic regression with selected features:</strong></h4><p>Logistic Regression finds a hyperplane which best seperates the given positive and negative data points.</p><pre>alpha = np.logspace(-4,4,9)<br>cv_auc_score = []</pre><pre>for i in alpha:<br>    clf = SGDClassifier(alpha=i, penalty=&#39;l1&#39;,class_weight = &#39;balanced&#39;, loss=&#39;log&#39;, random_state=28)<br>    clf.fit(X_train_final[selected_features], y_train)<br>    sig_clf = CalibratedClassifierCV(clf, method=&#39;sigmoid&#39;)<br>    sig_clf.fit(X_train_final[selected_features], y_train)<br>    y_pred_prob = sig_clf.predict_proba(X_val_final[selected_features])[:,1]<br>    cv_auc_score.append(roc_auc_score(y_val,y_pred_prob))<br>    print(&#39;For alpha {0}, cross validation AUC score {1}&#39;.format(i,roc_auc_score(y_val,y_pred_prob)))</pre><pre>cv_plot(alpha, cv_auc_score)</pre><pre>print(&#39;The Optimal C value is:&#39;, alpha[np.argmax(cv_auc_score)])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/1*vfGDJuYCqX659hIGTeajhg.png" /><figcaption>Cross validation results and plot for Logistic Regression model.</figcaption></figure><pre>best_alpha = alpha[np.argmax(cv_auc_score)]<br>logreg = SGDClassifier(alpha = best_alpha, class_weight = &#39;balanced&#39;, penalty = &#39;l1&#39;, loss=&#39;log&#39;, random_state = 28)<br>logreg.fit(X_train_final[selected_features], y_train)<br>logreg_sig_clf = CalibratedClassifierCV(logreg, method=&#39;sigmoid&#39;)<br>logreg_sig_clf.fit(X_train_final[selected_features], y_train)<br>y_pred_prob = logreg_sig_clf.predict_proba(X_train_final[selected_features])[:,1]<br>print(&#39;For best alpha {0}, The Train AUC score is {1}&#39;.format(best_alpha, roc_auc_score(y_train,y_pred_prob) ))    <br>y_pred_prob = logreg_sig_clf.predict_proba(X_val_final[selected_features])[:,1]<br>print(&#39;For best alpha {0}, The Cross validated AUC score is {1}&#39;.format(best_alpha, roc_auc_score(y_val,y_pred_prob) ))  <br>y_pred_prob = logreg_sig_clf.predict_proba(X_test_final[selected_features])[:,1]<br>print(&#39;For best alpha {0}, The Test AUC score is {1}&#39;.format(best_alpha, roc_auc_score(y_test,y_pred_prob) ))</pre><pre>y_pred = logreg.predict(X_test_final[selected_features])<br>print(&#39;The test AUC score is :&#39;, roc_auc_score(y_test,y_pred_prob))<br>print(&#39;The percentage of misclassified points {:05.2f}% :&#39;.format((1-accuracy_score(y_test, y_pred))*100))<br>plot_confusion_matrix(y_test, y_pred)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qj6anX8MPrfC1gAT-xn4Qw.png" /><figcaption>Logistic Regression model results</figcaption></figure><pre>from sklearn.metrics import roc_curve<br>fpr, tpr, thresholds = roc_curve(y_test, y_pred_prob)<br>auc = roc_auc_score(y_test,y_pred_prob)</pre><pre>plt.figure(figsize=(8,6))<br>plt.plot(fpr, tpr, marker=&#39;.&#39;)<br>plt.plot([0, 1], [0, 1], linestyle=&#39;--&#39;)<br>plt.title(&#39;ROC curve&#39;, fontsize = 20)<br>plt.xlabel(&#39;FPR&#39;, fontsize=15)<br>plt.ylabel(&#39;TPR&#39;, fontsize=15)<br>plt.grid()<br>plt.legend([&quot;AUC=%.3f&quot;%auc])<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tYZiLG4FUtzHs0JdjltNvQ.png" /><figcaption>ROC curve for Logistic Regression model with AUC=0.754</figcaption></figure><h4>Random Forest with selected features:</h4><p>The Random Forest is a model made up of many decision trees. Rather than just simply averaging the prediction of trees (which we could call a “forest”), this model uses two key concepts that gives it the name <em>random</em>:</p><ol><li>Random sampling of training data points when building trees</li><li>Random subsets of features considered when splitting nodes</li></ol><pre>alpha = [200,500,1000,2000]<br>max_depth = [7, 10]<br>cv_auc_score = []<br>for i in alpha:<br>    for j in max_depth:<br>        clf = RandomForestClassifier(n_estimators=i, criterion=&#39;gini&#39;, max_depth=j,class_weight=&#39;balanced&#39;,<br>                                     random_state=42, n_jobs=-1)<br>        clf.fit(X_train_final[selected_features], y_train)<br>        sig_clf = CalibratedClassifierCV(clf, method=&quot;sigmoid&quot;)<br>        sig_clf.fit(X_train_final[selected_features], y_train)<br>        y_pred_prob = sig_clf.predict_proba(X_val_final[selected_features])[:,1]<br>        cv_auc_score.append(roc_auc_score(y_val,y_pred_prob))<br>        print(&#39;For n_estimators {0}, max_depth {1} cross validation AUC score {2}&#39;.<br>              format(i,j,roc_auc_score(y_val,y_pred_prob)))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-T5biK4VByzE90tV4TZhQg.png" /><figcaption>Cross validation results for Random Forest model.</figcaption></figure><pre>best_alpha = np.argmax(cv_auc_score)<br>print(&#39;The optimal values are: n_estimators {0}, max_depth {1} &#39;.format(alpha[int(best_alpha/2)],<br>                                                                        max_depth[int(best_alpha%2)]))<br>rf = RandomForestClassifier(n_estimators=alpha[int(best_alpha/2)], criterion=&#39;gini&#39;, max_depth=max_depth[int(best_alpha%2)],<br>                            class_weight=&#39;balanced&#39;, random_state=42, n_jobs=-1)</pre><pre>rf.fit(X_train_final[selected_features], y_train)<br>rf_sig_clf = CalibratedClassifierCV(rf, method=&quot;sigmoid&quot;)<br>rf_sig_clf.fit(X_train_final[selected_features], y_train)</pre><pre>y_pred_prob = rf_sig_clf.predict_proba(X_train_final[selected_features])[:,1]<br>print(&#39;For best n_estimators {0} best max_depth {1}, The Train AUC score is {2}&#39;.format(alpha[int(best_alpha/2)], <br>                                                    max_depth[int(best_alpha%2)],roc_auc_score(y_train,y_pred_prob)))<br>y_pred_prob = rf_sig_clf.predict_proba(X_val_final[selected_features])[:,1]<br>print(&#39;For best n_estimators {0} best max_depth {1}, The Validation AUC score is {2}&#39;.format(alpha[int(best_alpha/2)],<br>                                                            max_depth[int(best_alpha%2)],roc_auc_score(y_val,y_pred_prob)))<br>y_pred_prob = rf_sig_clf.predict_proba(X_test_final[selected_features])[:,1]<br>print(&#39;For best n_estimators {0} best max_depth {1}, The Test AUC score is {2}&#39;.format(alpha[int(best_alpha/2)],<br>                                                        max_depth[int(best_alpha%2)],roc_auc_score(y_test,y_pred_prob)))</pre><pre>y_pred = rf_sig_clf.predict(X_test_final[selected_features])<br>print(&#39;The test AUC score is :&#39;, roc_auc_score(y_test,y_pred_prob))<br>print(&#39;The percentage of misclassified points {:05.2f}% :&#39;.format((1-accuracy_score(y_test, y_pred))*100))<br>plot_confusion_matrix(y_test, y_pred)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EPMdUsAyZ9olI1H4k6z4DQ.png" /><figcaption>Random Forest model results.</figcaption></figure><pre>from sklearn.metrics import roc_curve<br>fpr, tpr, thresholds = roc_curve(y_test, y_pred_prob)<br>auc = roc_auc_score(y_test,y_pred_prob)</pre><pre>plt.figure(figsize=(8,6))<br>plt.plot(fpr, tpr, marker=&#39;.&#39;)<br>plt.plot([0, 1], [0, 1], linestyle=&#39;--&#39;)<br>plt.title(&#39;ROC curve&#39;, fontsize = 20)<br>plt.xlabel(&#39;FPR&#39;, fontsize=15)<br>plt.ylabel(&#39;TPR&#39;, fontsize=15)<br>plt.grid()<br>plt.legend([&quot;AUC=%.3f&quot;%auc])<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SIftlsk28VYyjBaHBqT-jA.png" /><figcaption>ROC curve for Random Forest model with AUC=0.75</figcaption></figure><h4>LightGBM with selected features:</h4><p>Light GBM is a fast, distributed, high-performance gradient boosting framework based on decision tree algorithm, used for ranking, classification and many other machine learning tasks.</p><p>Since it is based on decision tree algorithms, it splits the tree leaf wise with the best fit whereas other boosting algorithms split the tree depth wise or level wise rather than leaf-wise. So when growing on the same leaf in Light GBM, the leaf-wise algorithm can reduce more loss than the level-wise algorithm and hence results in much better accuracy which can rarely be achieved by any of the existing boosting algorithms. Also, it is surprisingly very fast, hence the word ‘Light’.</p><pre>weight = np.ones((len(X_train_final),), dtype=int)<br>for i in range(len(X_train_final)):<br>    if y_train[i]== 0:<br>        weight[i]=1<br>    else:<br>        weight[i]=11</pre><pre><br>train_data=lgb.Dataset(X_train_final[selected_features], label = y_train, weight= weight )<br>valid_data=lgb.Dataset(X_val_final[selected_features], label = y_val)</pre><pre>cv_auc_score = []<br>max_depth = [3, 5, 7, 10]</pre><pre>for i in max_depth:<br>    <br>    params = {&#39;boosting_type&#39;: &#39;gbdt&#39;,<br>          &#39;max_depth&#39; : i,<br>          &#39;objective&#39;: &#39;binary&#39;,<br>          &#39;nthread&#39;: 5,<br>          &#39;num_leaves&#39;: 32,<br>          &#39;learning_rate&#39;: 0.05,<br>          &#39;max_bin&#39;: 512,<br>          &#39;subsample_for_bin&#39;: 200,<br>          &#39;subsample&#39;: 0.7,<br>          &#39;subsample_freq&#39;: 1,<br>          &#39;colsample_bytree&#39;: 0.8,<br>          &#39;reg_alpha&#39;: 20,<br>          &#39;reg_lambda&#39;: 20,<br>          &#39;min_split_gain&#39;: 0.5,<br>          &#39;min_child_weight&#39;: 1,<br>          &#39;min_child_samples&#39;: 10,<br>          &#39;scale_pos_weight&#39;: 1,<br>          &#39;num_class&#39; : 1,<br>          &#39;metric&#39; : &#39;auc&#39;<br>          }</pre><pre>lgbm = lgb.train(params,<br>                 train_data,<br>                 2500,<br>                 valid_sets=valid_data,<br>                 early_stopping_rounds= 100,<br>                 verbose_eval= 10<br>                 )<br>    y_pred_prob = lgbm.predict(X_val_final[selected_features])<br>    cv_auc_score.append(roc_auc_score(y_val,y_pred_prob))<br>    print(&#39;For  max_depth {0} and some other parameters, cross validation AUC score {1}&#39;.format(i,roc_auc_score(y_val,y_pred_prob)))</pre><pre>print(&#39;The optimal  max_depth: &#39;, max_depth[np.argmax(cv_auc_score)])</pre><pre>params = {&#39;boosting_type&#39;: &#39;gbdt&#39;,<br>          &#39;max_depth&#39; : max_depth[np.argmax(cv_auc_score)],<br>          &#39;objective&#39;: &#39;binary&#39;,<br>          &#39;nthread&#39;: 5,<br>          &#39;num_leaves&#39;: 32,<br>          &#39;learning_rate&#39;: 0.05,<br>          &#39;max_bin&#39;: 512,<br>          &#39;subsample_for_bin&#39;: 200,<br>          &#39;subsample&#39;: 0.7,<br>          &#39;subsample_freq&#39;: 1,<br>          &#39;colsample_bytree&#39;: 0.8,<br>          &#39;reg_alpha&#39;: 20,<br>          &#39;reg_lambda&#39;: 20,<br>          &#39;min_split_gain&#39;: 0.5,<br>          &#39;min_child_weight&#39;: 1,<br>          &#39;min_child_samples&#39;: 10,<br>          &#39;scale_pos_weight&#39;: 1,<br>          &#39;num_class&#39; : 1,<br>          &#39;metric&#39; : &#39;auc&#39;<br>          }</pre><pre>lgbm = lgb.train(params,<br>                 train_data,<br>                 2500,<br>                 valid_sets=valid_data,<br>                 early_stopping_rounds= 100,<br>                 verbose_eval= 10<br>                 )<br>y_pred_prob = lgbm.predict(X_train_final[selected_features])<br>print(&#39;For best max_depth {0}, The Train AUC score is {1}&#39;.format(max_depth[np.argmax(cv_auc_score)], <br>                                                                  roc_auc_score(y_train,y_pred_prob) ))    <br>y_pred_prob = lgbm.predict(X_val_final[selected_features])<br>print(&#39;For best max_depth {0}, The Cross validated AUC score is {1}&#39;.format(max_depth[np.argmax(cv_auc_score)], <br>                                                                            roc_auc_score(y_val,y_pred_prob) ))  <br>y_pred_prob = lgbm.predict(X_test_final[selected_features])<br>print(&#39;For best max_depth {0}, The Test AUC score is {1}&#39;.format(max_depth[np.argmax(cv_auc_score)], <br>                                                                 roc_auc_score(y_test,y_pred_prob) ))</pre><pre>y_pred = np.ones((len(X_test_final),), dtype=int)<br>for i in range(len(y_pred_prob)):<br>    if y_pred_prob[i]&lt;=0.5:<br>        y_pred[i]=0<br>    else:<br>        y_pred[i]=1</pre><pre>print(&#39;The test AUC score is :&#39;, roc_auc_score(y_test,y_pred_prob))<br>print(&#39;The percentage of misclassified points {:05.2f}% :&#39;.format((1-accuracy_score(y_test, y_pred))*100))<br>plot_confusion_matrix(y_test, y_pred)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r_hQRiY3Hvr524uerMUy1w.png" /><figcaption>LightGBM model Results</figcaption></figure><pre>from sklearn.metrics import roc_curve<br>fpr, tpr, thresholds = roc_curve(y_test, y_pred_prob)<br>auc = roc_auc_score(y_test,y_pred_prob)</pre><pre>plt.figure(figsize=(8,6))<br>plt.plot(fpr, tpr, marker=&#39;.&#39;)<br>plt.plot([0, 1], [0, 1], linestyle=&#39;--&#39;)<br>plt.title(&#39;ROC curve&#39;, fontsize = 20)<br>plt.xlabel(&#39;FPR&#39;, fontsize=15)<br>plt.ylabel(&#39;TPR&#39;, fontsize=15)<br>plt.grid()<br>plt.legend([&quot;AUC=%.3f&quot;%auc])<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g3yw2wlkCP46z4lmW0RWWg.png" /><figcaption>ROC curve for LightGBM model with AUC=0.787</figcaption></figure><h4>Overview of Results:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*xRIBVbIh7_ELKlnwILyNfg.png" /><figcaption>Final Results</figcaption></figure><p><strong>LightGBM</strong> gives the best performance and it is also faster to train when compared to Xgboost.</p><p><strong>Saving the final model as Pickle file:</strong></p><pre>#<em> </em>Saving the final model LightGBM as pickle file for the future use in productionizing the model</pre><pre><strong>with</strong> open(&#39;final_model.pkl&#39;,&#39;wb&#39;) <strong>as</strong> fp:<br>    pickle.dump(lgbm, fp)</pre><h4>References:</h4><p><a href="https://www.kaggle.com/c/home-credit-default-risk">https://www.kaggle.com/c/home-credit-default-risk</a></p><p><a href="https://www.youtube.com/watch?v=OAl6eAyP-yo">https://www.youtube.com/watch?v=OAl6eAyP-yo</a></p><p><a href="https://medium.com/@williamkoehrsen/random-forest-simple-explanation-377895a60d2d">https://medium.com/@williamkoehrsen/random-forest-simple-explanation-377895a60d2d</a></p><p><a href="https://www.analyticsvidhya.com/blog/2017/06/which-algorithm-takes-the-crown-light-gbm-vs-xgboost/">https://www.analyticsvidhya.com/blog/2017/06/which-algorithm-takes-the-crown-light-gbm-vs-xgboost/</a></p><p><a href="http://mlexplained.com/2018/01/05/lightgbm-and-xgboost-explained/">http://mlexplained.com/2018/01/05/lightgbm-and-xgboost-explained/</a></p><p>If you have any questions/suggestions, feel free to comment.<br>You can connect with me through <a href="https://www.linkedin.com/in/praveen-kotha-16533394/">linkedin</a>.</p><p>Since the ipynb file is huge, I can’t share it through Github.<br>Comment here, I can share it with you directly.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1871f52e3ef2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to find whether Train data and Test data comes from same data distribution]]></title>
            <link>https://medium.com/@praveenkotha/how-to-find-whether-train-data-and-test-data-comes-from-same-data-distribution-9259018343b?source=rss-681493d7f9d4------2</link>
            <guid isPermaLink="false">https://medium.com/p/9259018343b</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Praveen Kotha]]></dc:creator>
            <pubDate>Thu, 09 Aug 2018 12:43:52 GMT</pubDate>
            <atom:updated>2018-08-09T12:45:27.126Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/997/1*Epo5zIdp_RvpCgnrJ3feeQ.jpeg" /></figure><p>At times, there will be a scenario where the performance on test data is really bad and we don’t know the underlying reason for the same.</p><p>One possible reason could be, the test data distribution and train data distribution are not similar.</p><p>If we have 1 or 2 features we can use some plotting techniques and see the distribution of data.</p><p>How to find if we have more than 2 features or say for any real world data??</p><p>Follow this step by step procedure.</p><h4>Procedure:</h4><ul><li>Divide your Dataset (D)into train (Dtrain)and test (Dtest) sets.</li><li>Make all the labels of Dtrain to be 1(positive) which results in Dtrain¹ and Dtest to be 0(negative) which results in Dtest¹.</li><li>Now combining both Dtrain¹ and Dtest¹, we get a new dataset D¹.</li><li>Train a classifier on Dtrain¹ and get the accuracy on test data Dtest¹.</li></ul><blockquote>Case 1:</blockquote><p>If the accuracy we get on test data is low, we can conclude that both train and test distributions are similar.</p><p>Let me elaborate in concise way, when will accuracy be low in this case? If the predicted class label for test data(actual label is 0) is 1. That means if the model is predicting the test data to be label 1, they are actually similar to train data.</p><blockquote>Case 2:</blockquote><p>If the accuracy we get on test data is medium, we can conclude that both train and test distributions are not very similar.</p><blockquote>Case 3:</blockquote><p>If the accuracy we get on test data is high, we can conclude that both train and test distributions are not similar.</p><h4>Solution for this problem:</h4><p>So what can we possibly do to improve our performance on the test data ?</p><p>1. Dropping of drifting features.</p><p>2. Importance weight using Density Ratio Estimation.</p><p>If you have any questions/suggestions, feel free to comment.</p><p>You can connect with me through <a href="https://www.linkedin.com/in/praveen-kotha-16533394/">linkedin</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9259018343b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[All you need to know about K-NN machine learning algorithm.]]></title>
            <link>https://becominghuman.ai/all-you-need-to-know-about-k-nn-machine-learning-algorithm-40c3ca45131e?source=rss-681493d7f9d4------2</link>
            <guid isPermaLink="false">https://medium.com/p/40c3ca45131e</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Praveen Kotha]]></dc:creator>
            <pubDate>Wed, 08 Aug 2018 09:47:02 GMT</pubDate>
            <atom:updated>2018-08-13T14:16:17.411Z</atom:updated>
            <content:encoded><![CDATA[<p>K-NN (K-Nearest Neighbors) is one of the simplest machine learning algorithm to understand intutively. It is used in both Classification and Regression task.</p><p>K-NN is non-parametric and lazy method.</p><ul><li>The reason it is called non-parametric method is that, it doesn’t make any assumptions on the underlying data distribution. In linear regression the data is assumed to be normally distibuted which is not possible all the time, in that scenario linear regression fails to perform better. We can use K-NN, if we aren’t confident about the distribution of data.</li><li>It is called lazy method, because in the training phase it does not learn from the training data. Only in the testing phase, it starts finding nearest neighbors to the test point, which is computationally expensive.</li></ul><p>But how is it different from K-means algorithm??</p><p>K-NN is supervised learning algorithm used for classification and regression tasks, whereas K-means is unsupervised learning algorithm used for clustering tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/765/1*lvtThMM-kC6_R-WBU4cznQ.jpeg" /><figcaption>K-nearest neighbors with k=3</figcaption></figure><h4><strong>Objective</strong>:</h4><p>Given a test point ‘X<em>q’, </em>we need to determine the class label (if it is classification task) or real value (if it is regression task)</p><h4>Intuition:</h4><p>If majority of the points that are geometrically close to ‘X<em>q</em>’ are positive then the class label is going to be positive and viceversa.</p><p>That’s great, but how to find the geometrically close points??</p><p>We use some distance measures like euclidean, manhattan, minkowski, cosine similarity etc. to find the ‘K’ nearest neighbors.</p><figure><a href="https://www.eventbrite.com/e/chatbot-conference-tickets-45884424549?utm-medium=discovery&amp;utm-campaign=social&amp;utm-content=attendeeshare&amp;aff=escb&amp;utm-source=cp&amp;utm-term=listing"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oyftTzPL9qaCYc8z2EbcHg.png" /></a></figure><p>The choice of distance metric is subjective, euclidean distance suffers from curse of dimensionality for higher dimensional data. Like wise we need to choose distance metric based on the type of data and usecase.</p><p>Let’s see how to implement K-NN in steps.</p><h4><strong>K-NN algorithm with step wise procedure:</strong></h4><ol><li>Separate the data into training(m points) and testing (n points) sets.</li><li>Take a data point from the test set and measure the distance from that point to each point in the training set and we get the ‘m’ distances.</li><li>Sort the obtained ‘m’ distances in ascending order.</li><li>Pick the top ‘K’ distances from the sorted distances and their corresponding labels.</li><li>If majority of labels are positive, then the predicted label for the test point is positve and vice versa.</li><li>Repeat again from step 2 until we predict the labels for all the testing set points.</li></ol><h3>Trending AI Articles:</h3><blockquote><a href="https://becominghuman.ai/keras-cheat-sheet-neural-networks-in-python-738c0d170c2e">1. Keras Cheat Sheet: Neural Networks in Python</a></blockquote><blockquote><a href="https://becominghuman.ai/making-a-simple-neural-network-2ea1de81ec20">2. Making a Simple Neural Network</a></blockquote><blockquote><a href="https://becominghuman.ai/google-will-beat-apple-at-its-own-game-with-superior-ai-534ab3ada949">3. Google will beat Apple at its own game with superior AI</a></blockquote><blockquote><a href="https://becominghuman.ai/ai-nlp-workshop-7bc121986d61">4. AI &amp; NLP Workshop</a></blockquote><p>That’s ok, but how to perform regression task??</p><h4>K-NN for Regression:</h4><p>Instead of taking majority vote as in classification task. For regression we need to take mean or median for the corresponding target values of the top ‘K’ distances.</p><p>Since from the begining of this post you are talking about ‘K’ nearest neighbors. How do we select ‘K’ ??</p><h4>Selection of K:</h4><p>We generally select the optimal ‘K’ by using cross validation.</p><p>Couple of points to remember:</p><ul><li>If we take low ‘K’ value our model tends to overfit.</li><li>If we take high ‘K’ value our model tends to underfit.</li></ul><p>What will happen if I take K=m (no of data points in training set) ??</p><p>Irrespective of test data point, it only predicts either positive or negative for all the points in testing set. Suppose in our training set if we have m1 positive data points and m2 negative data points where m1+m2 = m.</p><ul><li>If m1&gt;m2, the model will predict Positive for all the points in the test set.</li><li>If m2&gt;m1, the model will predict Negative for all the points in the test set.</li></ul><p>If you are using K-NN for binary classification(2 classes) or with even number of classes(2,4,6,8…), it is good practise to choose K with odd number to avoid tie. In the same way, with odd number of classes(3,5,7,9…) choose K with even number.</p><h4>Impact of Outliers:</h4><ul><li>If ‘K’ value is low, the model is susceptible to outliers.</li><li>If ‘K’ value is high, the model is robust to outliers.</li></ul><p>Why is it so??</p><p>Let take K=1, suppose there is 1 outlier near to our test point and then the model predicts the label corresponding to the outlier. In this same scenario if we take K=7, there will be other 6 nearest neighbors(not outliers) and 1 outlier in the neighborhood of test point and when we take the majority vote we tend to get the result based on 6 nearest neighbors.</p><h4>Impact of scale:</h4><p>There is impact of scale in K-NN, if features range are very different(like f1 values range in 0–1000 and f2 values range in 0–1). It is always better to apply column standardization or normalization before you build your model.</p><p>Algorithms like Decision tree and Random forest are scale independent.</p><h4>Code implementation using Scikit-learn:</h4><pre>#Divide the data into X_train,Y_train and X_test,Y_test<br>from sklearn.model_selection import GridSearchCV<br>from sklearn.neighbors import KNeighborsClassifier<br>from sklearn.metrics import accuracy_score</pre><pre>#Using Grid search cv to find the optimal K(n_neighbors)<br>neigh = KNeighborsClassifier()<br>parameters = {&#39;n_neighbors&#39;:(1,3,5,7,9,11)}<br>gs_clf = GridSearchCV(neigh, parameters, cv = 5)<br>gs_clf.fit(X_train,Y_train)</pre><pre>#To select optimal K<br>optimal_k = gs_clf.best_params_.get(&#39;n_neighbors&#39;)<br>print(optimal_k)</pre><pre>#Use the obtained optimal_k to train our model<br>knn_final = KNeighborsClassifier(n_neighbors = optimal_k)<br>knn_final.fit(X_train,Y_train)<br>predictions_test = knn_final.predict(X_test)<br>predictions_train = knn_final.predict(X_train)</pre><pre>#The test and train accuracy<br>test_acc = accuracy_score(Y_test, predictions_test)*100<br>train_acc = accuracy_score(Y_train, predictions_train)*100</pre><h4>Pros of K-NN:</h4><ul><li>It is simple to understand and easy to implement.</li><li>Non-parametric nature can be useful at times.</li><li>K-NN can be extended easily for multi-class classification.</li></ul><h4>Cons of K-NN:</h4><ul><li>Testing phase is computationally expensive and cannot be used for low latency requirement.</li><li>Suffers from curse of dimensionality for higher dimensional data(d &gt; 500), if we use euclidean distance measure which is quite common.</li></ul><p>If you have any questions/suggestions, feel free to comment.</p><p>You can connect with me through <a href="https://www.linkedin.com/in/praveen-kotha-16533394/">linkedin</a>.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2F8f5f8b%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2F8f5f8b&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/c43026df6fee7cdb1aab8aaf916125ea/href">https://medium.com/media/c43026df6fee7cdb1aab8aaf916125ea/href</a></iframe><figure><a href="https://becominghuman.ai/artificial-intelligence-communities-c305f28e674c"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*2f7OqE2AJK1KSrhkmD9ZMw.png" /></a></figure><figure><a href="https://upscri.be/8f5f8b"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*v-PpfkSWHbvlWWamSVHHWg.png" /></a></figure><figure><a href="https://becominghuman.ai/write-for-us-48270209de63"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*Wt2auqISiEAOZxJ-I7brDQ.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=40c3ca45131e" width="1" height="1" alt=""><hr><p><a href="https://becominghuman.ai/all-you-need-to-know-about-k-nn-machine-learning-algorithm-40c3ca45131e">All you need to know about K-NN machine learning algorithm.</a> was originally published in <a href="https://becominghuman.ai/">Becoming Human: Artificial Intelligence Magazine</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>