<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Winners Okebaram on Medium]]></title>
        <description><![CDATA[Stories by Winners Okebaram on Medium]]></description>
        <link>https://medium.com/@whynniethegr8?source=rss-e81cf5bbfe4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*X1omCMqYys0JJ1ISyH6OYw.jpeg</url>
            <title>Stories by Winners Okebaram on Medium</title>
            <link>https://medium.com/@whynniethegr8?source=rss-e81cf5bbfe4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 12:02:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@whynniethegr8/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The European Soccer Database: Exploration with SQL]]></title>
            <link>https://medium.com/@whynniethegr8/the-european-soccer-database-exploration-with-sql-a2669eae65bd?source=rss-e81cf5bbfe4------2</link>
            <guid isPermaLink="false">https://medium.com/p/a2669eae65bd</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[soccer-analytics]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Winners Okebaram]]></dc:creator>
            <pubDate>Fri, 13 Jan 2023 02:16:20 GMT</pubDate>
            <atom:updated>2023-01-13T02:30:58.326Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*SNVnk-HiwpzEBoo8" /></figure><h4><strong>INTRODUCTION</strong>:</h4><p>Using European soccer data sets, which contained data related to common European soccer leagues, players’ basic information, and teams’ goals, etc., this project analyzes the characteristics of European soccer and players, explores data visualization regarding European soccer, and makes predictions of the results of matches. Based on Python 3 and some of the packages, such as NumPy, Matplotlib, Padas, and Sqlite3, the data set was improved to make it clear and user-friendly. Visualizations of data and basic statistics including Poisson Distribution were then used to determine the results. Finally, this project analyzed the average goals scored per league, the proportion of home team wins vs away team wins vs draws, the comparison of team attributes of effective teams to the rest of the teams, and predicts match results using Poisson distribution. Generally, this study analyzed data from leagues to matches to players.</p><h4><strong>Data Preprocessing:</strong></h4><p>The database ‘database.sqlite’ was connected to the Sql-Server via Jupyter notebook, and some essential data for the analysis were extracted, such as players, matches, leagues, countries, etc. The following steps are to process the data to make sure that it is clear, and can be perfectly fitted into the model.</p><pre># importing the necessary libraries<br>import numpy as np # linear algebra<br>import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)<br>import sqlite3 <br>import os<br>import matplotlib.pyplot as plt<br>import seaborn as sns<br>import ipywidgets as widgets<br>from ipywidgets import interact, interact_manual<br>from IPython.display import display, HTML<br>%matplotlib inline<br><br><br># connecting the database<br>conn = sqlite3.connect(&#39;database.sqlite&#39;)<br>c = conn.cursor()<br><br><br># printing all the tables present in database.sqlite <br>for row in c.execute(&quot;SELECT name FROM sqlite_master WHERE type=&#39;table&#39;;&quot;):<br>    print(list(row))</pre><pre># Read data into DataFrames from all the available tables<br><br>PlayerAttribute_df = pd.read_sql(&quot;Select * from Player_Attributes&quot;,conn)<br><br>Player_df = pd.read_sql(&quot;Select * from Player&quot;,conn)<br><br>Match_df = pd.read_sql(&quot;Select * from Match&quot;,conn)<br><br>League_df = pd.read_sql(&quot;Select * from League&quot;,conn)<br><br>Country_df = pd.read_sql(&quot;Select * from Country&quot;,conn)<br><br>Team_df = pd.read_sql(&quot;Select * from Team&quot;,conn)<br><br>TeamAttributes_df = pd.read_sql(&quot;Select * from Team_Attributes&quot;,conn)<br></pre><p>Upon finding the proportion of missing values present in the dataset, the table below was observed;</p><pre>#Function to find the proportion of missing values<br><br>def missing_values(input_df,null_percent=0):<br><br>    output_df = pd.DataFrame({&#39;missing_count&#39;:[],&#39;missing_prop&#39;:[]})<br>    nullcount_df = input_df.isna().sum()<br>    output_df[&#39;missing_count&#39;] = nullcount_df.iloc[0:]<br>    output_df[&#39;missing_prop&#39;] = output_df[&#39;missing_count&#39;]/len(input_df.index)*100<br>    output_df.index=nullcount_df.index<br>    if null_percent&gt;0:<br>        return output_df[output_df[&#39;missing_prop&#39;]&gt;=null_percent]<br>    else:<br>        return output_df</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*lvKWC7lVW2VvMqaaf-1wmg.png" /><figcaption><strong>Fig 1: Missing values</strong></figcaption></figure><h4>Query Analysis</h4><pre># Number teams each country have<br>query = pd.read_sql_query(<br>    &#39;&#39;&#39;<br>        SELECT <br>              c.name AS Country,<br>              COUNT(DISTINCT(team_long_name)) AS &#39;No. of Teams&#39;<br>              FROM Match AS m<br>              LEFT JOIN Country AS c<br>              ON m.country_id = c.id<br>              LEFT JOIN Team AS t <br>              ON m.home_team_api_id = t.team_api_id<br>              GROUP BY Country<br>    &#39;&#39;&#39;, conn<br>)<br>query</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/160/1*zrZGYDHS8TSqQpIVCdH9pg.png" /></figure><pre># total goals scored by home team grouped by country and Season<br>query1 = pd.read_sql_query(<br>    &#39;&#39;&#39;<br>        SELECT c.name AS Country,<br>               m.season AS Season,<br>               SUM(m.home_team_goal) AS &#39;Home Goal&#39;,<br>               SUM(m.away_team_goal) AS &#39;Away Goal&#39;<br>        FROM Match as m <br>        LEFT JOIN country AS c<br>        ON m.country_id = c.id<br>        GROUP BY Country, Season    <br>        ORDER BY Country     <br>    &#39;&#39;&#39;, conn<br>)<br>query1</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*Tp2Coz6AsUI-0zNNrw7a6A.png" /></figure><pre># goals scored by each team grouped by season<br>query2 = pd.read_sql_query(<br>    &#39;&#39;&#39;<br>        SELECT t.team_long_name AS Team, <br>               m.season as Season,<br>               SUM(m.home_team_goal) AS &#39;Home Goal&#39;,<br>               SUM(m.away_team_goal) AS &#39;Away Goal&#39;<br>        FROM Match AS m<br>        LEFT JOIN Team AS t<br>        ON m.home_team_api_id = t.team_api_id <br>        GROUP BY Team, Season<br>        ORDER BY Team<br>    &#39;&#39;&#39;, conn<br>)<br>query2</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/306/1*TpkJ8r8JxYqe8dh8JZ_yVg.png" /></figure><pre>query3 = pd.read_sql_query(<br>    &#39;&#39;&#39;<br>        SELECT c.name AS Country,<br>               t.team_long_name AS Team,<br>               m.season AS Season,<br>               SUM(m.home_team_goal) AS &#39;Home Goal&#39;,<br>               SUM(m.away_team_goal) AS &#39;Away Goal&#39;<br>        FROM Match as m<br>        LEFT JOIN Country AS c<br>        ON m.country_id = c.id<br>        LEFT JOIN Team AS t<br>        ON m.home_team_api_id = t.team_api_id<br>        GROUP BY Country, Team, Season<br>        ORDER BY Country<br>    &#39;&#39;&#39;, conn<br>)<br>query3</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/358/1*FpjAnQAgyLuaKzBmtOt1MQ.png" /></figure><pre># count of matches won, lost and tie<br>query4 = pd.read_sql_query(<br>    &#39;&#39;&#39;<br>    WITH sub_q AS (<br>        SELECT     <br>                   c.name AS Country,<br>                   t.team_long_name AS Team,<br>                   COUNT(CASE WHEN m.home_team_goal &gt; away_team_goal THEN &#39;Win&#39; END) AS Won,<br>                   COUNT(CASE WHEN m.home_team_goal &lt; away_team_goal THEN &#39;Lost&#39; END) AS Lost,<br>                   COUNT(CASE WHEN m.home_team_goal = away_team_goal THEN &#39;Draw&#39; END) AS Draw<br>         FROM Match AS m<br>         LEFT JOIN Country AS c<br>         ON m.country_id = c.id<br>         LEFT JOIN Team as t<br>         ON m.home_team_api_id = t.team_api_id<br>         GROUP BY Country, Team<br>         ORDER BY Country<br>         )<br>         SELECT ROW_NUMBER() OVER(ORDER BY Won DESC) AS &#39;Row Number&#39;,<br>                Country, <br>                Team, <br>                Won, <br>                Lost,<br>                Draw<br>        FROM sub_q<br>    &#39;&#39;&#39;, conn, index_col=&#39;Row Number&#39;<br>    )<br>query4</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/323/1*ZF650F9sToRyt6uXsEqTzw.png" /></figure><pre># total home goals scored grouped by country and team<br>query5 = pd.read_sql_query(<br>    &#39;&#39;&#39;<br>        SELECT name AS Name,<br>            team_long_name AS Team,<br>            --STRFTIME(&#39;%Y&#39;, date) AS Year, <br>            SUM(home_team_goal) AS Goal<br>        FROM Match AS m<br>        LEFT JOIN Country as c<br>        ON m.country_id = c.id<br>        LEFT JOIN Team AS t<br>        ON m.home_team_api_id = t.team_api_id  <br>        GROUP BY Name, Team<br>        ORDER BY Goal DESC<br>        <br>    &#39;&#39;&#39;, conn<br>)<br>query5</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/230/1*A2cDVcW6JgPuNuZ0BHyBfg.png" /></figure><pre># extracting Year, Month, Week and day from date<br>query7 = pd.read_sql_query(<br>    &#39;&#39;&#39;<br>        SELECT date AS Date,<br>               STRFTIME(&#39;%Y&#39;, date) AS Year,<br>               STRFTIME(&#39;%m&#39;, date) AS Month,<br>               STRFTIME(&#39;%w&#39;, date) AS Week,<br>               STRFTIME(&#39;%d&#39;, date) AS Day<br>        FROM Match<br>    &#39;&#39;&#39;, conn<br>)<br>query7</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/269/1*fkjDM_M5DbQmNVb-TZ2yHA.png" /></figure><pre># count of matches won, lost and tie<br>query8 = pd.read_sql_query(<br>    &#39;&#39;&#39;<br>    WITH sub_q AS (<br>        SELECT     <br>                   c.name AS Country,<br>                   season AS Season,<br>                   t.team_long_name AS Team,<br>                   --m.home_team_goal AS home_goal,<br>                   --m.away_team_goal AS away_goal,<br>                   COUNT(CASE WHEN m.home_team_goal &gt; away_team_goal THEN &#39;Win&#39; END) AS Won,<br>                   COUNT(CASE WHEN m.home_team_goal &lt; away_team_goal THEN &#39;Lost&#39; END) AS Lost,<br>                   COUNT(CASE WHEN m.home_team_goal = away_team_goal THEN &#39;Draw&#39; END) AS Draw<br>         FROM Match AS m<br>         LEFT JOIN Country AS c<br>         ON m.country_id = c.id<br>         LEFT JOIN Team as t<br>         ON m.home_team_api_id = t.team_api_id<br>         GROUP BY Country, Season<br>         ORDER BY Country<br>         )<br>         SELECT ROW_NUMBER() OVER(ORDER BY Won DESC) AS &#39;Row Number&#39;,<br>                Country, <br>                Season,<br>                Won, <br>                Lost,<br>                Draw<br>        FROM sub_q<br>    &#39;&#39;&#39;, conn, index_col=&#39;Row Number&#39;<br>    )<br>query8</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/280/1*EIQo3smR8cmt6Q2Eh33-2A.png" /></figure><h4><strong>EXPLORATORY DATA ANALYSIS</strong></h4><h4><strong>Average goals scored per league</strong></h4><pre>#Calculate Average Goals per League<br>LeagueAvgGoal_df = NewMatch_df.groupby([&#39;league&#39;]).mean().sort_values([&#39;total_no_goals&#39;],ascending=False).reset_index()<br><br>#Plot average goals scored per league<br>sns.set_style(&#39;white&#39;)<br>a=sns.catplot(kind = &#39;bar&#39;, x = &#39;total_no_goals&#39;, y = &#39;league&#39;, data = LeagueAvgGoal_df, ci = &#39;std&#39;, edgecolor=&#39;k&#39;)<br>a.fig.set_size_inches(12,8)<br>plt.xticks(fontsize = 12)<br>plt.yticks(fontsize = 12)<br>plt.xlabel(&#39;Average Goals Scored&#39;, fontsize = 15)<br>plt.ylabel(&#39;League&#39;, fontsize = 15)<br>plt.xlim(0,4)<br>plt.title(&#39;Avg. Goals scored per League&#39;,fontsize = 20)<br>for i in range(len(LeagueAvgGoal_df)):<br>    plt.text(LeagueAvgGoal_df[&#39;total_no_goals&#39;].iloc[i]+0.15,i+0.15, round(LeagueAvgGoal_df[&#39;total_no_goals&#39;].iloc[i],2), fontsize = 12)    </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*ME63ZrQnN1JnEirvkBvPyg.png" /><figcaption><strong>Fig 2: Average goals per league</strong></figcaption></figure><p>The plot above illustrated that the teams in the Netherlands Eredivisie league scored the most goals across any other 10 leagues all seasons. Teams in France’s Ligue 1 and Poland’s Ekstraklasa scored the least goals compared to any other league in all seasons. Therefore, we can draw the conclusion that teams have the best-attacking ability in the Dutch Eredivisie, and teams have the worst attacking ability in France’s Ligue 1 and Poland’s Ekstraklasa. Admittedly, the conclusion may have some flaws and it may be due to defensive abilities rather than offensive abilities. But the method and the trends are accurate with significant approximation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*rDaHzg_9JVi-ENyiCX6erQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*YqXCCNOMXHn2m-rX2T_zcg.png" /></figure><p>Figures 3 &amp; 4 illustrated that the home teams and away teams score the most goals in the Dutch Eredivisie than any other teams in the other ten leagues from the 2008/09 season to the 2015/16 season. Home teams in France’s Ligue 1 and Poland’s Ekstraklasa score the least goals compared to any other league from the 2008/09 season to the 2015/16 season. Also, the number of goals that home teams score is larger than that of goals that away teams score in all eleven leagues in Europe. Thus, we can connect these results to home advantages and away advantages. Home advantage is the psychological and physiological advantage that the home team has over the visiting team and it is prevalent in all sports, including soccer. We can properly speculate that home teams and away teams in the Netherlands Eredivisie had more home or away advantage (Bundesliga also had an away advantage). Home teams and away teams in Poland Ekstraklasa and France Ligue 1 had the worst home or away advantage.</p><h4><strong>The proportion of home team wins vs away team wins vs draws</strong></h4><pre>#Find the number of Home Team Wins, Away Team Wins &amp; Draws, by pivoting on League &amp; the match results and counting the number of matches played<br>#Also, sum the columns to find the total number of matches played in the League<br>LeagueProp=MatchMerged_df.pivot_table(index=&#39;league&#39;,columns=&#39;result&#39;,values=&#39;match_api_id&#39;,aggfunc=&#39;count&#39;)<br>LeagueProp[&#39;sum&#39;]=LeagueProp.sum(axis=1)<br><br><br>#Divide the match results columns with the sum column to find their proportions<br>LeagueProp=LeagueProp[[&#39;HTW&#39;,&#39;ATW&#39;,&#39;D&#39;]].divide(LeagueProp[&#39;sum&#39;],axis=0).multiply(100)<br><br>ax=LeagueProp.plot.barh(stacked=True,figsize=(10,8),width=0.75,color=sns.color_palette(&quot;hls&quot;, 8),edgecolor=&#39;w&#39;,alpha=0.8)<br><br>ax.legend([&#39;Home Team&#39;,&#39;Away Team&#39;,&#39;Draw&#39;],bbox_to_anchor=(1.2,1),loc=&#39;upper right&#39;)<br>plt.title(&#39;Predictability of an outcome in a League Match&#39;,fontsize=20)<br>plt.xlabel(&#39;Proportion of Home Team Wins vs Away Team Wins vs Draws&#39;,fontsize=15)<br>plt.ylabel(&#39;Leagues&#39;,fontsize=15)<br>plt.xticks(fontsize=12)<br>plt.yticks(fontsize=12)<br><br>for  i,j in enumerate(LeagueProp.index):<br>    plt.text(LeagueProp.loc[j,&#39;HTW&#39;]/2,i,str(round(LeagueProp.loc[j,&#39;HTW&#39;],2))+&#39; %&#39;,fontsize=12)<br>    plt.text(LeagueProp.loc[j,&#39;HTW&#39;]+LeagueProp.loc[j,&#39;ATW&#39;]/2,i,str(round(LeagueProp.loc[j,&#39;ATW&#39;],2))+&#39; %&#39;,fontsize=12)<br>    plt.text(LeagueProp.loc[j,&#39;HTW&#39;]+LeagueProp.loc[j,&#39;ATW&#39;]+LeagueProp.loc[j,&#39;D&#39;]/2,i,str(round(LeagueProp.loc[j,&#39;D&#39;],2))+&#39;%&#39;,fontsize=12)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/745/1*IxiAw96Tjrf-PqO252rObQ.png" /><figcaption><strong>Fig 5: Predictability of an Outcome in a League Match</strong></figcaption></figure><p>From figure 5, we observed that in every league the distribution of the Wins &amp; Draws in every league is approximately same, though the probability of Home Team winning is high.</p><h4><strong>Most Winning &amp; Losing Teams in History</strong></h4><pre>#Group the Match statistics on the basis of the Teams, and create new columns to calculate winning &amp; losing proportion<br>Consolidated_df=ScoreTable_df.groupby([&#39;team&#39;]).sum()<br>Consolidated_df[&#39;win_prop&#39;]=round(Consolidated_df[&#39;wins&#39;]/Consolidated_df[&#39;matches_played&#39;]*100,2)<br>Consolidated_df[&#39;loss_prop&#39;]=round(Consolidated_df[&#39;losses&#39;]/Consolidated_df[&#39;matches_played&#39;]*100,2)<br><br>#Sort based on columns win_prop &amp; loss_prop to find the Most Winning &amp; Losing teams respectively<br>MostWinningTeam_df=Consolidated_df.sort_values(&#39;win_prop&#39;,ascending=True).tail(10)[&#39;win_prop&#39;]<br>MostLosingTeam_df=Consolidated_df.sort_values(&#39;loss_prop&#39;,ascending=True).tail(10)[&#39;loss_prop&#39;]<br><br>plt.figure(figsize=(14,8))<br>plt.subplot(1,2,1)<br>MostWinningTeam_df.plot.barh(color=sns.color_palette(&#39;Greens&#39;, 10),edgecolor=&#39;k&#39;)<br>plt.title(&#39;Most Winning Teams in History&#39;,fontsize=20)<br>plt.xlim(0,100)<br>plt.xticks(fontsize=12)<br>plt.yticks(fontsize=12)<br>plt.xlabel(&#39;% of Wins&#39;,fontsize=15)<br>plt.ylabel(&#39;Teams&#39;,fontsize=15)<br>for i,j in enumerate(MostWinningTeam_df.index):<br>    plt.text(MostWinningTeam_df.loc[j]+1.5,i,str(MostWinningTeam_df.loc[j])+&#39;%&#39;,style=&#39;italic&#39;,fontsize=12)    <br><br>plt.subplot(1,2,2)<br>MostLosingTeam_df.plot.barh(color=sns.color_palette(&#39;Reds&#39;, 10),edgecolor=&#39;k&#39;)<br>plt.title(&#39;Most Losing Teams in History&#39;,fontsize=20)<br>plt.xlim(0,100)<br>plt.xticks(fontsize=12)<br>plt.yticks(fontsize=12)<br>plt.xlabel(&#39;% of Losses&#39;,fontsize=15)<br>plt.ylabel(&#39;&#39;)<br>for i,j in enumerate(MostLosingTeam_df.index):<br>    plt.text(MostLosingTeam_df.loc[j]+1.5,i,str(MostLosingTeam_df.loc[j])+&#39;%&#39;,style=&#39;italic&#39;,fontsize=12)<br><br>plt.subplots_adjust(wspace=0.5)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*DoXvuSMyHTPvTUZwzRawtA.png" /><figcaption><strong>Fig 6: Most Winning &amp; Losing Teams in History</strong></figcaption></figure><p>From Figure 6, it was observed that FC Barcelona (76.97%), Real Madrid CF (75%), and SL Benefica (74.6%) won most matches that they played. Whereas Pescara (73.68%), Le Havre AC (68.42%), and FC Penafiel (64.71%) lost most of their matches played.</p><h4><strong>Comparison of Team Attributes of Effective Teams to Rest of the Teams</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/1*zxoncKNMlJr62ZaJ14V0gQ.png" /><figcaption><strong>Figure 7: Teams Attribute Comparison</strong></figcaption></figure><p>From the above radar in Figure 7, we noticed defencePressure, buildUpPlayPassing, and ChanceCreationShooting are the top factors in determining a team’s effectiveness followed by chanceCreationPassing, chanceCreationShooting, and defenceAggression.</p><p>From the defense point of view, the effective teams had a higher defensive pressure &amp; aggression compared to the rest of the teams. In order words, the effective teams also had a high defensive line and the defenders were aggressive in terms of tackles, therefore, pressuring their opponents to commit tactical mistakes. From the buildup play point of view, the effective teams had a low buildup play passing when compared to the other teams. It was also observed that effective teams had a high chance of creating passes when compared to the rest of the teams.</p><h4>CONCLUSION</h4><p>I have successfully queried the dataset and had basic visualizations implemented. Here’s the link to the GitHub <a href="https://github.com/Whynnie/SQL-PROJECTS/tree/main/Exploring%20the%20European%20Soccer%20Database%20with%20SQL">repository</a>.</p><p>Thank you for walking through this with me. Happy learning!</p><h4><strong>REFERENCE</strong></h4><p>“Data Visualization Beginner’s Guide: A Definition, Examples, and Learning Resources” [Internet]. Available from:<strong> </strong><a href="https://www.tableau.com/lean/articles/data-visualization"><strong>https://www.tableau.com/lean/articles/data-visualization</strong></a><strong>.</strong></p><p><strong>Rein R, Memmert D.</strong> “Big data and tactical analysis in elite soccer: Future challenges and opportunities for sports science”. Springerplus 2016; 5(1): 1410.</p><p><strong>Katz A, Hayes A, Suresh T. </strong>Poisson distribution [Internet]. Available from:<strong> </strong><a href="https://brilliant.org/wiki/poisson-distribution/"><strong>https://brilliant.org/wiki/poisson-distribution/</strong></a><strong>.</strong></p><p><strong>Yiou Wang* (2020),</strong> “Data Analyses of European Soccer”, Volume-2 Issue-1.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a2669eae65bd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Credit Risk Modelling (Part II)]]></title>
            <link>https://medium.com/@whynniethegr8/credit-risk-modelling-part-ii-ea2b8d83cc9c?source=rss-e81cf5bbfe4------2</link>
            <guid isPermaLink="false">https://medium.com/p/ea2b8d83cc9c</guid>
            <category><![CDATA[financial-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[credit-risk]]></category>
            <category><![CDATA[fintech]]></category>
            <dc:creator><![CDATA[Winners Okebaram]]></dc:creator>
            <pubDate>Sat, 26 Nov 2022 20:50:11 GMT</pubDate>
            <atom:updated>2022-11-26T20:50:11.385Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*rKb5LixMqCrkJrS36vi5aA.jpeg" /></figure><p>We are all aware of, and keep track of, our credit scores, don’t we? That all-important number has been around since the 1950s and determines our creditworthiness. I suppose we all also have a basic intuition of how a credit score is calculated, or which factors affect it. Refer to <a href="https://medium.com/@whynniethegr8/credit-risk-modelling-part-i-39dea4e86f51">my previous article</a> for some further details on what a credit score is.</p><h3>Steps of PD Modeling</h3><p>In this article, we will go through detailed steps to develop a data-driven credit risk model in Python to predict the probabilities of default (PD) and assign credit scores to existing or potential borrowers.</p><ul><li>Data Preparation and Pre-processing</li><li>Feature Selection</li><li>Model Development</li><li>Model Validation</li></ul><p>We will determine credit scores using a highly interpretable, easy-to-understand, and implement scorecard that makes calculating the credit score a breeze.</p><p>I will assume a working Python knowledge and a basic understanding of certain statistical and credit risk concepts while working through this case study.</p><p>We have a lot to cover, so let’s get started.</p><h3>Data Preparation and Pre-processing</h3><p>The dataset used for this project contains all available data for more than 300,000 consumer loans issued from 2007 to 2015 by Lending Club: a large US peer-to-peer lending company. There are several different versions of this dataset. We have used a version available on <a href="http://kaggle.com/">kaggle.com</a>. You can find it <a href="https://www.kaggle.com/datasets/ethon0426/lending-club-20072020q1">here</a>.</p><pre># Reading the data<br>import numpy as np<br>import pandas as pd<br>loan_data = pd.read_csv(&quot;3.1 loan_data_2007_2014.csv&quot;, index_col= 0)<br>loan_data.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*XlB6gaFtSM0_gABZv_hIig.png" /><figcaption>Data</figcaption></figure><p>The concepts and overall methodology, as explained here, are also applicable to a corporate loan portfolio.</p><p>Initial data exploration reveals the following:</p><ul><li>18 features with more than 80% of missing values. Given the high proportion of missing values, any technique to impute them will most likely result in inaccurate results</li><li>Certain static features not related to credit risk, e.g., id, member_id, url, title</li><li>Other forward-looking features that are expected to be populated only once the borrower has defaulted, e.g., recoveries, collection_recovery_fee. Since our objective here is to predict the future probability of default, having such features in our model will be counterintuitive, as these will not be observed until the default event has occurred</li></ul><p>We will drop all the above features.</p><h4>Target Variable</h4><p>The target column in our dataset is loan status which has different unique values. These values will have to be transformed into binary. That is 0 for a bad borrower and 1 for a good borrower. The definition of a bad borrower in our case is one who falls under the following in our target column. <em>Charged off, Default, Late (31–120 days), Does not meet the credit policy. Status:Charged Off </em>The rest are classified as good borrowers.</p><pre># create a new column based on the loan_status column that will be our target variable<br>data[&#39;good_bad&#39;] = np.where(data.loc[:, &#39;loan_status&#39;].isin([&#39;Charged Off&#39;, &#39;Default&#39;, &#39;Late (31-120 days)&#39;,<br>                                                                       &#39;Does not meet the credit policy. Status:Charged Off&#39;]), 0, 1)<br># Drop the original &#39;loan_status&#39; column<br>data.drop(columns = [&#39;loan_status&#39;], inplace = True)</pre><p>The data was cleaned and preprocessed following the methodologies involved in credit risk modeling in a suitable format before developing any PD model. The link to the preprocessing GitHub repo can be found <a href="https://github.com/Whynnie/Customers-Credit-Risk-Analysis/blob/main/Customers&#39;%20Credit%20Risk%20-%20Preprocessing.ipynb">here</a>.</p><h4>Preliminary Data Exploration</h4><pre>loan_data_inputs_train = pd.read_csv(&#39;loan_data_inputs_train.csv&#39;, index_col = 0)<br>loan_data_targets_train = pd.read_csv(&#39;loan_data_targets_train.csv&#39;, index_col = 0)<br>loan_data_inputs_test = pd.read_csv(&#39;loan_data_inputs_test.csv&#39;, index_col = 0)<br>loan_data_targets_test = pd.read_csv(&#39;loan_data_targets_test.csv&#39;, index_col = 0)<br><br># Change display configuration of Pandas to display all columns, and a maximum of 100 rows for every DataFrame.<br>pd.options.display.max_rows = 100<br>pd.options.display.max_columns = None<br>loan_data_inputs_train.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*4YBqKrCn4J7RGvQwXE7iTA.png" /></figure><h4>Feature Selection</h4><p>Here<em> </em>we select a limited set of input variables in a new <em>dataframe</em>.</p><pre># Here we select a limited set of input variables in a new dataframe.<br>inputs_train_with_ref_cat = loan_data_inputs_train.reindex(columns = [&#39;grade:A&#39;,<br>&#39;grade:B&#39;,<br>&#39;grade:C&#39;,<br>&#39;grade:D&#39;,<br>&#39;grade:E&#39;,<br>&#39;grade:F&#39;,<br>&#39;grade:G&#39;,<br>&#39;home_ownership:RENT_OTHER_NONE_ANY&#39;,<br>&#39;home_ownership:OWN&#39;,<br>&#39;home_ownership:MORTGAGE&#39;,<br>&#39;addr_state:ND_NE_IA_NV_FL_HI_AL&#39;,<br>&#39;addr_state:NM_VA&#39;,<br>&#39;addr_state:NY&#39;,<br>&#39;addr_state:OK_TN_MO_LA_MD_NC&#39;,<br>&#39;addr_state:CA&#39;,<br>&#39;addr_state:UT_KY_AZ_NJ&#39;,<br>&#39;addr_state:AR_MI_PA_OH_MN&#39;,<br>&#39;addr_state:RI_MA_DE_SD_IN&#39;,<br>&#39;addr_state:GA_WA_OR&#39;,<br>&#39;addr_state:WI_MT&#39;,<br>&#39;addr_state:TX&#39;,<br>&#39;addr_state:IL_CT&#39;,<br>&#39;addr_state:KS_SC_CO_VT_AK_MS&#39;,<br>&#39;addr_state:WV_NH_WY_DC_ME_ID&#39;,<br>&#39;verification_status:Not Verified&#39;,<br>&#39;verification_status:Source Verified&#39;,<br>&#39;verification_status:Verified&#39;,<br>&#39;purpose:educ__sm_b__wedd__ren_en__mov__house&#39;,<br>&#39;purpose:credit_card&#39;,<br>&#39;purpose:debt_consolidation&#39;,<br>&#39;purpose:oth__med__vacation&#39;,<br>&#39;purpose:major_purch__car__home_impr&#39;,<br>&#39;initial_list_status:f&#39;,<br>&#39;initial_list_status:w&#39;,<br>&#39;term:36&#39;,<br>&#39;term:60&#39;,<br>&#39;emp_length:0&#39;,<br>&#39;emp_length:1&#39;,<br>&#39;emp_length:2-4&#39;,<br>&#39;emp_length:5-6&#39;,<br>&#39;emp_length:7-9&#39;,<br>&#39;emp_length:10&#39;,<br>&#39;mths_since_issue_d:&lt;38&#39;,<br>&#39;mths_since_issue_d:38-39&#39;,<br>&#39;mths_since_issue_d:40-41&#39;,<br>&#39;mths_since_issue_d:42-48&#39;,<br>&#39;mths_since_issue_d:49-52&#39;,<br>&#39;mths_since_issue_d:53-64&#39;,<br>&#39;mths_since_issue_d:65-84&#39;,<br>&#39;mths_since_issue_d:&gt;84&#39;,<br>&#39;int_rate:&lt;9.548&#39;,<br>&#39;int_rate:9.548-12.025&#39;,<br>&#39;int_rate:12.025-15.74&#39;,<br>&#39;int_rate:15.74-20.281&#39;,<br>&#39;int_rate:&gt;20.281&#39;,<br>&#39;mths_since_earliest_cr_line:&lt;140&#39;,<br>&#39;mths_since_earliest_cr_line:141-164&#39;,<br>&#39;mths_since_earliest_cr_line:165-247&#39;,<br>&#39;mths_since_earliest_cr_line:248-270&#39;,<br>&#39;mths_since_earliest_cr_line:271-352&#39;,<br>&#39;mths_since_earliest_cr_line:&gt;352&#39;,<br>&#39;delinq_2yrs:0&#39;,<br>&#39;delinq_2yrs:1-3&#39;,<br>&#39;delinq_2yrs:&gt;=4&#39;,<br>&#39;inq_last_6mths:0&#39;,<br>&#39;inq_last_6mths:1-2&#39;,<br>&#39;inq_last_6mths:3-6&#39;,<br>&#39;inq_last_6mths:&gt;6&#39;,<br>&#39;open_acc:0&#39;,<br>&#39;open_acc:1-3&#39;,<br>&#39;open_acc:4-12&#39;,<br>&#39;open_acc:13-17&#39;,<br>&#39;open_acc:18-22&#39;,<br>&#39;open_acc:23-25&#39;,<br>&#39;open_acc:26-30&#39;,<br>&#39;open_acc:&gt;=31&#39;,<br>&#39;pub_rec:0-2&#39;,<br>&#39;pub_rec:3-4&#39;,<br>&#39;pub_rec:&gt;=5&#39;,<br>&#39;total_acc:&lt;=27&#39;,<br>&#39;total_acc:28-51&#39;,<br>&#39;total_acc:&gt;=52&#39;,<br>&#39;acc_now_delinq:0&#39;,<br>&#39;acc_now_delinq:&gt;=1&#39;,<br>&#39;total_rev_hi_lim:&lt;=5K&#39;,<br>&#39;total_rev_hi_lim:5K-10K&#39;,<br>&#39;total_rev_hi_lim:10K-20K&#39;,<br>&#39;total_rev_hi_lim:20K-30K&#39;,<br>&#39;total_rev_hi_lim:30K-40K&#39;,<br>&#39;total_rev_hi_lim:40K-55K&#39;,<br>&#39;total_rev_hi_lim:55K-95K&#39;,<br>&#39;total_rev_hi_lim:&gt;95K&#39;,<br>&#39;annual_inc:&lt;20K&#39;,<br>&#39;annual_inc:20K-30K&#39;,<br>&#39;annual_inc:30K-40K&#39;,<br>&#39;annual_inc:40K-50K&#39;,<br>&#39;annual_inc:50K-60K&#39;,<br>&#39;annual_inc:60K-70K&#39;,<br>&#39;annual_inc:70K-80K&#39;,<br>&#39;annual_inc:80K-90K&#39;,<br>&#39;annual_inc:90K-100K&#39;,<br>&#39;annual_inc:100K-120K&#39;,<br>&#39;annual_inc:120K-140K&#39;,<br>&#39;annual_inc:&gt;140K&#39;,<br>&#39;dti:&lt;=1.4&#39;,<br>&#39;dti:1.4-3.5&#39;,<br>&#39;dti:3.5-7.7&#39;,<br>&#39;dti:7.7-10.5&#39;,<br>&#39;dti:10.5-16.1&#39;,<br>&#39;dti:16.1-20.3&#39;,<br>&#39;dti:20.3-21.7&#39;,<br>&#39;dti:21.7-22.4&#39;,<br>&#39;dti:22.4-35&#39;,<br>&#39;dti:&gt;35&#39;,<br>#&#39;mths_since_last_delinq:Missing&#39;,<br>&#39;mths_since_last_delinq:0-3&#39;,<br>&#39;mths_since_last_delinq:4-30&#39;,<br>&#39;mths_since_last_delinq:31-56&#39;,<br>&#39;mths_since_last_delinq:&gt;=57&#39;,<br>#&#39;mths_since_last_record:Missing&#39;,<br>&#39;mths_since_last_record:0-2&#39;,<br>&#39;mths_since_last_record:3-20&#39;,<br>&#39;mths_since_last_record:21-31&#39;,<br>&#39;mths_since_last_record:32-80&#39;,<br>&#39;mths_since_last_record:81-86&#39;,<br>&#39;mths_since_last_record:&gt;86&#39;<br>])<br><br># Here we store the names of the reference category dummy variables in a list.<br>ref_categories = [&#39;grade:G&#39;,<br>&#39;home_ownership:RENT_OTHER_NONE_ANY&#39;, <br>&#39;addr_state:ND_NE_IA_NV_FL_HI_AL&#39;,<br>&#39;verification_status:Verified&#39;,<br>&#39;purpose:educ__sm_b__wedd__ren_en__mov__house&#39;,<br>&#39;initial_list_status:f&#39;,<br>&#39;term:60&#39;,<br>&#39;emp_length:0&#39;,<br>&#39;mths_since_issue_d:&gt;84&#39;,<br>&#39;int_rate:&gt;20.281&#39;,<br>&#39;mths_since_earliest_cr_line:&lt;140&#39;,<br>&#39;delinq_2yrs:&gt;=4&#39;,<br>&#39;inq_last_6mths:&gt;6&#39;,<br>&#39;open_acc:0&#39;,<br>&#39;pub_rec:0-2&#39;,<br>&#39;total_acc:&lt;=27&#39;,<br>&#39;acc_now_delinq:0&#39;,<br>&#39;total_rev_hi_lim:&lt;=5K&#39;,<br>&#39;annual_inc:&lt;20K&#39;,<br>&#39;dti:&gt;35&#39;,<br>&#39;mths_since_last_delinq:0-3&#39;,<br>&#39;mths_since_last_record:0-2&#39;]</pre><p>From the data frame with input variables, we drop the variables with variable names in the list with reference categories.</p><pre>inputs_train = inputs_train_with_ref_cat.drop(ref_categories, axis = 1)<br>inputs_train.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/1*3h-v928zdC2R8LVEYNcByw.png" /></figure><h3>Model Deployment</h3><h4><strong>Building a Logistic Regression Model with P-Values</strong></h4><p>We need to assess which variables contribute to predicting borrower default. In our final model, we will keep only the relevant ones. We will use the logistic regression model to fit our training data. This model is widely used in credit risk modeling and can be used for large dimensions. It is easy to understand and interpret. The metric we will use for the evaluation of the model will be the Gini coefficient This metric is widely accepted by credit-scoring institutions.</p><p>For most statistical methods, the accepted approach is to check the Statistical significance of the coefficients of each dummy variable.<br>One of the most common ways to do that is to look at p-values</p><p>There are some built-in methods for calculating p-values in Scikit-Learn, but they are univariate. They only take into account each feature in the outcome as if there were no other features. In a regression model, the impact of all the features on the outcome is collective rather than independent. Therefore, the insights we get from such methods will be flawed.<br>The <strong><em>LogisticRegression</em> </strong>module does not have a built-in way to calculate these multivariate p-values. One of the cleanest ways to do this is to alter <em>.fit()</em> method from the <em>LogisticRegression</em> class itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/1*saho2pRYsQQBIWfb-xJz-g.png" /></figure><p>Creating an instance of an object from the newly created class ‘LogisticRegression_with_p_values()’ class.</p><pre>reg = LogisticRegression_with_p_values(max_iter=500)<br><br>## fit the input and the target features to the model<br>reg.fit(inputs_train, loan_data_targets_train)<br>feature_name = inputs_train.columns.values</pre><p>Creates a data frame with a column titled ‘Feature name’ and row values contained in the ‘feature_name’ variable.</p><pre>summary_table = pd.DataFrame(columns = [&#39;Feature name&#39;], data=feature_name)</pre><p>Creates a new column in the dataframe, called ‘Coefficients’, with row values the transposed coefficients from the ‘LogisticRegression’ object.</p><pre>summary_table[&#39;Coefficients&#39;] = np.transpose(reg.coef_)<br># Increases the index of every row of the dataframe with 1.<br>summary_table.index = summary_table.index + 1<br>summary_table.loc[0] = [&#39;Intercept&#39;, reg.intercept_[0]]<br># Sorts the dataframe by index.<br>summary_table = summary_table.sort_index()<br>summary_tablep</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/303/1*Dkuj3NDtI_fpsMo5s99nLg.png" /></figure><p>Take the result of the newly added method &#39;p_values&#39; and store it in a variable &#39;p_values&#39;.</p><pre>p_values = reg.p_values</pre><p>No p-value is calculated for the intercept, so we need to append an &#39;NaN&#39; value to the beginning of the p-values&#39; array to match the intercept with a null value:</p><pre># Add the intercept for completeness.<br># We add the value &#39;NaN&#39; in the beginning of the variable with p-values.<br><br>p_values = np.append(np.nan, np.array(p_values))</pre><p>Add a new column, ‘p_values’, to the ‘summary_table’ dataframe.</p><pre>summary_table[&#39;p_values&#39;] = p_values<br>summary_table</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/373/1*ro4pXiq4A5pOwrPG1GZing.png" /></figure><h4>Removing Statistically Insignificant variables</h4><p>Now that the p-values have been calculated for each input feature, the statistically insignificant variables, at a 5% significance level can be determined. The guidelines for removing the insignificant variables are:</p><ul><li>If all the dummy variables of an independent variable are statistically significant, they should all be INCLUDED in the final model.</li><li>If all the dummy variables of an independent variable are NOT statistically significant, they should all be EXCLUDED from the final model.</li><li>However, since each original variable is represented by several dummy variables; if one or a few dummy variables representing one original independent variable are statistically significant, it would be best to retain all dummy variables that represent that original independent variable.</li></ul><pre># Hypothesis : Ho: They are not significantly different<br>#              H1: They are significantly different<br><br># Decision: reject Ho if p-value &lt; 0.05<br><br># We are going to drop some features, the coefficients for all or almost all of the dummy variables for which,<br># are statistically insignificant (ie, their p_value &gt; 0.05).<br><br># We do that by specifying another list of dummy variables as reference categories, and a list of variables to remove.<br># Then, we are going to drop the two datasets from the original list of dummy variables.<br><br># return the statistically insignificant variables at 5% significance level.<br>summary_table.loc[summary_table[&#39;p_values&#39;] &gt; 0.05, :]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/310/1*IrSKcdj72TPkm72aeC72-Q.png" /></figure><p>Following the guidelines listed above, the following variables will be dropped:</p><ul><li>delinq_2yrs:0, delinq_2yrs:1-3 representing &quot;Delinquency in the Last Two Years&quot; were both removed because they are both statistically insignificant.</li><li>pub_rec:0-2, pub_rec:3-4, pub_rec:&gt;=5, representing &quot;Public Records&quot; were both removed because they are both statistically insignificant.</li><li>total_acc:&lt;=27, total_acc:28-51 representing &quot;Total Accounts&quot; were both removed because they are both statistically insignificant.</li><li>total_rev_hi_lim:&lt;=5K, total_rev_hi_lim:5K-10K,total_rev_hi_lim:10K-20K,total_rev_hi_lim:20K-30K,total_rev_hi_lim:30K-40K,total_rev_hi_lim:40K-55K,total_rev_hi_lim:55K-95K,total_rev_hi_lim:&gt;95K, - representing &quot;Total Revenue High Limit&quot; were all removed because they are only two of the seven dummy variables are statistically significant. Having only two dummy variables is not enough to retain all 7.</li><li>We kept the rest.</li></ul><pre># Therefore the Selected Variables<br><br>insignificant_feat = [&#39;delinq_2yrs:0&#39;, &#39;delinq_2yrs:1-3&#39;, &#39;pub_rec:3-4&#39;, &#39;pub_rec:&gt;=5&#39;, &#39;total_acc:28-51&#39;,  <br>                      &#39;total_acc:&gt;=52&#39;, &#39;total_rev_hi_lim:5K-10K&#39;, &#39;total_rev_hi_lim:10K-20K&#39;, <br>                      &#39;total_rev_hi_lim:20K-30K&#39;, &#39;total_rev_hi_lim:30K-40K&#39;, &#39;total_rev_hi_lim:40K-55K&#39;, <br>                      &#39;total_rev_hi_lim:55K-95K&#39;, &#39;total_rev_hi_lim:&gt;95K&#39;,]<br><br># drop the insignificant dummy variables<br>new_inputs_train = inputs_train_with_ref_cat.drop(insignificant_feat, axis = 1)<br>new_inputs_train.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/1*MAR-F675NH5fKSOWFStK0g.png" /></figure><h4>Model Training</h4><pre># Here we run a new model the newly selected variables.<br><br>reg2 = LogisticRegression_with_p_values()<br>reg2.fit(new_inputs_train, loan_data_targets_train)<br><br>feature_name = new_inputs_train.columns.values<br><br>summary_table = pd.DataFrame(columns = [&#39;Feature name&#39;], data = feature_name)<br>summary_table[&#39;Coefficients&#39;] = np.transpose(reg2.coef_)<br>summary_table.index = summary_table.index + 1<br>summary_table.loc[0] = [&#39;Intercept&#39;, reg2.intercept_[0]]<br>summary_table = summary_table.sort_index()<br>summary_table<br><br># We add the &#39;p_values&#39; here, just as we did before.<br>p_values = reg2.p_values<br>p_values = np.append(np.nan,np.array(p_values))<br>summary_table[&#39;p_values&#39;] = p_values<br>summary_table<br># Here we get the results for our final PD model.</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/1*hrxZX_nMkmFr5FmwfB-eCA.png" /><figcaption>Probability of Default Model</figcaption></figure><h4>Interpreting the coefficients in the PD model</h4><p>Direct comparisons are only possible between categories from one the same independent variable. Taking the <em>“Grade”</em> dummy variables as an example:</p><p><strong>Grade</strong>: Lending Club assigned loan grade. Values range from A (highest possible grade) to G (Lowest possible grade). The coefficients tell us that as the rating increases, the higher the probability of the borrower not defaulting on their loan. The values are monotonously increasing with the increase in rating.</p><p>Recall, we made “Grade G” the reference variable (because it had the least WoE; highest probability of default), and left it out of the model. Therefore, we can assume that its coefficient is equal to zero. This means that we can use it to compare the other dummy variables.</p><p>For instance, take “Grade D” with coef = 0.5167.<br>Hence, the probability of someone with a rating of D is better than someone with a rating of G is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*VQ0xmP249K7KQ2Q86ca4bQ.png" /></figure><p>Therefore, we can say that the odds of someone with a grade D-rating not defaulting on a loan are about 1.68 times greater than someone with a rating of G.</p><h3>Model Validation</h3><p>As stated in the strategy for preventing overfitting the model on the training data, the entire dataset was split into two (2) sets. The larger set (the train set) was used to train the model. To get a measure of the model’s true performance, the test set will be fed to the model as new input data.</p><p>In this section, we’ll explore these 2 criteria for evaluating the model performance of classification models</p><ol><li>Gini coefficient, and</li><li>Kolmogorov-Smirnov coefficient</li></ol><h4>Gini Coefficient</h4><p>In credit risk modeling, Gini Coefficient is used to measure the inequality between non-defaulted borrowers and defaulted borrowers within a population. Gini Coefficient is the percentage of the area above the secondary diagonal line enclosed between the concave curve and the secondary diagonal line. The greater the area, the better the model. It is measured by plotting the cumulative proportion of defaulted borrowers against the cumulative proportion of all borrowers.</p><blockquote><em>The Gini coefficient is a standard metric in risk assessment because the likelihood of default is relatively low. In the consumer finance industry, Gini can assess the accuracy of a prediction around whether a loan applicant will repay or default. A higher Gini is beneficial to the bottom line because requests can be assessed more accurately, which means acceptance can be increased and at less risk.</em></blockquote><p>The Gini coefficient can be calculated from the AUROC using the formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*UngYfxA51-ydpsiPzJcO_g.png" /><figcaption>Gini coefficient equation</figcaption></figure><pre># We plot the cumulative percentage of all along the x-axis and the cumulative percentage &#39;good&#39; along the y-axis,<br># thus plotting the Gini curve.<br>plt.plot(df_actual_predicted_probs[&#39;Cumulative Perc Population&#39;], df_actual_predicted_probs[&#39;Cumulative Perc Bad&#39;])<br><br># We plot a seconary diagonal line, with dashed line style and black color.<br>plt.plot(df_actual_predicted_probs[&#39;Cumulative Perc Population&#39;], df_actual_predicted_probs[&#39;Cumulative Perc Population&#39;], linestyle=&#39;--&#39;, color=&#39;k&#39;, label=&quot;Secondary diagonal&quot;)<br><br># We name the x-axis &quot;Cumulative % Population&quot;.<br>plt.xlabel(&#39;Cumulative % Population&#39;)<br><br># We name the y-axis &quot;Cumulative % Bad&quot;.<br>plt.ylabel(&#39;Cumulative % Bad&#39;)<br><br>plt.legend(loc=&quot;lower right&quot;)<br><br># We name the graph &quot;Gini&quot;.<br>plt.title(&#39;Gini Coefficient Curve&#39;, fontsize=15);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/1*nirzHpD0WwvuWBB2i7mcUw.png" /><figcaption>Gini Plot</figcaption></figure><p>From the Gini-Plot, the gini-coefficient is the proportion of the area enclosed by the diagonal line and the gini-curve is the area of the upper- triangle defined by the diagonal line.</p><p>Hence, the output for the Gini coefficient used in evaluating the model is indicated below.</p><pre>0.7070253267718734</pre><h4>Kolmogorov-Smirnov coefficient</h4><p>Kolmogorov-Smirnov coefficient is the maximum difference between the cumulative distribution functions of “good” and “bad” borrowers with respect to predicted probabilities. The greater this difference, the better the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/1*n5zHsEG2yf1355s8vxVKlA.png" /><figcaption>Kolmogorov-Smirnov Curve</figcaption></figure><p>From the Kolmogorov criterion, the result we got was about 0.3. The result isn’t too high but however, but it is significant from 0. Therefore, the two cumulative distribution functions are significantly far from each other and the model has satisfactory predictive power.</p><h3>Scorecard Development</h3><p>The final piece of our puzzle is creating a simple, easy-to-use, and implement credit risk scorecard that can be used by any layperson to calculate an individual’s credit score given certain required information about him and his credit history.</p><p>Remember the summary table created during the model training phase? We will append all the reference categories that we left out from our model to it, with a coefficient value of 0, together with another column for the original feature name (e.g., grade to represent grade:A, grade:B, etc.).</p><p>We will then determine the minimum and maximum scores that our scorecard should spit out. As a starting point, we will use the same range of scores used by FICO: from 300 to 850.</p><p>The coefficients returned by the logistic regression model for each feature category are then scaled to our range of credit scores through simple arithmetic. An additional step here is to update the model intercept’s credit score through further scaling which will then be used as the starting point of each scoring calculation.</p><p>At this stage, our scorecard will look like this (the Score-Preliminary column is a simple rounding of the calculated scores):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/0*DoEskM91DoRsrE-T.png" /></figure><p>Depending on your circumstances, you may have to manually adjust the Score for a random category to ensure that the minimum and maximum possible scores for any given situation remain between 300 and 850. Some trial and error will be involved here.</p><h3>Calculate Credit Scores for Test Set</h3><p>Once we have our final scorecard, we are ready to calculate credit scores for all the observations in our test set. Remember, our training and test sets are a simple collection of dummy variables with 1s and 0s representing whether an observation belongs to a specific dummy variable. For example, in the image below, observation 395346 had a C grade, owns its own home, and its verification status was Source Verified.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/0*H9QzLodoxr4CYUw7.png" /></figure><p>Accordingly, after making certain adjustments to our test set, the credit scores are calculated as a simple matrix dot multiplication between the test set and the final score for each category. Consider the above observations together with the following final scores for the intercept and grade categories from our scorecard:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/149/0*7dpq2hFty5jAVUI0.png" /></figure><p>Intuitively, observation 395346 will start with the intercept score of 598 and receive 15 additional points for being in the grade:C category. Similarly, observation 3766583 will be assigned a score of 598 plus 24 for being in the grade:A category. We will automate these calculations across all feature categories using matrix dot multiplication. The final credit score is then a simple sum of individual scores of each feature category applicable for an observation.</p><h3>Setting Loan Approval Cut-offs</h3><p>So how do we determine which loans should we approve and reject? What is the ideal credit score cut-off point, i.e., potential borrowers with a credit score higher than this cut-off point will be accepted and those less than it will be rejected? This cut-off point should also strike a fine balance between the expected loan approval and rejection rates.</p><p>To find this cut-off, we need to go back to the probability thresholds from the ROC curve. Remember that a ROC curve plots FPR and TPR for all probability thresholds between 0 and 1. Since we aim to minimize FPR while maximizing TPR, the top left corner probability threshold of the curve is what we are looking for. This ideal threshold is calculated using the <a href="https://en.wikipedia.org/wiki/Youden%27s_J_statistic">Youden’s J statistic</a> that is a simple difference between TPR and FPR.</p><p>The ideal probability threshold in our case comes out to be 0.187. All observations with a predicted probability higher than this should be classified as in Default and vice versa. At first, this ideal threshold appears to be counterintuitive compared to a more intuitive probability threshold of 0.5. But remember that we used the class_weight parameter when fitting the logistic regression model that would have penalized false negatives more than false positives.</p><p>We then calculate the scaled score at this threshold point. As shown in the code example below, we can also calculate the credit scores and expected approval and rejection rates at each threshold from the ROC curve. This can help the business to further manually tweak the score cut-off based on their requirements.</p><p>All the codes related to <a href="https://github.com/Whynnie/Customers-Credit-Risk-Analysis/blob/main/Customers&#39;%20Credit%20Risk%20-%20Model%20Validation.ipynb">scorecard development are here</a>!</p><h3>Conclusion</h3><p>This probability of default model can be further used in building application or behavioral scorecards, and further decision-making. The link to the Jupyter Notebook can be found <a href="https://github.com/Whynnie/Customers-Credit-Risk-Analysis/blob/main/Customers&#39;%20Credit%20Risk%20-%20Model%20Validation.ipynb">here</a> on GitHub. Feel free to play around with it or comment in case of any clarifications required or other queries.</p><p>As always, feel free to reach out to me on <a href="https://twitter.com/whynniethegr8"><strong><em>Twitter</em></strong></a><strong><em> &amp; </em></strong><a href="https://www.linkedin.com/in/winnersokebaram/"><em>L</em>inkedIn</a> if you would like to discuss anything related to data analytics and machine learning.</p><p>Till next time, happy learning!</p><h4>References</h4><p><a href="https://www.listendata.com/2019/08/credit-risk-modelling.html">https://www.listendata.com/2019/08/credit-risk-modelling.htm</a>l</p><ul><li><a href="https://www.listendata.com/2015/01/model-performance-in-logistic-regression.html">Check Predictive Model Performance</a></li><li><a href="https://www.listendata.com/2015/03/weight-of-evidence-woe-and-information.html">Weight of Evidence (WOE) and Information Value (IV) Explained</a></li></ul><p><a href="https://towardsdatascience.com/how-to-develop-a-credit-risk-model-and-scorecard-91335fc01f03">https://towardsdatascience.com/how-to-develop-a-credit-risk-model-and-scorecard-91335fc01f03</a></p><p>[1] Baesens, B., Roesch, D., &amp; Scheule, H. (2016). Credit risk analytics: Measurement techniques, applications, and examples in SAS. John Wiley &amp; Sons.</p><p>[2] Siddiqi, N. (2012). Credit risk scorecards: developing and implementing intelligent credit scoring. John Wiley &amp; Sons.</p><p>[3] Thomas, L., Edelman, D. &amp; Crook, J. (2002). Credit Scoring and its Applications.</p><p>[4] Mays, E. (2001). Handbook of Credit Scoring. Glanelake Publishing Company.</p><p>[5] Mironchyk, P. &amp; Tchistiakov, V. (2017). Monotone optimal binning algorithm for credit risk modeling.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ea2b8d83cc9c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Credit Risk Modelling (Part I)]]></title>
            <link>https://medium.com/@whynniethegr8/credit-risk-modelling-part-i-39dea4e86f51?source=rss-e81cf5bbfe4------2</link>
            <guid isPermaLink="false">https://medium.com/p/39dea4e86f51</guid>
            <category><![CDATA[risk-analysis]]></category>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[fintech]]></category>
            <category><![CDATA[financial-analysis]]></category>
            <category><![CDATA[fraud-detection]]></category>
            <dc:creator><![CDATA[Winners Okebaram]]></dc:creator>
            <pubDate>Thu, 24 Nov 2022 01:06:59 GMT</pubDate>
            <atom:updated>2022-11-27T11:38:09.964Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A walkthrough of statistical credit risk modeling, probability of default prediction, and credit scorecard development with Python</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/635/1*0hVBPgIkFipCRWj8XGnB0g.jpeg" /></figure><p>Most people working in the financial services industry, or otherwise are exposed to credit risk either as internal or external stakeholders, would have most likely come across this one way or another.</p><p>Credit risk occurs when the borrower in a debt contract defaults or delays in repaying the debt either in whole or part. Thus, credit risk involves the unwillingness or inability of a borrower to meet financial obligations about lending, hedging, trading, or any other financial transactions. Interestingly, credit risk modeling estimates the probability that the client will fail to pay you back, (Santo Mero, 1997).</p><p>Whether you are an individual, a startup, a small business owner, or the treasury head of a large entity, chances are that you would have heard of these concepts while applying for credit, equity fundraising, or some sort of internal risk assessment.</p><p>Credit risk is an important topic in the field of finance because banks and other financial institutions heavily invest in reducing their credit risk. The main reason behind the global financial crisis in 2008 was that mortgage loans were given to customers with poor credit scores.</p><p>Expectedly, it is of utmost importance for lenders to protect themselves against default and limit the risk they are exposed to. Credit risk modeling is the way to empirically ascertain how much risk they will be exposed to if a loan request is granted to a particular borrower. Once this risk is determined, the lenders can then decide to give out the loan if the credit risk does not exceed the acceptable thresholds.</p><p>However, a default can occur when a borrower is unable to make timely payments, misses payments, or avoids or stops making payments, James Chen (2021). The probability that a client will fail to pay back loans is called their default risk. Hence, a loan is in default when the lending agency is reasonably certain that the loan will not be paid, this results in a loss for the agency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/597/0*aLetY12ZF0Al-0W-.jpg" /></figure><p>Financial organizations are concerned about reducing the risk of default. As a result, commercial and investment banks, venture capital funds, asset management organizations, and insurance corporations, to mention a few, are increasingly depending on technology to anticipate which customers are most likely to default on their obligations.</p><p>Approving loans without proper scientific evaluation increases the risk of default. This can lead to the bankruptcy of lending agencies and consequently the destabilization of the banking system. This is what happened in the 2008 financial crisis which affected the world economy adversely.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*cfDxOMXI5mro1Ok1" /><figcaption>The 5c’s of Credit</figcaption></figure><h4>The 5c’s of Credit</h4><p>Credit risk can be measured by the “5 Cs of Credit”. It is a common phrase used to describe the five major judgmental factors used to determine a potential borrower’s creditworthiness. These 5 Cs of Credit incorporate both qualitative and quantitative financial measures, and the lender may analyze different documents, such as the borrower’s income statement, balance sheet, credit reports, and other documents that reveal the financial situation of the borrower, Thomas Brock (2021).</p><p>The 5 Cs of Credit refer to the Credit history [Character], Capacity to repay, Capital, the loan’s Conditions, and associated Collateral.</p><p><strong>a) Character:</strong> is the most comprehensive aspect of the evaluation of creditworthiness. It is the propensity for a borrower to repay a loan on time. Past defaults imply negligence or irresponsibility, which are undesirable character traits.</p><p><strong>b) Capacity:</strong> A borrower’s capacity to repay the loan is necessary for determining the lender’s risk exposure. One’s income amount, history of employment, and current job stability indicate the ability to repay outstanding debt. Other responsibilities, such as college-bound children or terminally ill family members, are also factored in to evaluate one’s future payment obligations.</p><p><strong>c) Capital:</strong> represents the overall pool of assets under the name of the borrower. It represents one’s investments, savings, and assets such as land, jewelry, etc. Loans are primarily repaid using overall household income; capital is additional security in case of unforeseen circumstances or setbacks such as unemployment.</p><p><strong>d) Conditions:</strong> refers to the specifics of any credit transaction, such as the principal amount or interest rate. Lenders assess risk based on how the borrower plans to use the money, should they receive it.</p><p><strong>e) Collateral:</strong> When assessed for a secured product such as a car or a home loan, borrowers must pledge certain assets under their name as collateral. This may include fixed assets such as the title of a parcel of land or financial assets. These features are not individualistic as they cannot be influenced by the borrower. Nevertheless, they indicate the level of risk associated with a certain investment.</p><h4>Expected Loss and Its Components</h4><p>The expected loss of a firm is the amount that the firm loses as a result of a default on a loan. The expected loss a firm will incur is composed of 3 components:</p><p><strong>a) Probability of default (PD)</strong>: the probability that a borrower will fail to make full repayment of the loan over a specified period, usually one year. (Which is our main interest in this project)</p><p><strong>b) Exposure at default (EAD):</strong> it is the outstanding amount of loss that a lender is exposed to at the given time of default.</p><p><strong>c) Loss-given default (LGD):</strong> the amount of money a lender stands to lose when a borrower defaults on the debt obligations. It is usually expressed as the percentage of the ED.</p><p>The expected loss is simply the product of these three components. Which is given as;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/524/1*8_rRpehanxHcjbUtMwBqVw.png" /><figcaption><strong>Expected Loss</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/0*x1JX6hgqjVQ-PmJJ" /></figure><h4><strong>Probability Of Default (PD)</strong></h4><p>The probability of default (PD) is an essential credit risk in the finance world. It provides an estimate of the possibility that a borrower will be unable to meet their debt obligations. In other words, it is the risk that the borrower will be unable or unwilling to repay its debt in full or part. The PD model must be easy to understand and interpret. Hence, it is modified into a simpler model called Score Cards.</p><p>Under Basel, I Accord (2004), a default event on a debt obligation is said to have occurred if;</p><p>i. It is unlikely that the obligor will be able to repay its debt to the bank without giving up any pledged collateral.</p><p>ii. The obligator is more than 90 days past due on a material credit obligation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/0*mfx_YdrNENxBs-nM.jpeg" /></figure><h4>Scorecards and Credit Scores</h4><p>A scorecard produces an individual creditworthiness assessment that directly corresponds to a specific probability of default. Because these creditworthiness assessments are named after the scorecards they are called Credit Scores.</p><p>A credit score is significant because it considers how often credit was used and how efficiently it was repaid. Credit scores are important for lenders because they reveal how capable an applicant is of taking on debt and repaying it in an efficient and timely way; this, in turn, reveals how risky it is for the lender to extend the applicant a loan or line of credit.</p><p>Companies that assemble and issue credit reports use different types of credit scores. The most common are FICO scores. Almost all lending institutions use the FICO scores to help determine how creditworthy an applicant is. FICO scores are also used to help determine the interest rate on any credit extended to an individual. FICO scores range from 300 (worst) to 850 (best).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*R3CF1QnjK1sjuSu4.gif" /></figure><h3>Credit Risk Modelling?</h3><p>A person’s credit risk is influenced by a variety of things. As a result, determining a borrower’s credit risk is a difficult undertaking. Credit risk modeling has entered the scene since there is so much money relying on our ability to appropriately predict a borrower’s credit risk.</p><p>Credit risk modeling is the practice of applying data models to determine two key factors. The first is the likelihood that the borrower will default on the loan. The second factor is the lender’s financial impact if the default occurs.</p><p>Credit risk models are used by financial organizations to assess the credit risk of potential borrowers.</p><blockquote><em>For companies involved in the financial system, preserving the financial health of clients is critical. However, you could be wondering how to protect each client’s financial well-being. The answer to this challenge entails assessing each client’s payment likelihood based on a set of criteria and devising tactics to anticipate customer wants.</em></blockquote><blockquote><em>As a result, the goal of this research is to forecast the likelihood of default on a specific obligation, in this example, credit cards. This will enable the creation of solutions that reduce the risk of the client’s financial health deteriorating. Furthermore, it is proposed to employ clustering techniques to locate homogeneous portions within the population and thus provide differentiated treatment to each client to assist in the creation of collection tactics.</em></blockquote><h3>Modeling Probability Of Default</h3><p>We are interested in the borrowers and respective loans if they have defaulted or not. Typically, the borrower is said to have defaulted if they are 90 days past a loan. A distinctive feature of the PD model is that all its independent / predictor variables must be categorical. The reason for this is that it is much easier to present the model in a simplified form and turn its features into a scorecard. The dependent variable should also be made binary; one class for a good loan and the other for a bad loan.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*JaEbhi-n1rS5VuD7.jpeg" /></figure><h4>Methodology</h4><p>Preprocessing the features is converting all variables to dummy variables. This is possible for only categorical variables/features. However, continuous variables have a different approach; we convert the continuous variables to dummy variables. This is done by converting them into categories of mini-sized intervals.</p><p>In reality, there’s a well-established methodology to turn continuous variables into categories. Once they are categorical, we proceed by building them up depending on their properties. These processes are called; Fine-Classing and Coarse Classing.</p><p><strong>a) Fine-Classing: </strong>It’s a technique that groups a variable’s value into several fine (equal-sized) bins. Through fine-classing, both discrete/continuous variables can be represented with categories. We start by getting the ability of each category to predict the dependent variable. ie; to check how the rate of good and bad loans varies across the categories. Therefore, the established metric that is used is the Weight of Evidence (WOE).</p><p><strong>Weight of Evidence (WOE)</strong>: shows the extent to which each of the different categories within an independent variable explains the dependent variable. In other words, it tells how much evidence the independent variable has concerning differences in the dependent variable. It also shows the extent to which each of the different categories of the independent variable explains the dependent variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pJn033EZAZ0CwxjV8Fs1LA.png" /><figcaption><strong>Weight of Evidence (WoE)</strong></figcaption></figure><p>The further the WoE is from 0, the better the category in differentiating the two categories of the response variable.</p><p><strong>b) Coarse Classing: </strong>this is a technique for applying a binning process to fine granular bins to merge those with similar risk and create fewer bins, usually up to 10. It’s the process of constructing new categories based on the initial ones.</p><p>Suppose there are k-categories of an independent variable, then we could calculate the WoE for each of the categories, then weigh the WoE for each category by;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*SelHtwojn1LpsgCvvDjHoA.png" /><figcaption><strong>Information Value (IV)</strong></figcaption></figure><p>This metric is called the <strong>Information Value (IV)</strong></p><p>The <strong>Information Value (IV) </strong>shows how much information the original independent variable brings concerning explaining the dependent variable. It identifies the independent variables which explain the response variable best. That is, we can use the IV for the preselection of predictors. The IV ranges from 0 ≤ IV ≤ 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*hhHE9qJcDYyViLl180RbDQ.png" /><figcaption><strong>Information Value Criterion Table</strong></figcaption></figure><h3><strong>Conclusion</strong></h3><p>We have briefly discussed the introduction to credit risk modeling and some methodologies involved. In the next article, we will be getting more practical by building customers’ probability of the default model. <a href="https://medium.com/@whynniethegr8/credit-risk-modelling-part-ii-ea2b8d83cc9c">Here’s the link</a></p><p>Stay tuned and happy learning!</p><h4>Follow me:</h4><p><a href="https://twitter.com/whynniethegr8"><strong><em>Twitter</em></strong></a></p><p><a href="https://www.linkedin.com/in/winnersokebaram/"><em>L</em>inkedIn</a></p><p><a href="https://github.com/Whynnie">GitHub</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39dea4e86f51" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Science Terms- Simply Explained]]></title>
            <link>https://medium.com/@whynniethegr8/data-science-terms-simply-explained-334e7a2e9141?source=rss-e81cf5bbfe4------2</link>
            <guid isPermaLink="false">https://medium.com/p/334e7a2e9141</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Winners Okebaram]]></dc:creator>
            <pubDate>Tue, 13 Sep 2022 16:53:46 GMT</pubDate>
            <atom:updated>2022-09-18T23:44:31.113Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Data Science is one of the most exciting fields of the 21st century. The power to understand human voices, recognize faces, understand text, and much more has vast applications across industries. In this article, we’re going to define some of the most important terms in more detail.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*22qR9WquFu9pJSl__VcdkA.jpeg" /></figure><h3>Data Science</h3><p>Data Science is a multifocal field, consisting of an intersection of Mathematics, Statistics, Computer Science, &amp; Domain Specific Knowledge of a specific field.</p><p>It is the application of Mathematics and Statistics to real-world problems to solve them faster using computers. It involves Software Development (the software which solves the problem), Machine Learning (to train the machine using mathematics), and Traditional Research (to make mathematical assumptions about the problem).</p><p>Data Science is applied to those problems where traditional algorithms will take too long to solve, or won’t be able to solve them as accurately. For example, predicting an area’s housing prices, controlling an autonomous vehicle’s movements, recognizing human faces, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/0*N5jokIeBY_xI4TLH.png" /><figcaption>The Data Science Venn Diagram</figcaption></figure><h4><strong>1. Dataset</strong></h4><p>A dataset is a particular instance of data that is used for analysis or model building at any given time. A dataset comes in different flavors: numerical data, categorical data, text data, image data, voice data, and video data. For beginning data science projects, the most popular type of dataset is a dataset containing numerical data that is typically stored in a comma-separated values (CSV) file format</p><h4>2. Data Wrangling</h4><p>Data wrangling is the process of converting data from its raw form to a tidy form ready for analysis. Data wrangling is an important step in data preprocessing and includes several processes like data importing, data cleaning, data structuring, string processing, HTML parsing, handling dates and times, handling missing data, and text mining.</p><h4>3. Data Visualization</h4><p>It is one of the main tools used to analyze and study relationships between different variables. Data visualization (e.g., scatter plots, line graphs, bar plots, histograms, Q-Q plots, smooth densities, boxplots, pair plots, heat maps, etc.) can be used for descriptive analytics. Data visualization is also used in machine learning for data preprocessing and analysis, feature selection, model building, model testing, and model evaluation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MAmSA1bbnHwZNWY5Q7v5jg.png" /><figcaption>Data Visualization</figcaption></figure><h4>4. Outliers</h4><p>An outlier is a data point that is very different from the rest of the dataset. Outliers are very common and are expected in large datasets. One common way to detect outliers in a dataset is by using a box plot. Outliers can significantly degrade the predictive power of a machine learning model.</p><p>Advanced methods for dealing with outliers include the RANSAC method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*E1YMlWSYsNvWsm5JLkH1Ag.jpeg" /></figure><h4>5. Data Imputation</h4><p>Most datasets contain missing values. However, the removal of samples or the dropping of entire feature columns is simply not feasible because we might lose too much valuable data. So, here we can use different interpolation techniques to estimate the missing values from the other training samples in our dataset. One of the most common interpolation techniques is mean imputation, where we simply replace the missing value with the mean value of the entire feature column.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7aj0qx4XHrnD4YThyqEoTw.jpeg" /><figcaption>Data imputation structure</figcaption></figure><h4>6. Data Scaling</h4><p>Scaling your features will help improve the quality and predictive power of your model. The model will be biased towards a particular feature without scaling your features. To bring features to the same scale, we could decide to use either normalization or standardization of features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*owQm4uXNJXiNJUfZkGgelQ.png" /><figcaption>Data scaling</figcaption></figure><h4>7. Data Partitioning</h4><p>In machine learning, the dataset is often partitioned into training and testing sets. The model is trained on the training dataset and then tested on the testing dataset. The testing dataset thus acts as the unseen dataset, which can be used to estimate a generalization error (the error expected when the model is applied to a real-world dataset after the model has been deployed).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/1*kdlyA8UDoKTpwRqjDWhTVA.jpeg" /><figcaption>Data partitioning</figcaption></figure><h4>8. Artificial Intelligence (AI)</h4><p>Artificial Intelligence is a term that is sometimes incorrectly used to refer to all Data Science problems. AI is referring to computer systems or programs that are intelligent enough to be able to complete tasks that generally require human intervention.</p><p>A computer system in AI is known as an “intelligent agent”, which takes input from the environment and performs actions that maximize the chances of achieving its goals. Improving the learning, knowledge representation, perception, and object manipulation capability of these “intelligent agents” are the short-term goals of Artificial Intelligence.</p><p>Artificial Intelligence is further subdivided into <a href="https://hackernoon.com/general-vs-narrow-ai-3d0d02ef3e28">Narrow AI &amp; General AI</a>. Narrow AI is only capable of doing a single task well such as understanding human speech, playing strategic games like Chess or Go, or operating vehicles autonomously. In contrast, a General AI should be capable of completing any task given to it, like a human. The long-term goal of AI is to create Artificial General Intelligence (AGI).</p><h4>9. Machine Learning (ML)</h4><p>Considered a subset of Artificial Intelligence, Machine Learning is the application of statistical models and algorithms to solve problems by finding patterns and inferences.</p><p>Given the large amount of data being generated in modern times, the objective of ML is for machines to be able to learn from the data itself, without human intervention or assistance. Data is the fundamental basis for all Machine Learning. If there was very little data or no data at all, the ML algorithms wouldn’t be able to derive useful inferences.</p><p>Therefore, in Machine Learning, the algorithm learns to solve the problem by itself using data, contrasting to a traditional program, where a programmer has to explicitly write a set of instructions. Applications of Machine Learning are in building recommender systems, clustering data points with similar characteristics together (for example, clustering customer data into market segments), or predicting future values based on past data (for instance, predicting stock or market price).</p><p>Machine Learning can either be Supervised, Unsupervised, or Reinforced.</p><p><strong>a) Supervised Learning: </strong>These are machine learning algorithms that perform learning by studying the relationship between the feature variables and the known target variable. Supervised learning has two subcategories:</p><ul><li>Continuous Target Variables: Linear Regression, Neighbors regression (KN), and Support Vector Regression (SVR).</li><li>Discrete Target Variables: Logistic Regression classifier, Support Vector Machines (SVM), Decision tree classifier</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/1*tvdxezeUM2ncnVr0RWH85w.jpeg" /><figcaption>Supervised Learning</figcaption></figure><p><strong>b) Unsupervised Learning: </strong>In unsupervised learning, we deal with unlabeled data or data of unknown structure. Using unsupervised learning techniques, we can explore the structure of our data to extract meaningful information without the guidance of a known outcome variable or reward function.</p><p>K-means clustering is an example of an unsupervised learning algorithm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/1*Lx4WugPgHkSPY7QU4eEjWQ.jpeg" /><figcaption>Unsupervised Learning</figcaption></figure><p><strong>c) Reinforcement Learning: </strong>Reinforcement Learning (RL) is a type of machine learning technique that enables an agent to learn in an interactive environment by trial and error using feedback from its actions and experiences.</p><p>Reinforcement learning uses rewards and punishment as signals for positive and negative behavior.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/1*lOubDpeubnoZtNG88ZQTSw.jpeg" /><figcaption>Reinforcement Learning</figcaption></figure><h4>10. Deep Learning (DL)</h4><p>Deep Learning is the application of Artificial Neural Networks to imitate the workings of the human brain. It is considered a subset of Machine Learning because it uses data to learn features, inferences, and patterns automatically.</p><p>Unlike in Machine Learning, feature engineering, which is an important and difficult step, is automatically taken care of in Deep Learning. In Deep Learning, “deep” layers of Artificial Neural Networks are used to progressively extract higher-level features from raw data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*2mvWqq_ItQ_nGuqLC-kQJw.png" /></figure><h4>11. Cross-validation</h4><p>Cross-validation is a method of evaluating a machine learning model’s performance across random samples of the dataset. In k-fold cross-validation, the dataset is randomly partitioned into training and testing sets. The model is trained on the training set and evaluated on the testing set. The process is repeated “k” times. The average training and testing scores are then calculated by averaging over the k-folds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*rsyiopNW4rrs8MA7yCqynw.jpeg" /><figcaption>Cross-validation</figcaption></figure><h4>12. Bias-variance Tradeoff</h4><p>A model having high bias and low variance assumes more assumptions about the form of the target function, and a model having high variance and low bias overlearns the training dataset. The parameters of the model should be tuned to get the best fit model, that performs the best in production.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*h4XETbo3hFx9PtKKJH90Dg.jpeg" /><figcaption>Bias-variance tradeoff</figcaption></figure><h4>13. Principal Component Analysis (PCA)</h4><p>Principal Component Analysis (PCA) is a statistical method that is used for feature extraction. It is used for high-dimensional and correlated data. The basic idea of PCA is to transform the original space of features into the space of the principal component.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AGm7H5hp32qTH4hoOhamcw.jpeg" /><figcaption>PCA</figcaption></figure><h4>14. Linear Discriminant Analysis (LDA)</h4><p>Linear Discriminant Analysis is a dimensionality reduction technique that is commonly used for supervised classification problems. It is used for modeling differences in groups i.e. separating two or more classes. Just like the PCA, It is used to project the features in higher dimension space into a lower dimension space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*ankFs3wCAVxTJOaYy4JXKA.jpeg" /></figure><h4>15. Model Parameters and Hyper-parameters</h4><p><strong>a) Model Parameters: </strong>These are the parameters in the model that must be determined using the training data set. These are the fitted parameters.</p><p>For example, suppose we have a model such as;</p><blockquote><em>house price = a + b* (age) + c* (size)</em></blockquote><p>To estimate the cost of houses based on the age of the house and its size (square foot), then a, b, and c will be our model or fitted.</p><p><strong>b) Hyperparameters: </strong>These are adjustable parameters that must be tuned to obtain a model with optimal performance. Some examples of hyperparameters in machine learning: Learning Rate, Number of Epochs, Regularization constant, Number of branches in a decision tree, Number of clusters in a clustering algorithm (like k-means)</p><p>It is important that during training, the hyperparameters be tuned to obtain the model with the best performance (with the best-fitted parameters)</p><h4>16. Evaluation Metrics</h4><p>In machine learning (predictive analytics), several metrics can be used for model evaluations. A supervised learning (discrete target) model, also referred to as a classification model, can be evaluated using metrics such as accuracy, precision, recall, f1 score, and the area under the ROC curve (AUC).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*QAIkkEixkNIBZrZj_z8TEw.png" /><figcaption>Evaluation metrics</figcaption></figure><h4>17. Math Concepts</h4><p><strong>a) Basic Calculus</strong>: Most machine learning models are built with a dataset having several features or predictors. Hence, familiarity with multivariable calculus is extremely important for building a machine learning model. Here are the topics you need to be familiar with: Functions of several variables; Derivatives and gradients; Step function, Sigmoid function, Logit function, ReLU (Rectified Linear Unit) function; Cost function; Plotting of functions; Minimum and Maximum values of a function</p><p><strong>b)</strong> <strong>Basic Linear Algebra:</strong> Linear algebra is the most important math skill in machine learning. A data set is represented as a matrix. Linear algebra is used in data preprocessing, data transformation, dimensionality reduction, and model evaluation. Here are the topics you need to be familiar with: Vectors; Norm of a vector; Matrices; Transpose of a matrix; The inverse of a matrix; The determinant of a matrix; Trace of a Matrix; Dot product; Eigenvalues; Eigenvectors</p><p><strong>c) Optimization Methods: </strong>Most machine learning algorithms perform predictive modeling by minimizing an objective function, thereby learning the weights that must be applied to the testing data to obtain the predicted labels. Here are the topics you need to be familiar with: Cost function/ Objective function; Likelihood function; Error function; Gradient Descent Algorithm and its variants (e.g., Stochastic Gradient Descent Algorithm)</p><h4>18. Statistics and Probability Concepts</h4><p>Statistics and Probability are used for the visualization of features, data preprocessing, feature transformation, data imputation, dimensionality reduction, feature engineering, model evaluation, etc. Here are the topics you need to be familiar with: Mean, Median, Mode, Standard deviation/ variance, Correlation coefficient, the covariance matrix, Probability distributions (Binomial, Poison, Normal), p-value, Bayes Theorem (Precision, Recall, Positive Predictive Value, Negative Predictive Value, Confusion Matrix, ROC Curve), Central Limit Theorem, R_2 score, Mean Square Error (MSE), A/B Testing, Monte Carlo Simulation</p><h4>19. Regularization</h4><p>Regularization is a technique used to reduce errors by fitting the function appropriately on the given training set and avoiding overfitting. The commonly used regularization techniques are;</p><p><strong>a) L1 regularization:</strong> LASSO (Least Absolute Shrinkage and Selection Operator) Regression adds “absolute value of magnitude” of coefficient as penalty term to the loss function(L).</p><p><strong>b) L2 regularization</strong>: Ridge Regression adds “squared magnitude” of coefficient as penalty term to the loss function (L)</p><p><strong>c) Dropout regularization: </strong>Dropout is a regularization technique, which aims to reduce the complexity of the neural network algorithm to prevent overfitting. As a consequence, the neural network will learn different, redundant representations; the network can’t rely on the particular neurons and the combination (or interaction) of these to be present. Another nice side effect is that the training process will be faster.</p><h3>Conclusion</h3><p>We have defined some of the typical Data Science terms and hope that the terminologies are getting more clear with the explanations. There are many more terms, which we will discuss in future articles, such as:</p><ul><li>Natural Language Processing</li><li>Neural Networks (CNN, RNN)</li><li>Computer Vision</li><li>Big Data</li><li>Cloud Computing</li><li>Active Learning</li><li>and more…</li></ul><h4>Follow me:</h4><p><a href="https://twitter.com/whynniethegr8"><strong><em>Twitter</em></strong></a></p><p><a href="https://www.linkedin.com/in/winnersokebaram/"><em>L</em>inkedIn</a></p><p><a href="https://github.com/Whynnie">GitHub</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=334e7a2e9141" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>