<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Suhaib  Ali Kamal on Medium]]></title>
        <description><![CDATA[Stories by Suhaib  Ali Kamal on Medium]]></description>
        <link>https://medium.com/@suhaibkamal93?source=rss-95b354a96834------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*_8X7RaPH3coffT1DB_tlAg.jpeg</url>
            <title>Stories by Suhaib  Ali Kamal on Medium</title>
            <link>https://medium.com/@suhaibkamal93?source=rss-95b354a96834------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 16:57:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@suhaibkamal93/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Survival Analysis-A theoretical perspective]]></title>
            <link>https://suhaibkamal93.medium.com/survival-analysis-a-theoretical-perspective-6c224dddb760?source=rss-95b354a96834------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c224dddb760</guid>
            <category><![CDATA[survival-analysis]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Suhaib  Ali Kamal]]></dc:creator>
            <pubDate>Tue, 08 Mar 2022 19:08:57 GMT</pubDate>
            <atom:updated>2022-03-12T05:13:47.180Z</atom:updated>
            <content:encoded><![CDATA[<p>Survival Analysis is a branch of machine statistics where the outcome variable is the time it takes until an event occurs. For example, a study in which patients afflicted by a virus or a life-ending disease are tested to determine the survival time depending one the features of the patient eg age, gender, other related diseases etc. Survival analysis is also applied in other domains such as mechanical engineering in assessing system reliability and in marketing to assess how customers churn in a subscription.</p><p>However, while it is very similiar to a regression problem there is one primary difference . A lot of the subjects in the study might not experience the event. For example in a customer churn example a lot of the customers might still be customers and using the service. In survival analysis , this is called censored data.</p><p>Censoring can come through multiple type:</p><ol><li><strong>Right Censoring: </strong>In this process the event time is as large as the observed time. For example, in a study of customers going through a life threatening disease, some customers will be able to survive through the experiement and in this case the event will not have occurred.</li><li><strong>Left Censoring</strong>: In this type of experiment, the observed time will be as larger than the event time. For example a study of pregnant women where the study time is more than 9 months.</li><li><strong>Interval Censoring: </strong>This is where we do not know the event time but we do not know that it falls in some interval. For example surveying a patient once a month in order to assess whether the event has occurred or not.</li></ol><p>The survival function basically returns the probability distribution of an event occurring after time <em>t</em>. Mathematically it can be represented as</p><p>S(t)=P(T&gt;t)</p><p>where the survival function S(t) gives us the probability function of a subject surviving after time T. A typical survival curve is downward sloping indicating that as time t increases the probability of survival decreases. On a practical level, it is very intuitive as generally as the time passes the probability of a patient from a disease or a customer churning reduces.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*cSY-Ziqc9TxCkHwUeldzZw.jpeg" /></figure><p>One could also argue that one could also compute the probability of the patient dying at time t however we cant do that because of censoring . Censoring does not allow us to compute the probability of an event happening so instead we calculate the probability of survival at a specified time.</p><p><strong>Hazard Function</strong></p><p>The hazard function is opposed to the survival function. It is the event rate. The hazard at time t is the potential per time for the event to occur.</p><p><strong>Mean &amp; Median Survival Time</strong></p><p>The mean survival time of a subject can be calculated by the area under the survival curve from zero to survival time. It could be calculated by taking the integral of the survival function with respect to time.</p><p>The median survival time is the time at which the survival probability is 0.5.</p><h3>Estimating the probability distribution</h3><p>There are multiple non-parametric &amp; parametric approaches of modelling the distribution of a survival function. .</p><p><strong>Weibull distribution</strong></p><p>One of the most popular approaches is Weibull distribution which are ideal for modelling the failure of machines. A sample Weibull distribution looks like the below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/513/1*gNk7ZQXvbsQZSj4QQGba-g.gif" /></figure><p>A Weibull distribution allows us to extend the function in an exponentially by changing its parameter. A b&lt;1 indicates that the hazard rate is expected to fal while a b&gt;1 implies that the hazard rate is expected to increase.</p><p><strong>Exponential Distribution</strong></p><p>In an exponential distribution the hazard rate is expected to be constant. Mathematically it can be represented as follows:</p><p><strong>Hazard Function </strong>: h(t)=p</p><p><strong>Survival function: </strong>S(t)=e^-pt</p><p><strong>Density Function: </strong>pe^pt</p><p>An exponential distribution is ideal where the event rate is constant over time.</p><h3><strong>Comparision of models</strong></h3><p>One of the most common ways to assess the accuracy of a survival model is the <strong>Brier score</strong>. The Brier score gives the accuracy of any probability metric compared to the actual. Mathematically it can be represented as below:</p><p>Brier Score=1/n * Σ(ft — ot)2</p><p>where</p><p>n=sample size</p><p>f=forecasted probability</p><p>t=time</p><p>o=original probability.</p><p>However, the problem with this metric is that it takes a lot of time to train. Another useful metric in assessing could be the <strong>AIC</strong>(Akaike Information Criterion) mathematically represented as</p><p>AIC=-2logL+2k</p><p>where L=log-likelihood of the model</p><p>and k=number of parameters</p><h3><strong>Non-parametric methods</strong></h3><h4>Kaplan Meier Curve</h4><p>One of the most common modelling approaches in the Kaplan Meier Estimator.</p><p>For survival analysis we assess the variable in this case we will define it as Y as the minimum of the event time or the observed time. In other words</p><p>Y=min(E,C)</p><p>where E is the event time and C is the censoring time.</p><p><strong>Log-Rank test</strong></p><p>Suppose we need to see whether there is a difference between the survival distributions of two groups. For example in an experiment we might need to see whether the survival distributions are different for a control or a treatment group. We could also perform a t-test however as explained before we could run into problems due to censoring.</p><p>A log-rank test is a large sample chi-square test that uses a chi-square statistic to compare two or more KM curves. The log-rank test can be mathematically represented as below.</p><p>W = (X − E(X) )/Var(X).</p><p><strong>Other topics</strong></p><p>In the next few writeups we will perform Survival analysis on python and will also discuss the Cox’s proportional hazards model. Stay tuned</p><p><strong>References</strong></p><blockquote><em>Gareth J., Daniela W., Trevor H. &amp; Robert T. </em><strong>An Introduction to Statistical Learning: with Applications in R (Springer Texts in Statistics)</strong><em>. Springer.</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c224dddb760" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predicting customer churn for a telco]]></title>
            <link>https://suhaibkamal93.medium.com/predicting-customer-churn-for-a-telco-70a821a84ce8?source=rss-95b354a96834------2</link>
            <guid isPermaLink="false">https://medium.com/p/70a821a84ce8</guid>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[customer-churn]]></category>
            <dc:creator><![CDATA[Suhaib  Ali Kamal]]></dc:creator>
            <pubDate>Mon, 21 Jun 2021 03:17:37 GMT</pubDate>
            <atom:updated>2021-06-21T03:17:37.625Z</atom:updated>
            <content:encoded><![CDATA[<p>Customer churn is a term associated to those customers who is lost after purchase. The cost to acquire a customer is significant enough for companies to analyse and understand the reasons behind the churn.</p><p>For this exercise we are gong to use a dataset from a telecom company he data can be found on the following link.<a href="https://www.kaggle.com/blastchar/telco-customer-churn">https://www.kaggle.com/blastchar/telco-customer-churn</a>.</p><p>The dataset has close to 21 columns with 18 independent variables , a customer id variable and a dependent variable in the form of customer churn.The first part of the code is to analyse the data and see the data types of the columns</p><pre>df=pd.read_csv(&#39;Customer Churn.csv&#39;)<br>df.head(5)</pre><p>Almost all of the variables in the dataset are categorical variables barring the total charges and the monthly charges.We can first check whether there are any missing values</p><pre>df.isnull().sum()</pre><p>After confirming that there are no missing values we will perform EDA to identify any relationships or patterns in the data. Let us first try to find the distribution of the tenure of the telco customers.</p><pre>sns.boxplot(y=df[&quot;tenure&quot;],x=df[&quot;Churn&quot;]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/1*IJ-d7Ky4drn9Bx4kkuFm8A.png" /><figcaption>Tenure vs Churn</figcaption></figure><p>Let us further analyse the type of services telco customers avail and how does it relate to their churn rates.Let us first see the relationship between the type of Internet service and their churn rates.</p><pre>internet=pd.crosstab(df[&quot;InternetService&quot;],df[&quot;Churn&quot;],normalize=&quot;columns&quot;).plot(kind=&quot;bar&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/1*XOiXa0mJK-KiR9hN_kFr2Q.png" /><figcaption>Intenet coonnection vs churn rates</figcaption></figure><p>As can be seen from the title, people with a fibre optic connection are far more likely to churn than people with a DSL connection. More importantly we can also deduce that the type of Internet service has asignificant impact on the customer churn rate.Let us see how the duration of the contact affects customer churn</p><pre>internet=pd.crosstab(df[&quot;Contract&quot;],df[&quot;Churn&quot;],normalize=&quot;columns&quot;).plot(kind=&quot;bar&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/376/1*RGzMCFHrB-hrhFwWo6iGCA.png" /><figcaption>Contract duration vs churn</figcaption></figure><p>People with longer contacts have less churn as opposed to people who sign up for month-to month contracts.</p><h3><strong>Feature Engineering</strong></h3><p>For this section we will have to generate additional columns for our categorical variables in the shape f dummy variables.</p><pre>df[&#39;TotalCharges&#39;]=pd.to_numeric(df[&quot;TotalCharges&quot;],downcast=&#39;float&#39;,errors=&#39;coerce&#39;)<br>df=pd.get_dummies(df,columns=[&#39;gender&#39;,&#39;Partner&#39;,&#39;Dependents&#39;,&#39;PhoneService&#39;,&#39;MultipleLines&#39;,&#39;InternetService&#39;,&#39;OnlineSecurity&#39;,&#39;OnlineBackup&#39;,&#39;DeviceProtection&#39;,&#39;TechSupport&#39;,&#39;StreamingTV&#39;,&#39;StreamingMovies&#39;,&#39;Contract&#39;,&#39;PaperlessBilling&#39;,&#39;PaymentMethod&#39;,&#39;Churn&#39;])<br>df.drop([ &#39;Churn_No&#39;, &#39;gender_Female&#39;, &#39;Partner_No&#39;,<br>        &#39;Dependents_No&#39;, &#39;PhoneService_No&#39;, &#39;PaperlessBilling_No&#39;],axis=1,inplace=<strong>True</strong>)</pre><p>We now have a total of 41 columns in our dataset. There are multiple ways to deal with data that has these many dimensions . One of the way is to reduce the dimensionality of the dataset through a principal component analysis. Whether the principal component analysis can be conducted on a categorical dataset is another question. This link on Stack Exchannge provides a different alternative called a multiple correspondence analysis (MCA).<a href="https://stats.stackexchange.com/questions/5774/can-principal-component-analysis-be-applied-to-datasets-containing-a-mix-of-cont">https://stats.stackexchange.com/questions/5774/can-principal-component-analysis-be-applied-to-datasets-containing-a-mix-of-cont</a></p><p>However, for the scope of this project we are going to skip this process and proceed to building our machine learning algorithm.The first part of the process will be to divide the datae into test and the training set</p><pre>X=df.iloc[:,:-1].values<br>y=df[&#39;Churn_Yes&#39;].values<br><br>X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.2)</pre><p>Once the data is segregated we can proceed to use logistic regression for our data.</p><pre>lr=LogisticRegression()<br>lr.fit(X_train,y_train)<br>y_pred=lr.predict(X_test)<br>print(classification_report(y_pred,y_test))</pre><p>The results from the logistic regression are shown below</p><pre>precision    recall  f1-score   support<br><br>           0       0.91      0.84      0.87      1107<br>           1       0.54      0.70      0.61       302<br><br>    accuracy                           0.81      1409<br>   macro avg       0.72      0.77      0.74      1409<br>weighted avg       0.83      0.81      0.81      1409</pre><p>The model has an accuracy score of 81% with precision of 54% and a recall of 70% for the positive customer churn value. Let us try to us a decisio tree classifier to see whether we can further improve the result</p><pre>dc=DecisionTreeClassifier()<br>dc.fit(X_train,y_train)<br>y_pred=dc.predict(X_test)<br>Tree&#39;,accuracy_score(y_pred,y_test),&#39;<strong>\n</strong>&#39;<br>        &#39;Confusion Matrix with Decision Tree&#39;, confusion_matrix(y_pred,y_test))</pre><p>The results are shown below</p><pre>Accuracy Score with Decision Tree 0.7381121362668559 <br>Confusion Matrix with Decision Tree [[843 194]<br> [175 197]]</pre><p>The accuracy score has fallen from 81% to 73%.We will now use a GridSearchCV on a random forest classifier to see whether the score of the model can be improved.</p><pre><strong>from</strong> <strong>sklearn.model_selection</strong> <strong>import</strong> RandomizedSearchCV</pre><pre>estimators=range(50,100)<br>max_features = [&#39;auto&#39;, &#39;sqrt&#39;]<br>max_depth = range(4,12)<br>min_samples_split = range(2,8)<br>min_samples_leaf = range(1,8)<br>bootstrap = [<strong>True</strong>, <strong>False</strong>]<br>criterion=[&#39;gini&#39;,&#39;entropy&#39;]</pre><pre>random_grid = {&#39;n_estimators&#39;:estimators,<br>              &#39;max_features&#39;:max_features,<br>              &#39;max_depth&#39;:max_depth,<br>              &#39;min_samples_split&#39;:min_samples_split,<br>              &#39;min_samples_leaf&#39;:min_samples_leaf,<br>              &#39;bootstrap&#39;:bootstrap,<br>              &#39;criterion&#39;:criterion}</pre><pre>rf=RandomForestClassifier()<br>rf=RandomizedSearchCV(estimator=rf,param_distributions=random_grid,cv=5,verbose=1)</pre><pre>rf.fit(X_train,y_train)<br>best_estimate=rf.best_params_<br>rf2=RandomForestClassifier(**best_estimate)<br>rf2.fit(X_train,y_train)<br>y_pred=rf2.predict(X_test)<br>print(classification_report(y_pred,y_test))</pre><p>We used a Randomised Search CV to identify the best estimates for the random forest classifier and then using those estimates we built a random forest classifier. Let us see the results below</p><pre>precision    recall  f1-score   support<br><br>           0       0.92      0.83      0.87      1122<br>           1       0.51      0.70      0.59       287<br><br>    accuracy                           0.80      1409<br>   macro avg       0.71      0.77      0.73      1409<br>weighted avg       0.83      0.80      0.81      1409</pre><p>The score is still as good as logistic regression. We can also use the feature importances method to find out the key factors affecting customer churn</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/524/1*zjfLGV2YAXcBeypBnVGf_w.png" /></figure><p>The three most important features are</p><ol><li>Monthly Charges</li><li>Type of contract(month-to-month, one-year,two-year)</li><li>Online security</li></ol><p>Strategists can better focus on thesefactors to reduce customer churn and increase customer satisfaction.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=70a821a84ce8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploratory Data Analysis:English Premier League]]></title>
            <link>https://suhaibkamal93.medium.com/exploratory-data-analysis-english-premier-league-d9581623dd5?source=rss-95b354a96834------2</link>
            <guid isPermaLink="false">https://medium.com/p/d9581623dd5</guid>
            <category><![CDATA[sports-analytics]]></category>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <category><![CDATA[english-premier-league]]></category>
            <category><![CDATA[pandas]]></category>
            <dc:creator><![CDATA[Suhaib  Ali Kamal]]></dc:creator>
            <pubDate>Sat, 15 May 2021 00:46:42 GMT</pubDate>
            <atom:updated>2021-05-16T15:20:19.700Z</atom:updated>
            <content:encoded><![CDATA[<p>An in-depth analysis into the history of the English Premium league plus some cool functions</p><p>The EPL is one of the most popular and comptitive football leagues in the world with millions of fans watching from all around the world. In this article, we will go through some exploratory data analysis of the EPL data. We will also go through some cool functions to derive table standing and relegations escapes.</p><p>The dataset that we have in hand can be downloaded from the following link.<a href="https://www.kaggle.com/saife245/english-premier-league">https://www.kaggle.com/saife245/english-premier-league</a></p><p>For this analysis we are goind to download the dataset for 18 seasons starting from 2000/01 to 2017/18. The dataset is in different excel files(18 files) so we are going to need a for loop to download the list of files</p><pre>cwd = os.path.abspath(&#39;&#39;) <br>files = os.listdir(cwd)<br>df = pd.DataFrame()<br>for file in files:<br>     if file.endswith(&#39;.csv&#39;):<br>            df = df.append(pd.read_csv(file), ignore_index=True) </pre><p>Using the above code, we were able to download 18 different files and were also able to merge into one complete file.After conducting some data preprocessing dropping missing values we had a set of 6460 rows with the following columns.</p><p>Div = League Division<br>Date = Match Date (dd/mm/yy)<br>HomeTeam = Home Team<br>AwayTeam = Away Team<br>FTHG = Full Time Home Team Goals<br>FTAG = Full Time Away Team Goals<br>FTR = Full Time Result (H=Home Win, D=Draw, A=Away Win)<br>HTHG = Half Time Home Team Goals<br>HTAG = Half Time Away Team Goals<br>HTR = Half Time Result (H=Home Win, D=Draw, A=Away Win)</p><p>Match Statistics (where available)<br>Attendance = Crowd Attendance<br>Referee = Match Referee<br>HS = Home Team Shots<br>AS = Away Team Shots<br>HST = Home Team Shots on Target<br>AST = Away Team Shots on Target<br>HHW = Home Team Hit Woodwork<br>AHW = Away Team Hit Woodwork<br>HC = Home Team Corners<br>AC = Away Team Corners<br>HF = Home Team Fouls Committed<br>AF = Away Team Fouls Committed<br>HO = Home Team Offsides<br>AO = Away Team Offsides<br>HY = Home Team Yellow Cards<br>AY = Away Team Yellow Cards<br>HR = Home Team Red Cards<br>AR = Away Team Red Cards</p><p>After reducing the dataset to these columns, I felt there were some columns that needed to be added. To begin, the FTR result column only stated whether the result was in favour of the home team or the away team .To change this I added two more columns:(Winning Team &amp; Losing team). The code for the data transformation is shown below</p><pre>conditions=[df[&#39;FTR&#39;]==&#39;H&#39;,df[&#39;FTR&#39;]==&#39;A&#39;,df[&#39;FTR&#39;]==&#39;D&#39;]<br>choices=[df.AwayTeam,df.HomeTeam,&#39;Match Drawn&#39;]<br>choices2=[df.HomeTeam,df.AwayTeam,&#39;Match Drawn&#39;]<br>df[&#39;Losing Team&#39;]=np.select(conditions,choices,default=np.nan)<br>df[&#39;Winning Team&#39;]=np.select(conditions,choices2,default=np.nan)</pre><p>Furthermore, the dataset also does not have the season. To add season we could sort the data in an ascending format use a for loop to add season. The EPL typically has 380 games so the season will have have to change after 380 rows of data.</p><p>Now that we have gone through some data transformations let us try to get some basic data analysis and answer some question.</p><h3><strong>What is the average number of goals scored trend for the last 18 years?</strong></h3><p>To answer this question we can use the groupby function in pandas to plot a lineplot. The code is show below.</p><pre>df.groupby(&#39;Season&#39;)[&#39;Total Goals&#39;].mean().plot(kind=&#39;line&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/378/1*95z0zz_rMYEJlZcXoCplWw.png" /><figcaption>Avg goals/match scored by season</figcaption></figure><p>The above line chart shows that the average goals scored per match has been around 2.5 to 2.8 with 2012 to 2015 seasons being the most prolific in terms of goals scored.</p><h3><strong>Q2) Compare the teams home season records in the last 18 years</strong></h3><p>For this we will add another column that will calculate points per match for hme and away teams. We will use the same format as we did previously with the winning team column.</p><pre>conditions=[df[&#39;FTR&#39;]==&#39;H&#39;,df[&#39;FTR&#39;]==&#39;A&#39;,df[&#39;FTR&#39;]==&#39;D&#39;]<br>choices=[3,0,1]<br>choices2=[0,3,1]<br>df[&#39;HomeTeamPoints&#39;]=np.select(conditions,choices,default=np.nan)<br>df[&#39;AwayTeamPoints&#39;]=np.select(conditions,choices2,default=np.nan)</pre><p>After the transformation we can add a groupby column and plot a bar chart to see how many points on average do teams earn on their home turfs.</p><pre>df.groupby(&quot;HomeTeam&quot;)[&#39;HomeTeamPoints&#39;].mean().sort_values(ascending=False).head(5).plot(kind=&#39;barh&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/427/1*zAKqdaY9BrAqXdQvJyAscg.png" /><figcaption>Avg points/home game</figcaption></figure><p>As can be seen from the above table, the top4 of the 2000’s have the best home record.</p><p>Using this dataset we will try to draw a season table for any year. We will later develop this into a user function where the user will be able to input the season and get the season table.</p><pre>def table():<br>    x=input(&quot;Please enter the season table that you want?&quot;)<br>    df_season=df[df[&#39;Season&#39;]==x]<br>    group1=df_season.groupby(&#39;HomeTeam&#39;)[&#39;HomeTeamPoints&#39;].sum()<br>    group2=df_season.groupby(&quot;AwayTeam&quot;)[&#39;AwayTeamPoints&#39;].sum()<br>    final=group1+group2<br>    final=final.sort_values(ascending=False)<br>    return final</pre><p>The above function will be able to derive a season table for any year that we want. This table can then be used for further analysis. For example using the same code we can derive the average minimum points required to escape relegation.</p><pre>sum1=0<br>for i in season:<br>    df_season=df[df[&#39;Season&#39;]==i]<br>    group1=df_season.groupby(&#39;HomeTeam&#39;)[&#39;HomeTeamPoints&#39;].sum()<br>    group2=df_season.groupby(&quot;AwayTeam&quot;)[&#39;AwayTeamPoints&#39;].sum()<br>    final=group1+group2<br>    final=final.sort_values(ascending=False)<br>    sum1=sum1+final.iloc[16]</pre><pre>print(&quot;Average points required to escape relegation is &quot;, round(sum1/18,0)</pre><p>The above code’s result is 38 which implies that an average points is enough to escape relegation and stay in the EPL.Similiarly we can use it to see the minimum points required to get a Top4 place.</p><pre>sum1=0<br>for i in season:<br>    df_season=df[df[&#39;Season&#39;]==i]<br>    group1=df_season.groupby(&#39;HomeTeam&#39;)[&#39;HomeTeamPoints&#39;].sum()<br>    group2=df_season.groupby(&quot;AwayTeam&quot;)[&#39;AwayTeamPoints&#39;].sum()<br>    final=group1+group2<br>    final=final.sort_values(ascending=False)<br>    sum1=sum1+final.iloc[3]</pre><pre>print(&quot;Average points required to achieve Top4 place is &quot;, round(sum1/18,0)</pre><p>The result for the above code is 70 implying that an average of 70 points is enough to secure a Champions League place. We can now movefurther and assess how important the referees are in determining match results.</p><p>For this we are going to conduct a hypothesis test to see whether there is any significant difference between the referees attitudes to home teams and awat teams. For this analysis, we will add two more columns( Home Team booking points)&amp;(Away Team Booking Points). The code will assign 15 booking points to every yellow card and 25 booking points to every red card.</p><pre>df[&#39;HBP&#39;]=df[&#39;HY&#39;]*15+df[&#39;HR&#39;]*25<br>df[&#39;ABP&#39;]=df[&#39;AY&#39;]*15+df[&#39;AR&#39;]*25</pre><p>Next, we are going to conduct a t-test to determine whether there is a significant difference between the referees attitudes to home teams and away teams.</p><pre>stat, p = ttest_ind(df.HBP, df.ABP)<br>print(&#39;Statistics=%.3f, p=%.3f&#39; % (stat, p))</pre><p>The p-value is around 0 indicating that there is a significant difference between the referees attitudes to home teams and away teams.</p><p>One of the things that EPL is renowned for is close matches and exciting turnarounds. We can analyse this data to see how easily is it to change half time results for losing teams both home and away.</p><pre>sns.heatmap(pd.crosstab(df.FTR,df.HTR,normalize=&#39;columns&#39;),cmap=&#39;viridis&#39;,annot=True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/1*Rhnpts8HC6xYXiTHu8uLpQ.png" /></figure><p>For example according to the above heatmap, in only 5.1% of the matches in the EPL have the away team been able to overturn a half time losing result.However, if the away team is winning at half time they have a 69% chance to end up being the winner. Moreover if the home team is winning at half time 81% times the home team ends up being the winner.</p><p>Lastly, we are going to analyse head to head results for some major teams.I have created a function where the user can enter teams according to his own will and the code will prepare a table listing out the head-to head record.</p><pre>def win_loss():<br>    team1=input(&quot;Please enter your first team&quot;)<br>    team2=input(&quot;Please enter your second team&quot;)<br>    new_df=df[(df[&#39;HomeTeam&#39;]==team1) | (df[&#39;HomeTeam&#39;]==team2)]<br>    new_df=new_df[(new_df[&#39;AwayTeam&#39;]==team1) | (new_df[&#39;AwayTeam&#39;]==team2)]<br>    return new_df[&#39;Winning Team&#39;].value_counts()</pre><p>For example using this code we can see the head to head record between Arsenal and Chelsea.</p><pre>Chelsea        14<br>Match Drawn    13<br>Arsenal         9</pre><p>I hope you liked it. To connect please visit my LinkedIn profile through the following link.</p><p><a href="https://www.linkedin.com/in/suhaib-kamal/">https://www.linkedin.com/in/suhaib-kamal/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d9581623dd5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SQL in 10 queries :Learning by Doing]]></title>
            <link>https://suhaibkamal93.medium.com/sql-in-10-queries-learning-by-doing-3afc7ee6757a?source=rss-95b354a96834------2</link>
            <guid isPermaLink="false">https://medium.com/p/3afc7ee6757a</guid>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[mysql]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Suhaib  Ali Kamal]]></dc:creator>
            <pubDate>Tue, 27 Apr 2021 18:25:34 GMT</pubDate>
            <atom:updated>2021-04-28T01:27:34.912Z</atom:updated>
            <content:encoded><![CDATA[<h3>SQL in 10 queries :Learning by Doing</h3><p>One of the key skills that a good data analyst needs to have is the ability to extract data from large databases. SQL is a domain-specific langauge that helps you to access and manipulate large databases.</p><p>In this article we are going to go over some of the basic commands that are important to learn to have good mastery over SQL. We are going to use a public database Sakila that lists information about movies and customers.The database has a lot of tables that are connected to each other through primary and foreign keys.The entity relationship diagram is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/823/1*-8pkHWOthtIit2Bge1WOLg.png" /></figure><p>We are going to go through a series of questions and try to learn the concePts and commands used in SQL.</p><p>Q1) You need to find the ID number, first name, and last name of an actor, of whom you know only the first name, “Joe.” What is one query would you use to obtain this information.</p><pre>SELECT actor_id,first_name,last_name<br>FROM actor<br>WHERE lower(first_name)=&#39;joe&#39;;</pre><p>The above query use SELECT which helps us retreive the columns that we want from our query and the WHERE clause which is used to list out conditions for our queries.</p><p>Q2) Find all actors whose last names contain the letters LI. This time, order the rows by last name and first name, in that order:</p><pre>SELECT first_name,last_name<br>FROM actor<br>WHERE upper(last_name) LIKE &#39;%LI%&#39;</pre><p>The above query in addition to the SELECT and the WHERE clause uses the LIKE clause which is a wildcard operator meaning that any name containing the words LI will be included in the query and the ORDER clause is a sorting command.</p><p>Q3)Using IN, display the country_id and country columns of the following countries: Afghanistan, Bangladesh, and China</p><pre>SELECT country_id,country<br>FROM country<br>WHERE country IN (&#39;Afghanistan&#39;,&#39;Bangladesh&#39;,&#39;China&#39;)</pre><p>Q4)List last names of actors and the number of actors who have that last name, but only for names that are shared by at least two actors</p><pre>SELECT last_name,count(*) AS num_actor<br>FROM actor<br>GROUP BY last_name<br>HAVING num_actor&gt;2<br>ORDER BY 2 desc;</pre><p>The above query uses the GROUP clause which is used to aggregate the data based on a column or a condition. The HAVING clause is used to perform conditions on the the grouped values.</p><p>Q5) Use JOIN to display the first and last names, as well as the address, of each staff member. Use the tables staff and address</p><pre>SELECT staff.first_name,staff.last_name,address.address<br>FROM staff LEFT JOIN address<br>ON staff.address_id=address.address_id;</pre><p>The JOIN clause is perhaps one of the most used clauses in SQL and also causes a fair bit of confusion. In this example we used the LEFT JOIN which means that all the records of staff will be included but unmatched records from the address table will not be included. In other words, staff members who do not have an address will be included.</p><p>Q6) List each film and the number of actors who are listed for that film. Use tables film_actor and film. Use inner join.</p><pre>SELECT film.film_id,film.title,count(actor_id)<br>FROM film INNER JOIN film_actor ON<br>film.film_id=film_actor.film_id<br>GROUP BY film.film_id,film.title<br>ORDER BY 3 desc</pre><p>This example shows how an inner join can used to show matching records from two tables.</p><p>Q7) The music of Queen and Kris Kristofferson have seen an unlikely resurgence. As an unintended consequence, films starting with the letters K and Q have also soared in popularity. Use subqueries to display the titles of movies starting with the letters K and Q whose language is English.</p><pre>SELECT film.title<br>FROM film <br>WHERE title LIKE &#39;K%&#39; OR title LIKE &#39;L%&#39;<br>AND language_id = (SELECT language_id FROM language WHERE name=&#39;English&#39;);</pre><p>This is an example of a subquery. In this example we have used a subquery instead of a JOIN clause. This is another way to write a subquery and to access data.</p><p>Q8) You want to run an email marketing campaign in Canada, for which you will need the names and email addresses of all Canadian customers. Use joins to retrieve this information.</p><pre>SELECT customer.first_name,customer.last_name,customer.email<br>FROM customer LEFT JOIN address ON customer.address_id=address.address_id<br>JOIN city ON address.city_id=city.city_id<br>JOIN country ON city.country_id=country.country_id<br>WHERE country=&#39;Canada&#39;;</pre><p>This query uses three JOINS to acccess results from a database.We will now move to more advanced questions relating to JOINS and results.</p><p>Q9) Construct a query that produces a top 3 movie list in the documentary category.</p><pre>SELECT film.title,count(*)<br>FROM film JOIN film_category ON film.film_id=film_category.film_id<br>JOIN category on film_category.category_id=category.category_id<br>JOIN inventory ON film.film_id=inventory.film_id<br>JOIN rental ON inventory.inventory_id=rental.inventory_id<br>WHERE category.name=&#39;Documentary&#39;<br>GROUP BY title<br>ORDER BY  2 DESC<br>LIMIT 3;</pre><p>Q10)List the top five genres in gross revenue in descending order. (Hint: you may need to use the following tables: category, film_category, inventory, payment, and rental.)</p><pre>select category.name category_name, sum(payment.amount) category_revenue<br>from category <br>LEFT JOIN film_category <br>on category.category_id = film_category.category_id<br>LEFT JOIN film <br>on film_category.film_id = film.film_id<br>LEFT JOIN inventory <br>on film.film_id = inventory.film_id<br>LEFT JOIN rental <br>on inventory.inventory_id = rental.inventory_id<br>LEFT JOIN payment <br>on rental.rental_id = payment.rental_id<br>GROUP BY category.name<br>ORDER BY category_revenue desc<br>limit 5;</pre><p>This query uses all the things that we discussed above. It uses multiple JOIN clauses and also uses GROUP BY clause.</p><p>We saw how SQL can be used to query information from a database . Mastering these commands can provide you with a decent grip on SQL, something which is important if one wants to move into a data career.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3afc7ee6757a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[HR Analytics in Python]]></title>
            <link>https://suhaibkamal93.medium.com/hr-analytics-in-python-2a29a4eb3625?source=rss-95b354a96834------2</link>
            <guid isPermaLink="false">https://medium.com/p/2a29a4eb3625</guid>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[boosting]]></category>
            <category><![CDATA[human-resources]]></category>
            <dc:creator><![CDATA[Suhaib  Ali Kamal]]></dc:creator>
            <pubDate>Mon, 19 Apr 2021 00:44:02 GMT</pubDate>
            <atom:updated>2021-04-21T23:33:59.724Z</atom:updated>
            <content:encoded><![CDATA[<p>Attrition Analysis and Prediction in Python</p><h3>Introduction</h3><p>In this article , we are going to conduct HR attrition analysis in Python. The dataset that we are going to use is a very popular dataset that can found on the following link: <a href="https://www.kaggle.com/pavansubhasht/ibm-hr-analytics-attrition-dataset">https://www.kaggle.com/pavansubhasht/ibm-hr-analytics-attrition-dataset</a></p><p>The purpose of this exercise is to identify factors that lead to attrition and then predict employee attrition using multiple machine learning algorithms.The dataset has thirty-two independent variables that range from tenure to the distance from the workplace. A comprehensive dataset like this requires an extensive EDA process before a machine learning algorithm can be built.</p><p>The most importat part of any EDA process is to identify patterns through different data visualisation procedures. We are going to start with calculating the attrition rates of each department.</p><pre>df.groupby(&#39;Department&#39;).mean()[&#39;Attrition&#39;].plot(kind=&#39;bar&#39;,color=[&#39;Green&#39;,&#39;Blue&#39;,&#39;Pink&#39;])<br>plt.title(&quot;Attrition Rate by Department&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/384/1*0BCKwswFZnWT8XCayLpSQw.png" /><figcaption>Attrition Rate by department</figcaption></figure><p>As can be seen from the above bar chart the Sales department has the highest attrition rates for the company.Let us now look into the factors that affect attrition rates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*zlhgYhKQfSDfziZZELzEBw.png" /><figcaption>Attrition Rate/Distance from Home</figcaption></figure><p>The above diagram illustrates that employees with higher distance have higher attrition rates as compared to people with closer distance to their homes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*cj673qc_NTzOf1n-wDRg8w.png" /><figcaption>Attrition/Working years</figcaption></figure><p>The above diagram illustrates that newer employer are much more likely to leave organisation as compared to older employees.There are so many factors to consider so let us move forward and take a holistic view by creatinga correlation matrix amongst all the variables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-_a19cNs3rWpMoyrw666Fw.png" /><figcaption>Correlation Matrix</figcaption></figure><p>The correlation matrix does not indicate any high degree of correlation with the dependent variable. However, it does provide us with a holistic view off all the factors.Let us move onto build an algorithm</p><h3>Feature Engineering</h3><p>The first part of this algorithm is to segregate the categorical and the numerical features The following code can be used to segregate the categorical features and use dummy variables instead.</p><pre>cat_var=[&#39;BusinessTravel&#39;,&#39;Department&#39;,&#39;EducationField&#39;,&#39;Gender&#39;,&#39;JobRole&#39;, &#39;MaritalStatus&#39;, &#39;Over18&#39;,&#39;OverTime&#39;]<br>X=pd.get_dummies(df,columns=cat_var,drop_first=<strong>True</strong>)</pre><pre>num_var=[]<br><strong>for</strong> i <strong>in</strong> X.columns:<br>    <strong>if</strong> X[i].nunique()&gt;3:<br>        num_var.append(i)</pre><h3>Logistic Regression</h3><p>Logistic regression is a very popular classification algorithm that uses the sigmoid function to build an S-curve to predict probability of an outcome.It has several characteristics that make it suitable for a large number of classification problems</p><ol><li>It is much more easier to train,implement and efficient as compared to other machine learning algorithms</li><li>It has far fewer assumptions about variables as compared to past machine learning algorithms</li><li>It has a decent degree of accuracy especially where the number of features is lower than the the number observations</li></ol><pre>lr=LogisticRegression(C=100,max_iter=10000)<br>lr.fit(X_train,y_train)<br>y_pred=lr.predict(X_test)<br>print(accuracy_score(y_test,y_pred))<br>print(classification_report(y_test,y_pred)</pre><p>According to the classification report the accuracy of the model is 87% however its recall is lower at 43% of positive cases. The logistic regression model is providing excellent results, however the purpose of the problem is to identufy employees that are likely to leave.This is the reason that recall then becomes a very important measure. Recall measures the fraction of values that are identified correctly. It is calculated by the following formula</p><p><strong>True Positive/(True Positive+ False Negative)</strong></p><p>In this example, we want to limit false negative as we want to identify all employees who might leave. In this example the cost of a false negative is far higher than the cost of a false positive, which is why it is important to build a model that can improve recall.</p><p>We are now going to build a decision tree classifier to see whether we can improve model performance. The decision tree classifier has the advantage that it is easily interpretable and does not require any specific distribution for its features. Let us see what the decision tree classifer tells us about the problem.</p><pre><strong>from</strong> <strong>sklearn.tree</strong> <strong>import</strong> DecisionTreeClassifier<br>dc=DecisionTreeClassifier()<br>dc.fit(X_train,y_train)<br>dc.predict(X_test)<br>print(classification_report(y_test,y_pred)</pre><p>The accuracy in this case has gone down to 66% from 87% , however the positive thing is total recall for positive results has improved to 0.66. This is a pretty good result as our model is now more useful than the previous ones. Using the features importance feature we can see how each feature relates to attrition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*oogLng1SHN3bFfqpK98aew.png" /><figcaption>Feature importance for Attrition</figcaption></figure><p>As can be seen from the above illustration, the employee working experience and the monthly income are the biggest factors that relate to attrition. This is important from a learning perspective as it could be used by the HR department to guide future policy and decision making.</p><h3>Boosting</h3><p>Now we are going to apply boosting algorithms to see whether we can improve the model even further. We are going to use the Gradient Boosting Classifier to build a machine learning algorithm to see how far can the model’s performance can be improved.</p><p><strong>from</strong> <strong>sklearn.ensemble</strong> <strong>import</strong> GradientBoostingClassifier gbc=AdaBoostClassifier(n_estimators=100) gbc.fit(X_test,y_test) y_pred=gbc.predict(X_test) print(classification_report(y_test,y_pred))</p><p>The results have improved drastically. The accuracy score is now 98% with the recall score for positive results at 88%.This example shows the power of using boosting algorithms and how they can imporve overall model performance.</p><p>In this article we saw how data science can be used to identify patterns and predict the future. Data-driven organisations can harness this data to improve decision-making and improve organisational performance</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a29a4eb3625" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predicting heart diseases through machine learning]]></title>
            <link>https://suhaibkamal93.medium.com/predicting-heart-diseases-through-machine-learning-6f867a674078?source=rss-95b354a96834------2</link>
            <guid isPermaLink="false">https://medium.com/p/6f867a674078</guid>
            <category><![CDATA[xgboost]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Suhaib  Ali Kamal]]></dc:creator>
            <pubDate>Wed, 14 Apr 2021 01:05:24 GMT</pubDate>
            <atom:updated>2021-04-14T01:05:24.736Z</atom:updated>
            <content:encoded><![CDATA[<h3>Predicting heart disease through machine learning</h3><p>Classification problems is one of the most common areas where machine learning algorithms are applied with excellent results. The biggest difference between a regression problem and a classification problem is that in a classification problem the target variable is categorical/binary.</p><p>In this article, we will go over the heart rate disease dataset published by UCI Machine Learning Repository where the target variable is heart disease.We will go over multiple algorithms and will also see how boosting can improve model results.</p><p>The dataset has multiple categorical and continous independent variables which can be used to predict heart disease in a patient.After going through the process of identifying missing values(if any) we will proceed to EDA to identify any patterns in the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*NNAL8SeaH5KHIwHePFVBZg.png" /><figcaption>Left: Distribution of Age Right: Age vs Target</figcaption></figure><p>As can be seen from the above diagram younger people are more prone to heart disease than older people which is surprising. Let us see whether gender has an impact on heart disease</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/1*ZBWA7X3aAx2u6hQEwOcMSg.png" /><figcaption>Male:0 Female:1</figcaption></figure><p>As can be seen from the above diagram, male are more prone to heart disease as compared to women. We can also see how cholestrol affects the rate of heart disease</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/719/1*hMtRk2L0U3koTUrgTr2gbA.png" /><figcaption>Cholestrol affect on heart disease</figcaption></figure><p>What can be seen from the above diagrams is that cholestrol itself does not have a huge impact on heart diseases.EDA is a crucial part of data science, and its importance is often underrated. For this article, we are going to move forwards to build the actual model.However before we move ,we can also the correlation matrix to get a better sense of the overall data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/1*P0HOWBOii0G3lDrKnf4mbw.png" /><figcaption>Correlation matrix</figcaption></figure><h3>Feature Engineering</h3><p>Before building the model, it is important to make sure that the independent variables are engineered in a way that can improve the model performance. For the categorical features we need to convert them to dummy variables for which the code is shown below.</p><pre>df=pd.get_dummies(df,columns=[&#39;cp&#39;,&#39;restecg&#39;,&#39;slope&#39;,&#39;ca&#39;,&#39;thal&#39;],drop_first=<strong>True</strong>)<br>y=df.target<br>X=df.drop(&quot;target&quot;,axis=1)</pre><p>Furthermore, for the continous variables it is important to standardize them to improve model performance.One important thing to note in this transformation is to segregate the training and the test set prior to the standardisation. For the training set we are going to use the StandardScaler from the scikit library and then transform it in the test set,</p><pre><strong>from</strong> <strong>sklearn.preprocessing</strong> <strong>import</strong> StandardScaler<br>scaler=StandardScaler()<br>num_columns=[&#39;age&#39;,&#39;trestbps&#39;,&#39;chol&#39;,&#39;thalach&#39;,&#39;oldpeak&#39;]<br>X_train[&#39;age&#39;]=scaler.fit_transform(X_train[&#39;age&#39;].values.reshape(-1,1))<br>X_train[&#39;trestbps&#39;]=scaler.fit_transform(X_train[&#39;trestbps&#39;].values.reshape(-1,1))<br>X_train[&#39;chol&#39;]=scaler.fit_transform(X_train[&#39;chol&#39;].values.reshape(-1,1))  <br>X_train[&#39;thalach&#39;]=scaler.fit_transform(X_train[&#39;thalach&#39;].values.reshape(-1,1))   <br>X_train[&#39;oldpeak&#39;]=scaler.fit_transform(X_train[&#39;oldpeak&#39;].values.reshape(-1,1))</pre><pre>X_test[&#39;age&#39;]=scaler.transform(X_test[&#39;age&#39;].values.reshape(-1,1))<br>X_test[&#39;trestbps&#39;]=scaler.transform(X_test[&#39;trestbps&#39;].values.reshape(-1,1))<br>X_test[&#39;chol&#39;]=scaler.transform(X_test[&#39;chol&#39;].values.reshape(-1,1))  <br>X_test[&#39;thalach&#39;]=scaler.transform(X_test[&#39;thalach&#39;].values.reshape(-1,1))   <br>X_test[&#39;oldpeak&#39;]=scaler.transform(X_test[&#39;oldpeak&#39;].values.reshape(-1,1))</pre><p>After the transformations are done we can move to the next step: using machine learning algorithm</p><h3>Algorithms</h3><p>We will start with the Logistic regression algorithm to see how it performs on the dataset.</p><pre><strong>from</strong> <strong>sklearn.linear_model</strong> <strong>import</strong> LogisticRegression<br><strong>from</strong> <strong>sklearn.model_selection</strong> <strong>import</strong> cross_val_score<br><strong>from</strong> <strong>sklearn.metrics</strong> <strong>import</strong> accuracy_score,confusion_matrix,classification_report<br>lr=LogisticRegression(C=1.0,penalty=&#39;l2&#39;)<br>lr.fit(X_train,y_train)<br>print(&quot;The cross validation score mean is &quot;,cross_val_score(lr,X_train,y_train,cv=3).mean())<br>pred=lr.predict(X_test)<br>print(classification_report(y_test,pred))<br>sns.heatmap(confusion_matrix(y_test,pred),annot=<strong>True</strong>)</pre><p>Ths gives us a model accuracy of 56% with a recall of 90%. This is a decent result but let us see how a decision tree performs on the dataset.</p><pre><strong>from</strong> <strong>sklearn.tree</strong> <strong>import</strong> DecisionTreeClassifier<br>dc=DecisionTreeClassifier()<br>dc.fit(X_train,y_train)<br>print(&quot;The cross validation score is &quot;,cross_val_score(dc,X_train,y_train).mean())<br>pred=dc.predict(X_test)<br>print(classification_report(y_test,pred))<br>sns.heatmap(confusion_matrix(y_test,pred),annot=<strong>True</strong></pre><p>The decision tree performs slightly better with an accuracy of 57% . The decision tree has only slightly improved the performance of the model but perhaps using a random forest classifier can improve accuracy.</p><pre><strong>from</strong> <strong>sklearn.ensemble</strong> <strong>import</strong> RandomForestClassifier<br>rf=RandomForestClassifier()<br>rf.fit(X_train,y_train)<br>print(&quot;The cross validation score is &quot;,cross_val_score(rf,X_train,y_train).mean())<br>pred=rf.predict(X_test)<br>print(classification_report(y_test,pred))<br>sns.heatmap(confusion_matrix(y_test,pred),annot=<strong>True</strong>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/346/1*S9Oq33vRwLCeIkHsxdzjjg.png" /><figcaption>Confusion Matrix</figcaption></figure><p>The random forest classifer has improved model performance as the accuracy is now 71%. The heatmap is the confusion matrix of the dataset which shows that there are 19 false negatives and 3 false positives. In a heart disease case, the cost of a false negative is very high. Let us see whether boosting algorithm can improve this.We are going use a very popular boosting algorithm called XG Boost which has actually been used to win Kaggle competitions. The code for the algorithm is written below</p><h3>Boosting</h3><pre><strong>import</strong> <strong>xgboost</strong> <strong>as</strong> <strong>xgb</strong><br>data_dmatrix = xgb.DMatrix(data=X,label=y)<br>xg_reg = xgb.XGBClassifier(objective =&#39;reg:logistic&#39;, colsample_bytree = 0.3, learning_rate = 0.1,<br>                max_depth = 5, alpha = 10, n_estimators = 10<br>xg_reg.fit(X_train,y_train)<br>pred = xg_reg.predict(X_test)<br>sns.heatmap(confusion_matrix(y_test,pred),annot=<strong>True</strong>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/346/1*pvTvFmvvULkVBmn8Ixmlog.png" /><figcaption>Confusion matrix</figcaption></figure><p>The above diagram displays the confusion matrix. As can been seen the performance of the model has improved significantly with fewer false negatives. The XGBoost model has an accuracy in excess of 80%.</p><p>We saw in this example how machine learning can be applied to a dataset to predict heart diseases. We started with exploratory data analysis, then moved on to feature engineering and built the machine learning algortims. We concluded with the XGBoost model to show how boosting can improve model performance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6f867a674078" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Customer Segmentation in Python using K-means]]></title>
            <link>https://suhaibkamal93.medium.com/customer-segmentation-in-python-using-k-means-c39a6680586c?source=rss-95b354a96834------2</link>
            <guid isPermaLink="false">https://medium.com/p/c39a6680586c</guid>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[k-means]]></category>
            <category><![CDATA[unsupervised-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Suhaib  Ali Kamal]]></dc:creator>
            <pubDate>Fri, 09 Apr 2021 17:56:41 GMT</pubDate>
            <atom:updated>2021-04-12T00:46:03.928Z</atom:updated>
            <content:encoded><![CDATA[<p>Analytics is now increasingly being used to guide business decisions in a variety of domains. Businesses these days have more data than ever on their consumers and the market. As a result it has become important for businesses to leverage that data to assisst in their decision making.</p><p>Marketing is one domain where data analysis is being used to provide actionable insights to marketing managers. One of the key tasks of any brand manager is to identify customer segments. A customer segment is a group of the population that has shared characteristics like age,education,preferences and attitudes. Segments can help marketing managers formulate strategies to target customers and hence increase sales.In this example we are going to conduct cluster analysis on a sample of mall customers.</p><p>The dataset that we are going is to use is a list of mall customers for which we have their gender,age ,annual income and spending.After reading the file it is important to do some exploratory data analysis to get a feel of the data.For this we will use the matplotlib library.</p><pre>sns.scatterplot(cust[&quot;Age&quot;],cust[&quot;Spending Score (1-100)&quot;],color=&quot;green&quot;,hue=cust[&quot;Gender&quot;])<br>plt.title(&quot; Spending Score vs Age&quot;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*_rLfZlm8pVQUv2q6FY4dmQ.png" /><figcaption>Scatter plot of spending score/age</figcaption></figure><p>One of the first plots that we can look upon is the spending score vs age where we can clearly see that there people between the ages of 20–40 have higher spending scores than the rest of the sample.There is some correlation between age and the spending score. For further analysis , we can also look into the correlation matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/399/1*udIi5YU6v6dANKL4sXmRbg.png" /><figcaption>Correlation plot</figcaption></figure><p>As can been be seen age and spending score has very high correlation as compared to the rest of the factors.These plots can help us identify relationships and patterns in our data . Now lets move onto writing the code for cluster analysis.</p><p>The clustering algorithm that we are going to use is K-means which is a very popular algorithm. It identifies k number of centroids and then allocates every point in the data to the closest cluster.K-means is an iterative procedure which means the centroids are recalculated after every allocation iteration until no improvement can be made in the clusters or the maximum number of iterations has been reached.</p><p>Before moving onto the cluster analysis it is important to scale the variables for which I have written the following code.</p><pre><strong>from</strong> <strong>sklearn.preprocessing</strong> <strong>import</strong> StandardScaler<br>cust_transform[&quot;Age&quot;]=scaler.fit_transform(cust[&quot;Age&quot;].values.reshape(-1,1))<br>cust_transform[&quot;Annual Income&quot;]=scaler.fit_transform(cust[&quot;Annual Income&quot;].values.reshape(-1,1))<br>cust_transform[&quot;Spending Score&quot;]=scaler.fit_transform(cust[&quot;Spending Score&quot;].values.reshape(-1,1))</pre><p>One of the biggest decisions while doing a k-means cluster analysis is to decide upon the number of clusters. For this data we are going to use the silhouette coefficient which is calculated by the following formula</p><p>x-y/max(x,y)</p><p>where x is the mean inter cluster distance and y is the mean intra cluster distance. The score ranges from -1 to +1 where the higher the score the better segregated the clusters are. We are going to use the sci-kit library to calculate the silhouette score for a range of clusters</p><pre><strong>from</strong> <strong>sklearn.metrics</strong> <strong>import</strong> silhouette_score<br>silhouette_coefficients=[]<br><strong>for</strong> i <strong>in</strong> range(2,12):<br>     kmeans = KMeans(n_clusters=i)     <br>     kmeans.fit(cust_transform)<br>     score = silhouette_score(cust_transform, kmeans.labels_)<br>     silhouette_coefficients.append(score)<br><br>sns.lineplot(x=range(2,12),y=silhouette_coefficients,color=&quot;green&quot;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/378/1*ijOc9hI0Vz5IRNUeHC920w.png" /><figcaption>Optimal number of clusters : 6</figcaption></figure><p>As can be seen from the above the optimal number of clusters are 6 and so we segregate the data into 6 clusters.After the clusters I am going to visualise the clusters and try to identify any patterns or insights that can help us in decision making.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*zri1dPr1lmdF1_HAkrrFRQ.png" /></figure><p>From the above diagram we can see that cluster 0 consists of people that have higher annual income but lower spending score. Cluster #2 is the cluster which has high annual income and also a high spending score.This is the cluster that we can target for expensive purchases.We can further analyse our cluster along different dimensions</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*6p3ewCNOePA1MrfWvx0L2w.png" /></figure><p>Here the clusters are visualised along spending score and age. From the previous visualisation we know that Cluster 2 consists of high income people with high spending score. However from this diagram we can see that they are aged between 20–40 years old.Furthermore Cluster 4 also has higher spending score but their annual income is lower .</p><p>Cluster analysis has a lot of applications in business related decisions especially marketing.Here we used cluster analysis to create market segments which can help marketing managers identify groups or segment who they can target to increase sales.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c39a6680586c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>