<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alexandra Pukhova on Medium]]></title>
        <description><![CDATA[Stories by Alexandra Pukhova on Medium]]></description>
        <link>https://medium.com/@alepukhova526?source=rss-d61aa9d3a4a9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*JdzozuZKPuPMzQM-.</url>
            <title>Stories by Alexandra Pukhova on Medium</title>
            <link>https://medium.com/@alepukhova526?source=rss-d61aa9d3a4a9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 10:06:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@alepukhova526/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Battling Data Imbalance in IBM HR Attrition Challenge]]></title>
            <link>https://medium.com/@alepukhova526/battling-data-imbalance-in-ibm-hr-attrition-challenge-3a26337a4943?source=rss-d61aa9d3a4a9------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a26337a4943</guid>
            <dc:creator><![CDATA[Alexandra Pukhova]]></dc:creator>
            <pubDate>Fri, 24 Apr 2020 14:55:44 GMT</pubDate>
            <atom:updated>2020-04-24T14:59:03.695Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*GUAaWQty2wd5OkaO.png" /><figcaption>Picture taken from <a href="https://elitedatascience.com/">https://elitedatascience.com/</a>imbalanced-classes</figcaption></figure><p>Let me begin with some stats: as per HBR, an average U.S. corporate employer spends a staggering $4,129 per single hire (Cappelli, 2019). Given that the U.S. fills 66 million jobs a year… You do the math. As one may guess, after a company spends this much money on recruitment (add training, orientation, etc.), they want to hold tight to their employees. As a result, predicting employee attrition* has become one of the key challenges and focus areas for Human Resources (HR). Whereas some employees will leave inevitably e.g. due to retirement, others will be motivated by other churn factors that may be more complex and interdependent.</p><p>To take a stab at the problem of predicting employee churn, we will work with IBM HR Attrition data. We will compare the performance of three models and use the best-performing one to determine the factors, affecting attrition the most. Given the data imbalance (discussed below), we will focus on two areas of improvement:</p><ul><li>Fine-tuning the models to optimize for imbalance-immune metrics;</li><li>Performing data augmentation using Synthetic Minority Oversampling Technique (SMOTE) to restore the label balance.</li></ul><p>* Attrition is a reduction in the workforce.</p><p>The full notebook with the code used in this post can be found <a href="https://github.com/AlexandraPukhova/Managing-Op-Complexity/blob/master/IBM_Employee_Attrition.ipynb">here</a>.</p><p><strong>Introducing the data</strong></p><p>We have a dataset with ~1500 samples and 28 features. The features have both numerical data, such as age, the number of years with the firm, etc., and categorical, specifically, nominal data such as field of education and current department. Attrition is the dependent variable that has two outcomes: 0 = the employee stays, 1 = they leave. As such, we have a binary classification problem with some categorical data.</p><p>I follow regular data pre-processing procedures such as encoding of categorical variables and scaling. Please see the GitHub repo linked above for further details.</p><p>Now, what models should we try? There is a wide range of classifiers, based on different algorithms. The long answer is that your choice should be appropriate for your data type and your decision question. The short answer is that you may want to try several different algorithms from different algorithm families and see which one sticks.</p><p>But before doing that, we need to decide what metric we should use to compare the future models. The first step is to look at the data imbalance — if we have a majorly overrepresented class, we cannot use the regular accuracy metric, which is the total number of correct predictions over total predictions. Accuracy is not a useful metric for imbalanced datasets because the model can simply learn to predict the majority class most of the time and be right! However, when presented with new data, such an unskilled model may perform worse than a coin flip.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/0*pXjrI8uU24xc9CKO" /></figure><p>Percentage of 1s: 16%, Total Attrition = 1: 237, Total samples: 1470</p><p>As we can see, the data seems rather imbalanced, with only ~16% of 1 labels. However, it is somewhat predictable that there will be more employees that stay than those who leave, so there are too few samples with target variable 1. As such, the data set represents a very realistic data challenge. If not accuracy, what can we use?</p><p>In order to avoid falling victim to the curse of class imbalance, we rely on two other metrics: recall and precision. Recall answers the following question:</p><ul><li>What percentage of employees that end up leaving does the model successfully find?</li></ul><p>Precision inquires:</p><ul><li>Of all the employees that the algorithm predicts will leave, how many of them do actually leave?</li></ul><p>Whereas both questions are relevant, they can translate into two different actions. If we focus on maximizing precision, we are trying to minimize “over-shooting”. That is if the model predicts that someone will leave, we take action to make them stay. What if they actually weren’t going to leave and we just wasted our resources to keep someone who would stay regardless? With recall, however, we try to minimize “under-shooting” (higher recall = fewer false negatives): if someone is going to leave, did the model identify them correctly? If not, we will have made no effort at all to save the employee! As such, the company must decide what matters most to them.</p><p>My two cents is that you’d rather be safe than sorry. So, we will use Recall as the primary metric for model evaluation and comparison. However, since we don’t want to compromise precision too much, we will also take into account the metric that combines both recall and precision: F1. F1 is a harmonic mean between precision and recall, meaning that it simply conveys the balance between the two. The higher the F1 score, the better.</p><p>Time to build our models. We will try three different classifiers:</p><ol><li>Decision Tree</li><li>XGBoost</li><li>Logistic Regression</li></ol><p>Since we have a rather limited amount of data and a class imbalance, we need to check for overfitting and ensure that we are utilizing data to the fullest. In order to do that, we will use RepeatedStratifiedKFold(), a resampling procedure that splits the data set into k groups and treats each one as an instance of a training data set. It then summarizes the skill of the model, based on the scores. Using the basic models, we get the following results:</p><pre>Recall for Decision Tree 0.3601</pre><pre>Precision for Decision Tree 0.382</pre><pre>F1 for Decision Tree 0.3585</pre><pre>Recall for XGBoost 0.3449</pre><pre>Precision for XGBoost 0.6346</pre><pre>F1 for XGBoost 0.4311</pre><pre>Recall for Logistic Regression 0.3616</pre><pre>Precision for Logistic Regression 0.7039</pre><pre>F1 for Logistic Regression 0.4601</pre><p>Looks frustrating, doesn’t it? Certainly, being able to identify only ~35% of your employee churn is a scary prospect. However, let’s not get give up just yet. There are two places where we can search for the problem: the models and the data itself. Let’s start with the models.</p><p><strong>Hyperparameter tuning</strong></p><p>Firstly, we will try to improve the models we’ve built by searching for the most optimal hyperparameters for each model. Scikitlearn has a GridSearchCV classifier that performs an exhaustive search over combinations of parameter values that we preliminarily specify. The full list of hyperparameters for each classifier (you can find it in the GitHub notebook) is rather long, and it also takes a long time to iterate over too many parameter options. So, we will manually specify the parameter grid for the classifier to examine.</p><pre>dt_param_grid = {&#39;criterion&#39;: [&#39;gini&#39;, &#39;entropy&#39;],</pre><pre>&#39;max_features&#39;: [&#39;auto&#39;, &#39;sqrt&#39;, &#39;log2&#39;],</pre><pre>&#39;max_depth&#39;: [int(x) for x in np.linspace(10, 110, num = 11)],</pre><pre>&#39;min_samples_split&#39;: [2, 5],</pre><pre>&#39;min_samples_leaf&#39;: [1, 2],</pre><pre>&#39;random_state&#39;: [0]}</pre><pre><br>xgb_param_grid = {&#39;n_estimators&#39;: [100, 200],</pre><pre>&#39;booster&#39;: [&#39;gbtree&#39;, &#39;dart&#39;],</pre><pre>&#39;learning_rate&#39;: [0.25, 0.3, 0.35, 0.4] ,</pre><pre>&#39;max_depth&#39;: [3, 4],</pre><pre>&#39;gamma&#39;: [0.0, 0.1],</pre><pre>&#39;colsample_bytree&#39;: [0.4]}</pre><pre><br>lr_param_grid = {&#39;C&#39;: [0.1, 1, 10],</pre><pre>&#39;penalty&#39;: [&#39;l1&#39;, &#39;l2&#39;, &#39;elasticnet&#39;],</pre><pre>&#39;solver&#39;: [&#39;newton-cg&#39;, &#39;lbfgs&#39;, &#39;liblinear&#39;, &#39;sag&#39;, &#39;saga&#39;],</pre><pre>&#39;max_iter&#39;: [300, 500, 700],</pre><pre>&#39;random_state&#39;: [0]}</pre><p>Luckily, we can also choose by which metric we want to optimize the hyperparameters. As we have previously established, recall matters most in this decision problem, so we will specify ‘recall_score’ as the refit parameter in the GridSearchCV classifier.</p><p>Here are the top three combinations of hyperparameters for each model and their mean recall, precision, and F1 scores.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*spRcIRenaegvNhM4r_tGVg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xP5EB4UCX3PAyWlo_u846Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lkw7yqw0ayj6rajYY-txXA.png" /></figure><p>We can see that all models get better, but only marginally. We would need something better than 45% to show to the C-Suite, am I right?</p><p>The next path we are going to explore is synthetically restoring the label balance in the original data set.</p><p><strong>Data Augmentation: Oversampling</strong></p><p>Data augmentation is a technique generally used in image classification tasks. Simply speaking, you create new images by making small perturbations to the existing ones. As you may guess, the key challenge is to ensure that the classification is right, after you’ve interfered with the sample’s features. With image classification, you can rely on the relationship between pixels to ensure that your changes don’t change the target label (Korneev, 2019). However, augmenting categorical data is a much riskier business. There are several techniques that help you achieve the right balance by varying the number of samples per label. Specifically, over- and under-sampling. We will oversample the minority class to bring the data balance to the desired level.</p><p>The imbalanced-learn package from the Scikitlearn library provides an easy-to-implement Synthetic Minority Oversampling Technique (SMOTE) class. One thing to note is that oversampling should only be done on the training set, whereas the test set should also be stratified but stay intact.</p><p>Here’s the distribution of Attrition class labels pre- and post-SMOTE.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/0*tLPBJ2zdtCihPFjr" /><figcaption>Counter({0: 924, 1: 178})</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/0*DATZzqBoL0sX28Ob" /><figcaption>Counter({0: 924, 1: 924})</figcaption></figure><p>We can now train our basic models on the augmented balanced training set. We obtain the following results for the three metrics that we chose:</p><pre>Recall for DT 0.4066</pre><pre>Precision for DT 0.3131</pre><pre>F1 for DT 0.353</pre><pre>Recall for XGB 0.3515</pre><pre>Precision for XGB 0.5429</pre><pre>F1 for XGB 0.4213</pre><pre>Recall for LR 0.6944</pre><pre>Precision for LR 0.3884</pre><pre>F1 for LR 0.495</pre><p>All three models improved but to different extents. Whereas the decision tree and XGBoost only received marginal improvements, logistic regression can now score ~ 70% recall! As compared to the tuning results, the models that least gained the least from tuning got the most improvements from being trained on augmented data.</p><p><strong>Feature Importance</strong></p><p>Now, let’s finally look at what plays the most important role in determining employee attrition. We train a Logistic Regression on augmented data, show it the test set, and extract the coefficients for all the features. The Top 10 features and their feature weights are listed below:</p><pre>[(&#39;YearsSinceLastPromotion&#39;, 1.8288596995685864),<br>(&#39;OverTime&#39;, 1.7551717513868836),<br>(&#39;NumCompaniesWorked&#39;, 1.2363779913652675),<br>(&#39;DistanceFromHome&#39;, 0.9351309000922813),<br>(&#39;JobRole_Laboratory Technician&#39;, 0.8018018621142567),  (&#39;BusinessTravel_Travel_Frequently&#39;, 0.7800932447765359),  (&#39;YearsAtCompany&#39;, 0.63279144334254),<br>(&#39;MaritalStatus_Single&#39;, 0.6215242500476372),<br>(&#39;JobRole_Sales Representative&#39;, 0.611415076609277),  (&#39;EducationField_Human Resources&#39;, 0.5536769725879849)]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/504/0*4DbVUCudHnbY_vVN" /></figure><p>Years since the last promotion along with hours worked overtime seem to be the two biggest predictors of employee attrition. Doing a sanity check on our results, we can conclude that it’s reasonable that the longer the employee has spent without promotion, the more likely they are to get discouraged and search for other opportunities. It is also logical that the more hours one works overtime, the more distressed they are likely to become. An interesting question to pose would be to compare results geographically — whereas some states have very stringent labor laws that require generous overtime pay that may be a worthy compensation to some employees, others only increase the pay rate marginally. (Sorry for being U.S. centric yet again). The third-most important factor is companies_worked_at. This feature can be seen as one of the loyalty indicators. For example, if the person has switched multiple workplaces, they may be less opposed to change and are likely to leave. A curious addition to this feature would be age: if the person is young and they’ve only ever worked for one company, it may say very little about their loyalty.</p><p>It is important to stress that correlation does not imply causation and we cannot take these results as a matra. However, we can make several recommendations to the employer. Firstly, since rewards seem to be one of the key themes, the employer can focus on rewarding overtime workers and ensuring the regularity of [reasonable] promotions across the firm. They should also make an emphasis on loyalty and looking at the fourth-most predictor, offer more comfort to their employees, such as provide transport to and from work to relieve the commute burden. Lastly, based on the model predictions per every candidate, the firm could simply prepare better for the leaving of specific employees.</p><p><strong>Next steps</strong></p><p>Firstly, the two techniques that I have explored are not mutually exclusive, quite the opposite-they are complimentary. When combined, they are likely to yield better performance. Moreover, there are several improvements that we could make to both techniques separately. As for hyperparameter tuning, you can directly vary the label threshold to see how precision and recall get affected. Most models will have a 0.5 cutoff, so by varying it, you can find the operating point of the model where both recall and precision are maximized. As for data augmentation, it is advisable to combine minority class oversampling with majority class undersampling, e.g using RandomUndersampler() class.</p><p>I will leave both of these exercises to the avid reader.</p><p><strong>References</strong></p><ul><li>Brownlee, J. (2020) SMOTE for Imbalanced Classification with Python. <a href="https://machinelearningmastery.com/smote-oversampling-for-imbalanced-classification/">https://machinelearningmastery.com/smote-oversampling-for-imbalanced-classification/</a></li><li>Cappelli, P. (2019). Your Approach to Hiring Is All Wrong. Harvard Business Review (HBR). Retrieved from<a href="https://hbr.org/2019/05/recruiting"> https://hbr.org/2019/05/recruiting</a></li><li>Korneev, E. (2019). Augmenting categorical datasets with synthetic data for machine learning. Retrieved from<a href="https://towardsdatascience.com/augmenting-categorical-datasets-with-synthetic-data-for-machine-learning-a25095d6d7c8"> https://towardsdatascience.com/augmenting-categorical-datasets-with-synthetic-data-for-machine-learning-a25095d6d7c8</a></li><li>Subhash, P. (2017). IBM HR Analytics Employee Attrition &amp; Performance, Version 1. Retrieved from <a href="https://www.kaggle.com/pavansubhasht/ibm-hr-analytics-attrition-dataset/metadata">https://www.kaggle.com/pavansubhasht/ibm-hr-analytics-attrition-dataset/metadata</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a26337a4943" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Principal Component Analysis for Logistic Regression with scikit-learn]]></title>
            <link>https://medium.com/@alepukhova526/principal-component-analysis-for-logistic-regression-with-scikit-learn-869b04b2f923?source=rss-d61aa9d3a4a9------2</link>
            <guid isPermaLink="false">https://medium.com/p/869b04b2f923</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Alexandra Pukhova]]></dc:creator>
            <pubDate>Fri, 20 Dec 2019 20:00:11 GMT</pubDate>
            <atom:updated>2019-12-20T20:01:13.341Z</atom:updated>
            <content:encoded><![CDATA[<p>The more features the merrier! (?) Not exactly. In machine learning, there are two main reasons why a greater number of features does not always work in your favor. Firstly, we may fall victim to the “curse of dimensionality” that results in exponentially decreasing sample density, i.e. sparser feature space (Xu, 2018). Secondly, the more features we have, the more storage space and computation time we require (the two resources that are relatively scarce for a layman computer scientist!).</p><p>In this article, I will explore the question of how one can use Principal Component Analysis (PCA) to speed up the running time of the algorithm, without sacrificing much of the model performance and sometimes even improving it. In part I, I will provide the context of the problem as well as discuss feature engineering that my group and I (hereafter referred to as “we”) have developed. Then, in part II, I will work through applying PCA to the dataset that contains new features and will run two classifiers on the PCA-transformed data.</p><p>*The full notebook with the code used in this post can be found <a href="https://github.com/AlexandraPukhova/Practical-Data-Science-Tutorial/blob/master/Modeling_notebook_Sasha.ipynb">here</a>.</p><p><strong>Dataset (Source: Kaggle)</strong></p><p><a href="https://www.kaggle.com/c/sf-crime">In this Kaggle competition</a>, we were asked to predict categories of crime in the City and County of San Francisco using SFPD (San Francisco Police Department) Crime Incident Reporting system data from 1/1/2003 to 5/13/2015. As the title suggests, it is a multinomial classification problem. Let’s first load the data:</p><pre><em># Load data</em><br>data = pd.read_csv(&#39;train_eng_full.csv&#39;)<br>test = pd.read_csv(&#39;test_eng_full.csv&#39;)</pre><p>and look at the main data characteristics:</p><ul><li><strong>Classes: </strong>39</li><li><strong>Total samples: </strong>~870k</li><li><strong>Dimensionality</strong> (prior to Feature Engineering): 6</li></ul><p>As can be seen above, there is originally a relatively small number of features, namely:</p><p>As can be seen above, there is originally a relatively small number of features, namely:</p><ul><li>Dates — timestamp of the crime incident;</li><li>Descript — detailed description of the crime incident;</li><li>DayOfWeek — the day of the week;</li><li>PdDistrict — the name of the Police Department District;</li><li>Resolution — how the crime incident was resolved;</li><li>Address — the approximate street address of the crime incident;</li><li>X, Y — the coordinates of the crime incident.</li></ul><p>The last feature is the Category of the crime incident — the target variable that is only available in the training dataset.</p><p>It seems like we have the most important information provided, but what else can we extract from it? As it turns out, A LOT more information! The coordinates were used to extract zip codes and obtain U.S. Census (demographic) data whereas the timestamps were used to obtain information such as the time of day of the crime, the season, and the concurrence with public holidays. Thus, after extensive temporal and spatial feature engineering, our group found itself with 80 features.</p><p>Now, how do we make an informed decision about which features to use? Of course, we will be tempted to try and feed our model all of the features, but what if having so many variables puts us at danger of overfitting the model (Brems, 2019)? There are two interconnected solutions to this problem: feature elimination and feature extraction.</p><p>Feature elimination, i.e. somewhat arbitrarily dropping features, is problematic, given that, we will not obtain any information from the eliminated variables at all (Brems, 2019). Feature extraction, however, seems to circumvent this problem by creating new independent variables from the combination of existing variables. Principal Component Analysis is one such algorithm for feature extraction. In the process, it also drops the least important variables (i.e. performs feature elimination) but retains the combinations of the most important variables.</p><p>Before applying PCA, let’s first split the training set into random train and validation subsets</p><pre><strong>from</strong> <strong>sklearn.model_selection</strong> <strong>import</strong> train_test_split</pre><pre>y = y_train<br>features = X_train<br><br>X_train, X_test, y_train, y_test = train_test_split(features, y, test_size=0.1)<br><br>print (X_train.shape, y_train.shape)<br>print (X_test.shape, y_test.shape)</pre><p>and scale our data, or simply put, standardize it. The reason why this step is necessary is that most of the time, our features can have quite a wide range of magnitudes. PCA attempts to “get features with maximum variance and variance happens to be higher for high-magnitude features (Asaithambi, 2017).” Therefore, in order not to skew PCA towards high magnitude features, we want to standardize all features first. Standardization implies replacing values with their Z scores so that the distribution of features has a mean value 0 and standard deviation (SD) of 1 (Asaithambi, 2017).</p><p>Here, I will use a standard scaler in order to achieve that.</p><pre><strong>from</strong> <strong>sklearn.preprocessing</strong> <strong>import</strong> StandardScaler</pre><pre>scaler = StandardScaler()<br>scaler.fit(X_train) <br><br>train_for_pca = scaler.transform(X_train)<br>test_for_pca = scaler.transform(X_test)</pre><p>Before I dive into PCA, let me first clarify the metric we were asked to use: logistic (or cross-entropy) loss. We can only compute log loss on two or more labels in the models that make probabilistic predictions on the labels rather than predict the exact labels (Sklearn, n.d.). For this reason, I am inclined to use classifiers such as Logistic Regression and some form of Artificial Neural Network (ANN), such as the Multi-layer Perceptron (MLP) classifier in my model that returns probabilistic predictions. MLP is similar to logistic regression with multiple layers.</p><pre>clfs = {&#39;Log_reg&#39;:LogisticRegression(penalty=&#39;l1&#39;,C=1.5,<br>solver=&#39;saga&#39;, max_iter=100, multi_class=&#39;multinomial&#39;, n_jobs=-1), <br>        &#39;MLP&#39;:MLPClassifier(solver=&#39;lbfgs&#39;,alpha=1e-5,                        hidden_layer_sizes=(5, 2),random_state=1)}</pre><p>In PCA, we can specify the number of components that we want to retain or, alternatively, the variance that we want to retain which will correspond to retaining a certain component count. Let’s first create a benchmark for both models where all the components are used (and where 100% of the variance is preserved).</p><pre><em># Creating a dataframe to store the results of experiments</em><br><br>benchmark_cols = [&#39;Variance retained&#39;,&#39;n_Components&#39;,&#39;Time(s)&#39;,&#39;Log_loss value&#39;]<br>lr_benchmark = pd.DataFrame(columns = benchmark_cols)<br>mlp_benchmark = pd.DataFrame(columns = benchmark_cols)</pre><p>Source code: (Kumar, 2018)</p><p>There is one problem that one is not guaranteed to face but should be aware of: sometimes, not all classes will be present in the training set, and various classifiers handle such cases differently. Whereas the MLP classifier recognizes the full set of possible classes regardless of their occurrence in the training set (i.e. for the ones it has not seen, the predicted probability will simply be set to 0), Logistic regression omits non-existent labels altogether. One way to fix it is to manually set the probabilities of the missing labels to 0:</p><pre>classes_not_trained = set(clf.classes_).symmetric_difference(all_classes)<br>    prob = clf.predict_proba(test_for_pca)<br><br>    <strong>for</strong> row <strong>in</strong> prob:<br>        prob_per_class = (list(zip(clf.classes_, prob))<br>                        + list(zip(classes_not_trained,repeat(0.))))</pre><p>Source code: (Foo, 2013)</p><p>The function that computes the base case log_loss for both models with all the components is the following:</p><pre><em># Benchmark at 100% variance anf full number of features.</em><br><strong>from</strong> <strong>itertools</strong> <strong>import</strong> repeat<br><br><strong>def</strong> set_benchmark(clf, train_for_pca, test_for_pca, y_train, y_test, labels, benchmark):<br>    <strong>global</strong> benchmark_cols<br>    <strong>global</strong> variance<br>    <br>    n_components = train_for_pca.shape[1]<br>    clf = clf<br>    all_classes = labels<br>    <br>    start = time.time()<br>    clf.fit(train_for_pca, y_train)<br>    end = time.time()<br>    timing = end-start<br>    <br>    classes_not_trained = set(clf.classes_).symmetric_difference(all_classes)<br>    prob = clf.predict_proba(test_for_pca)<br><br>    <strong>for</strong> row <strong>in</strong> prob:<br>        prob_per_class = (list(zip(clf.classes_, prob))<br>                        + list(zip(classes_not_trained,repeat(0.))))<br><br>    logloss = log_loss(y_test, prob, labels=all_classes) <br>    <br>    a = dict(zip(benchmark_cols,  [variance,n_components,timing,logloss]))<br>    benchmark = benchmark.append(a,ignore_index=<strong>True</strong>)<br><br>    print(benchmark)</pre><p>Source: (Kumar, 2018)</p><p>If we run the set_benchmark, we get the following results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J_YnzI39aw-w0NejL-hpRQ.png" /></figure><p>Now that we know the base case performance of each model, let’s apply PCA and see how the performance, as well as the algorithm execution time, vary with different levels of preserved variance. The way we can vary this component is to pass different variances to the n_components argument. If the value passed lies between 0 and 1, the algorithm “selects the number of components such that the amount of variance that needs to be explained is greater than the percentage specified by n_components (Sklearn, n.d.).”</p><p>Most of the code is repeated from the benchmark-setting function above.</p><pre><strong>def</strong> pca(clf,variance,train_for_pca,y_train,test_for_pca,y_test,labels):<br>    pca = PCA(variance)<br>    pca.fit(train_for_pca)<br>    n_components = pca.n_components_<br>    train_for_pca = pca.transform(train_for_pca)<br>    test_for_pca = pca.transform(test_for_pca)<br>    <br>    clf = clf<br>    all_classes = labels<br>    start = time.time()<br>    clf.fit(train_for_pca, y_train)<br>    end =  time.time()<br>    timing = end-start   <br><br>    <em># Determine the classes not present in the training set</em><br>    classes_not_trained = set(clf.classes_).symmetric_difference(all_classes)<br>    prob = clf.predict_proba(test_for_pca)<br>    <br>    <strong>for</strong> row <strong>in</strong> prob:<br>        prob_per_class = (list(zip(clf.classes_, prob))<br>                        + list(zip(classes_not_trained,repeat(0.))))<br><br>    logloss = log_loss(y_test, prob, labels=all_classes) <br>    <br>    <strong>return</strong>([variance,n_components,timing,logloss])</pre><p>The following function adds the results for different retained variances to the table:</p><pre><strong>def</strong> add_to_table_lr(pca):<br>    <strong>global</strong> lr_benchmark<br>    <strong>global</strong> benchmark_cols<br>    a = dict(zip(benchmark_cols,pca))<br>    lr_benchmark = lr_benchmark.append(a,ignore_index=<strong>True</strong>)</pre><p>An identical function exists for MLP, see Github (Modeling_notebook_Sasha) for reference.</p><p>We obtain the following results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q-pvAa9q7lUG4fWL-QWaDA.png" /></figure><p>Now that we have the relationships between the performance metric and the different variables in a tabular form, we can plot such relationships to see where we reached the optimal performance metric, using each classifier (reminder: we are trying to minimize log_loss).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0RmiKeMFcCsNReWGeWsBsQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XcS4WvcxibguYEinv16hcw.png" /></figure><p>As we can see above, even though PCA did not help to improve the overall performance of each model, it showed how the reduction in the number of components improved the running time dramatically, especially for Logistic Regression.</p><p><strong>Conclusion</strong></p><p>Models are sensitive to which format and dimensionality of data we feed them. Some models are designed to work with highly sparse feature states (read: SVM) and others, like Logistic Regression, work better with lower dimensionality. In the case of the San Francisco Crime classification challenge, we were working with classification models that predicted probabilistic outcomes and used log_loss as a performance metric that we were trying to minimize. PCA helped improve the run time for both models, however, in the case of MLP, it resulted in a smaller decrease in performance.</p><p><strong>Future improvements</strong></p><p>Instead of PCA, we could try to apply Linear Discriminant Analysis (LDA). The difference between the two is that LDA is supervised and PCA is not. LDA is also used for dimensionality reduction.</p><p><strong>References</strong></p><ul><li>Asaithambi, S. (2017, December 22). Why, How and When to Scale your Features. Retrieved from <a href="https://medium.com/greyatom/why-how-and-when-to-scale-your-features-4b30ab09db5e.">https://medium.com/greyatom/why-how-and-when-to-scale-your-features-4b30ab09db5e.</a></li><li>Brems, M. (2019, June 10). A One-Stop Shop for Principal Component Analysis. Retrieved from <a href="https://towardsdatascience.com/a-one-stop-shop-for-principal-component-analysis-5582fb7e0a9c.">https://towardsdatascience.com/a-one-stop-shop-for-principal-component-analysis-5582fb7e0a9c.</a></li><li>Foo, F. (2013). Training a sklearn LogisticRegression classifier without all possible labels. Retrieved from <a href="https://stackoverflow.com/questions/15034664/training-a-sklearn-logisticregression-classifier-without-all-possible-labels.">https://stackoverflow.com/questions/15034664/training-a-sklearn-logisticregression-classifier-without-all-possible-labels.</a></li><li>Kumar, A. (2018, September 20). Logistic Regression with PCA Algorithm — A Guide with Example. Retrieved from <a href="https://acadgild.com/blog/logistic-regression-with-pca-algorithm.">https://acadgild.com/blog/logistic-regression-with-pca-algorithm.</a></li><li>San Francisco Crime Classification. (n.d.). Retrieved from <a href="https://www.kaggle.com/c/sf-crime.">https://www.kaggle.com/c/sf-crime.</a></li><li>sklearn.decomposition.PCA. (n.d.). Retrieved from <a href="https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html.">https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html.</a></li><li>sklearn.metrics.log_loss. (n.d.). Retrieved from <a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.log_loss.html.">https://scikit-learn.org/stable/modules/generated/sklearn.metrics.log_loss.html.</a></li><li>Xu, C. (2018, June 18). Why is Dimensionality Reduction so Important? Retrieved from <a href="https://medium.com/@cxu24/why-dimensionality-reduction-is-important-dd60b5611543.">https://medium.com/@cxu24/why-dimensionality-reduction-is-important-dd60b5611543.</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=869b04b2f923" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>