<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rakesh Ravi on Medium]]></title>
        <description><![CDATA[Stories by Rakesh Ravi on Medium]]></description>
        <link>https://medium.com/@rakeshravi_14055?source=rss-77f5576baa39------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*drxd2SX_UMqIizaDchiM4A.png</url>
            <title>Stories by Rakesh Ravi on Medium</title>
            <link>https://medium.com/@rakeshravi_14055?source=rss-77f5576baa39------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 03:04:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rakeshravi_14055/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Just how good is Novak Djokovic?]]></title>
            <link>https://medium.com/data-science/just-how-good-is-novak-djokovic-401ce07c7ef3?source=rss-77f5576baa39------2</link>
            <guid isPermaLink="false">https://medium.com/p/401ce07c7ef3</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[sports]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[tennis]]></category>
            <dc:creator><![CDATA[Rakesh Ravi]]></dc:creator>
            <pubDate>Sun, 10 Feb 2019 17:32:00 GMT</pubDate>
            <atom:updated>2019-02-12T03:44:29.359Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Data Analyst-Tennis Player breakdown of the events that conspired at the Australian Open Finals 2019</h4><p>This year’s Australian Open Finals was a rematch of the finals of the tournament from the year 2012 between Novak Djokovic and Rafael Nadal. The match from 2012 had all the ingredients to be considered one of the greatest matches in the history of the game. The contest had everything from long grueling rallies, supreme defensive play patterns, winning shots that were from another planet and a display of the highest order of mental grit. The match provided a new definition of suffering on the tennis court and the players could barely stand during the presentation ceremony.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n-GdamkGglBTQT-9LZDosg.jpeg" /><figcaption><a href="https://scroll.in/field/910864/watch-tireless-djokovic-downs-nadal-in-a-near-six-hour-epic-to-win-2012-australian-open">Picture Courtesy</a></figcaption></figure><p>The 2012 AO final was a fairly even match up until the fifth set where Nadal was able to break Djokovic early in the set and was expected to cruise to a routine five-set victory. Djokovic, being the unrelenting fighter, continued to apply pressure on the Nadal serve and held his serves in a decisive fashion to keep himself relevant in the match. In the sixth game of the fifth set, Nadal missed a routine backhand down the line to an open court which proved to be pivotal and brought Novak back into the match. (Video Link Below)</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FXDWNrvXL9Gs%3Fstart%3D18749%26feature%3Doembed%26start%3D18749&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DXDWNrvXL9Gs&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FXDWNrvXL9Gs%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/e98a42f4b4f42d06a162d515e52882f0/href">https://medium.com/media/e98a42f4b4f42d06a162d515e52882f0/href</a></iframe><p>The rest is history as Djokovic found a way to break Nadal’s serve and win his third Australian Open title. The 2019 Australian Open final was expected to be another titanic battle between the two legends but no one will recall this match as being a close contest. Djokovic decisively got the better of Nadal in three set beatdown of a match that lasted little over the 2 hour mark to claim his seventh Australian Open title.</p><h3><strong>Neo-Nadal</strong></h3><p>To understand just how good Djokovic was in the game, its important to understand how good Nadal was playing over the last two weeks. Nadal did not drop a single set and bulldozed his way through some notable contestants en route to the finals. In fact, Nadal looked like he was in the form of his life before entering the finals.</p><p><strong>Serve</strong></p><p>After an ankle surgery in November and a plethora of knee injuries over the course of his career, Nadal decided to tweak his service motion. The modified service motion was intended to pack more punch on the shot and help Nadal be more aggressive and shorten the points in order to preserve his body.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*056s22-aLgpUaHTUZQTyzw.png" /><figcaption>Rafael Nadal’s Serve Stats</figcaption></figure><p>In the three departments that matter (first serve percentage, first serve points won and second serve points won), Nadal seems to have bested his career best stats in winning points off of serve. This resulted in Nadal holding 92.6% of all the service games which is also a new a career high. In addition to that, the first serve is now 5 mph faster than his serve last year. The effectiveness of the serve has helped him keep his matches short and stay on the driving seat throughout.</p><p><strong>Groundstrokes and Volley</strong></p><p>Just a few years ago, Nadal was known for his unbelievable retrieval skills, returns of serves from 10 feet behind the baseline and the ability to restart a point multiple times. In the last 3 years in particular, Nadal has switched up his tactics and incorporated offensive play patterns in his game. This was in full display in the 2017 Australian Open final where Nadal went head to head with Federer with onslaught of offensive plays. After the recent ankle surgery, Nadal shortened his backswing and started coming in flatter on the shots. It’s no secret that Nadal has an amazing volley (<a href="https://www.youtube.com/watch?v=sfUuQZT-oEc">check this out</a>)and after incorporating that into his singles game, it has only become stronger over the past few years. This has helped him to start the points on an offensive note and finish the points quickly.</p><p><strong>Dominance</strong></p><p>Nadal started the Australian open 2019 with a straight set win over James Duckworth from Australia. The scoreline definitely indicates that Nadal was rusty from a lack of matchplay but his serve was broken twice during the match. From the second round on, his serve really start to click and he dominated his way to the finals without dropping serve and a set. He convincingly picked apart Stefano Tsitsipas (Roger Federer’s assailant)in the semifinals.</p><h3><strong>Peak-Djokovic</strong></h3><p>Djokovic’s elbow injury threw a spanner in the works of his domination which kept him out of contention in grands slams for the better part of 2017 and 2018 before he broke his grand slam draught by winning Wimbledon 2018. While the fans felt like his resurgence from the injury was complete but few people were hopeful of him dominating the way he did in the finals of the Australian Open 2019.</p><p><strong>Tweaked Service Motion</strong></p><p>After pulling out of the second half of the 2017, team Djokovic came up with a serve motion that was lighter on his elbows. The elbow injury had made Djokovic’s serve inconsistent, unreliable and lacked the punch to bully opponents around. The tweaked service motion didnt seem to change Djokovic’s fortunes and had to undergo surgery before competing without any pain. The service motion that Djokovic displayed over the first half of the 2018 season was inspired by Agassi’s service motion. Djokovic goes on a tear after the US hardcourt summer to fire Agassi and Stepanek and trash his new service motion. Rehiring Marian Vajda saw his serve drastically improve and get back to 2016-ish standards. I will let the numbers speak for itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kIPEpWd6FEgV3xVBzrWHPQ.png" /></figure><p>Federer and Djokovic are not known for the most powerful serves but are known for hitting the right spots at the right time. Djokovic’s serves are hitting the right spots more often than ever. 8.3% of all his serves turned out to be aces and he has never done better in the ace department before. The next statistic to check will be the first serve percentage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kTNlUnsjgucqhJB9uH87GQ.png" /></figure><p>Djokovic’s first serve percentage was the reason why he was able to obliterate Nadal giving him absolutely no looks on his serve games. The first serve was firing on all cylinders and was irritatingly consistent for all of his opponents. Despite having a terrible start to the season and losing to players ranked outside of the top 100, Novak was able to get his first serve percentage to a respectable number during the second part of the season.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HgBOcCapb3tCCVdSL9c7Lw.png" /></figure><p>If you thought the serve has just improved in accuracy and precision, you would be wrong. The effectiveness of the serve is undeniable when you take a look at the numbers. We have established that the number of first serves are higher than it has ever been and each serve is paying more dividends than it ever has. He recorded a career high first serve points won this year and this is including a dismal performance in the Doha championships. There is a saying that “you are only as strong as your weakest link” which brings us to the second serve and Djokovic has improved considerably improved the second serve nearly 10 percentage points over his career average. In the past, especially 2011, Djokovic’s physicality used to take over after his second serve was neutralized but this should not be true in 2019 because he has gotten older and consequently weaker than his former self. Miraculously, he has been better than ever at long baseline rallies and his groundstrokes have become more penetrating than before.</p><p><strong>Return Game</strong></p><p>I can sit and rave for a whole day on how good Novak’s returns have been in the past but can it get any better?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_VhEidLJibKCNeMW87Dfbg.png" /></figure><p>The number of return points won by Novak was certainly not high as it was in 2011 win when he went on a tear and nearly shutdown Roger and Rafa’s career. In 2011, he had to take on Nadal and Federer who were in pretty good form and as a result, the return actually needed to be that good. Its surprising that Novak is still able to compete with his previous record set in 2011 and equalled his 2016 self where he ended the season with 2 grand slams. Roger’s return of serve is at a career average of 40% which is not too bad for a player completely out of his prime but what separates Novak from Roger are the percentage of break points won. Roger, during the Australian Open, won 14.7% of the breakpoints and Novak won 39.4% of the breakpoints that he earned. Check below how this year’s breakpoint stats stacks up with the other years where he was playing sublime tennis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*temj2u8gtlpQDDbqj9xhZQ.png" /></figure><p><strong>Ground “Body Blow” Strokes</strong></p><p>Unfortunately, ATP does not provide enough data to assess someone’s forehand and backhand game but on a qualitative level, the forehand and backhand was firing on all cylinders. After a disappointing loss to Roberto Bautista Agut where Novak was playing so passively and looked like he couldnt hit through any of his opponents (Marton Fucsovics, Nikoloz Basilashvilli). He picked up his game nicely from Round 1 of the Australian Open and improved every drastically after every Round of play. In the semifinals against Pouille, he committed 5 unforced errors and in the finals against Nadal, he committed 9 unforced errors which is just ridiculous.</p><p>His court position was very offensive and stepped in to flatten his backhand multiple times to neutralize Nadal’s loopy cross-court forehand. Take a look at the shot in the link below.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FqgRCBINX6N0%3Fstart%3D42%26feature%3Doembed%26start%3D42&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DqgRCBINX6N0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FqgRCBINX6N0%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/bef7dd37f5dcb19097c9467a2af0b6f1/href">https://medium.com/media/bef7dd37f5dcb19097c9467a2af0b6f1/href</a></iframe><p>Clearly, the changes that Marion has incorporated has taken Novak’s game to the next level and without any real competition (Unless Nadal and Federer are able to get back to their 2017 form), he might just go on to win 2 or maybe 3 grand slams this year.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=401ce07c7ef3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/just-how-good-is-novak-djokovic-401ce07c7ef3">Just how good is Novak Djokovic?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[One-Hot Encoding is making your  Tree-Based Ensembles worse, here’s why?]]></title>
            <link>https://medium.com/data-science/one-hot-encoding-is-making-your-tree-based-ensembles-worse-heres-why-d64b282b5769?source=rss-77f5576baa39------2</link>
            <guid isPermaLink="false">https://medium.com/p/d64b282b5769</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[random-forest]]></category>
            <category><![CDATA[kaggle]]></category>
            <dc:creator><![CDATA[Rakesh Ravi]]></dc:creator>
            <pubDate>Fri, 11 Jan 2019 06:19:54 GMT</pubDate>
            <atom:updated>2022-07-24T16:08:50.916Z</atom:updated>
            <content:encoded><![CDATA[<h3>One-Hot Encoding is making your Tree-Based Ensembles worse, here’s why?</h3><h4>Optimizing Tree-Based Models</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*V3CjG5g7-1Jcpjg0" /><figcaption>Photo by <a href="https://unsplash.com/@maritafox?utm_source=medium&amp;utm_medium=referral">Marita Kavelashvili</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><blockquote><em>I am a Master’s Student in Data Science at the University of Virginia and I am working on lots of Machine Learning Projects. Every week, I will be addressing one problem that I faced when working on a project along with a practical solution.</em></blockquote><p>Most real-world datasets are a mix of categorical and continuous variables. While continuous variables fit into all machine learning models quite easily, the implementation of categorical variables is vastly different in different models and different programming languages. For instance, R deals with categories using a variable type called factor which elegantly fits all models without an issue and Python would require you to convert the categories into label encoding for the models to work. Python is my choice of programming language and I routinely perform some sort of encoding for categorical variables. In general, one hot encoding provides better resolution of the data for the model and most models end up performing better. It turns out this is not true for all models and to my surprise, random forest performed consistently worse for datasets with high cardinality categorical variables.</p><p><strong>Test Setup</strong></p><p>Being a Kobe Bryant fan, I decided to use his <a href="https://www.kaggle.com/c/kobe-bryant-shot-selection">shot selection data</a> to build a model to predict whether a shot given the circumstances would land in or not. Based on feature importance analysis, it was clear that the two variables describing the type of shot were the most important in predicting the outcome of a given shot and both variables were categorical. I ran random forest on the dataset with label encoding (assuming that there was an order) and with one-hot encoding and the outcome was nearly the same thing. In fact, it didnt seem to matter whether there was any order.</p><p><strong>How did the models fare?</strong></p><p>In addition to accuracy of prediction, I have also used the logarithmic loss of the model for comparison as that was the evaluation criteria for the Kaggle competition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*vWhYH9KaUeDhjNB6cOU_sw.png" /><figcaption>Snapshot of Performance</figcaption></figure><p>We would normally expect one-hot encoding to improve the model but as the screenshot suggests, the model with one-hot encoding performing significantly worse than the model without it. One-hot encoding contributed to a decrease of 0.157 units of logarithmic loss. Kaggle competition winners are decided in slightest of margins and an offset of 0.05 could be the difference between being in the top 75% and top 25% on the leaderboard. Now that we have established one-hot encoding is making the models worse, let’s examine the reasons behind why this happens.</p><p><strong>Why does this happen?</strong></p><p>To understand this, let’s get into the inner dynamics of tree-bases models. For every tree-based algorithm, there is a sub-algorithm that is used to split the dataset into two bins based on a feature and a value. The splitting algorithm considers all possible splits (based on all features and all possibles values for each feature) and finds the most optimum split based on a criterion. We will not get into details of the criterion (there are multiple ways to do this) but qualitatively speaking, the criterion helps the sub-algorithm select the split that minimizes the impurity of bins. Purity is a proxy for the number of positive or negative samples in a particular bin in a classification problem. I found a very useful visualization that can helped me <a href="http://www.r2d3.us/visual-intro-to-machine-learning-part-1/">grasp</a> this concept.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rJldhOQ8qofb-UsvEDwAdg.png" /><figcaption>Dense Decision Tree (Model without One Hot Encoding)</figcaption></figure><p>If a continuous variable is chosen for a split, then there would be a number of choices of values on which a tree can split and in most cases, the tree can grow in both directions. The resulting tree from a dataset containing a majority of continuous variables that would look like something in the figure to the left.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*waMbIQifR03o_1hHzNYgbw.png" /><figcaption>Sparse Decision Tree (Model with One Hot Encoding)</figcaption></figure><p>Categorical variables are naturally disadvantaged in this case and have only a few options for splitting which results in very sparse decision trees. The situation gets worse in variables that have a small number of levels and one-hot encoding falls in this category with just two levels. The trees generally tend to grow in one direction because at every split of a categorical variable there are only two values (0 or 1). The tree grows in the direction of zeroes in the dummy variables. If that didn’t make sense, follow the example below closely.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/534/1*UXFyRYdfpCYwfkAuMcDjMg.png" /><figcaption>Dataset A with Dummy Variables</figcaption></figure><p>For instance, consider the following dataset which contains dummy variables for<br>categorical variables that have levels A, B, C and D. If we were to fit a decision tree to this data just to understand the splits, it would resemble the following.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*jOMNT-nHwABGVchKX0Pi3Q.jpeg" /><figcaption>Decision Tree for Dataset A</figcaption></figure><p>In conclusion, if we have a categorical variable with q levels, the tree has to choose from ((2^q/2)-1) splits. For a dummy variable, there is only one possible split and this induces sparsity.</p><p><strong>Effect on Model Performance</strong></p><p>Now that we have understood why the decision trees for datasets with dummy variable look like the above figure, we can delve into understanding how this affects prediction accuracy and other performance metrics.</p><ol><li>By one-hot encoding a categorical variable, we are inducing sparsity into the dataset which is undesirable.</li><li>From the splitting algorithm’s point of view, all the dummy variables are independent. If the tree decides to make a split on one of the dummy variables, the gain in purity per split is very marginal. As a result, the tree is very unlikely to select one of the dummy variables closer to the root.</li></ol><p>One way to verify this is to check the feature importance for both models and see which features come out on top.</p><p><strong>Feature Importance</strong></p><p>In spite of delivering better performance, the tree based models have a slightly tarnished reputation primarily because of its interpretability which arises due to the way in which the feature importance is calculated. Feature importance in most tree-ensembles is calculated based an importance score. The importance score is a measure of how often the feature was selected for splitting and how much gain in purity was achieved as a result of the selection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/462/1*aMaOMQ0bIt9txo_YMcSiTQ.png" /></figure><p>The most important feature is the action_type which is a high cardinality categorical variable and clearly much more important than the ones preceding it. To provide some context, I had one-hot encoded action_type and combined_shot_type which were both high cardinality categorical variable. Obviously, dummy variables increase the dimensionality of the dataset and the curse of dimensionality comes into play and produces the chart below for feature importance. This is clearly not interpretable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/1*VT1vwxH9k_Ra1B6JkMwSJw.png" /></figure><p>To take a magnified view, I decided to look at the top 15 features which seemed to influence the prediction the most.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/481/1*gMBpcUHN0LFCYr6JQ95_dg.png" /></figure><p>When we examine the chart above, it’s clear that only one dummy variable is featured in the top 15 and that too in the 14th position which validates our hypothesis of variable selection for splitting. One-hot encoding has also obscured the order of importance of features that weren’t involved in the encoding and this makes the model inefficient.</p><p><a href="https://github.com/rakeshravidata/Kobe-Shot-Selection-Prediction-">Link to Github</a></p><blockquote><strong><em>TLDR</em> <br></strong>One-hot encoding categorical variables with high cardinality can cause inefficiency in tree-based ensembles. Continuous variables will be given more importance than the dummy variables by the algorithm which will obscure the order of feature importance resulting in poorer performance.</blockquote><p><strong>Extra Reading</strong></p><ol><li>Visualization — <a href="http://www.r2d3.us/visual-intro-to-machine-learning-part-1/">http://www.r2d3.us/visual-intro-to-machine-learning-part-1/</a></li><li>Textbook — <a href="https://web.stanford.edu/~hastie/ElemStatLearn//">https://web.stanford.edu/~hastie/ElemStatLearn//</a></li><li>Engaging Discussion — <a href="https://www.kaggle.com/c/zillow-prize-1/discussion/38793">https://www.kaggle.com/c/zillow-prize-1/discussion/38793</a></li><li>Summary of all types of Encoding — <a href="https://medium.com/data-design/visiting-categorical-features-and-encoding-in-decision-trees-53400fa65931">https://medium.com/data-design/visiting-categorical-features-and-encoding-in-decision-trees-53400fa65931</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d64b282b5769" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/one-hot-encoding-is-making-your-tree-based-ensembles-worse-heres-why-d64b282b5769">One-Hot Encoding is making your  Tree-Based Ensembles worse, here’s why?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What do you do when : All the variables in your dataset are categorical and have to build a model?]]></title>
            <link>https://medium.com/@rakeshravi_14055/what-do-you-do-when-all-the-variables-in-your-dataset-are-categorical-and-have-to-build-a-model-db555a6c0be7?source=rss-77f5576baa39------2</link>
            <guid isPermaLink="false">https://medium.com/p/db555a6c0be7</guid>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[logistic-regression]]></category>
            <category><![CDATA[beginners-guide]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Rakesh Ravi]]></dc:creator>
            <pubDate>Sun, 23 Dec 2018 22:00:37 GMT</pubDate>
            <atom:updated>2018-12-31T05:16:52.905Z</atom:updated>
            <content:encoded><![CDATA[<h3>A Practical Approach to Dealing With Categorical Variables During Modelling</h3><blockquote><em>I am a Master’s Student in Data Science at the University of Virginia and I am working on lots of Machine Learning Projects. Every week, I am going to address one problem that I faced when working on a project along with a practical solution.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dmh91HqnhxnMtF3tt2Z32A.jpeg" /></figure><p>All Data Analysts have reached a point in solving a problem where they end up with a dataset that has nothing but categorical variables. You are probably thinking that this isnt a big issue because you can just one-hot encode them (If you have not heard of one-hot encoding — read <a href="https://medium.com/@michaeldelsole/what-is-one-hot-encoding-and-how-to-do-it-f0ae272f1179">this</a>) or create label encoders. But what do you when you have a dataset with a large number of categorical variables and your local machine is not a supercomputer?</p><p><strong>Curse of Dimensionality</strong></p><p>I have been working on a project to predict flight delays using supervising learning methods. The dataset that I used was from Kaggle (click <a href="https://www.kaggle.com/usdot/flight-delays/ke%E2%80%A6">here</a> to access the dataset) and was acquired from the US Department of Transportation for the year 2015. Each row in the dataset is an individual flight, and contains information on the airline, airports (name, location, origin airport, destination airport) and flight level information such as time schedule (day,month,year), flight distance, departure time and delay, etc. Our dataset contains around 6 million flights for 14 different airlines amongst 322 airports. As these are records that have happened in the past, most columns or features are expected to be categorical in nature.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*shu_QNp6umKtZvKX80tblQ.png" /><figcaption>Snapshot of the Dataset</figcaption></figure><p>To give you a rough idea of the scale, there are 322 different airport codes in both airport fields (Origin and destination), nearly 14 airlines, 12 months and nearly 30 days in each month. If you were to use one-hot encoding for the problem, the number of columns in the dataset will increase to 522,567,360. Most classification models require a high amount of computation power to train on 500 million rows which is generally available in cloud computing services like AWS or GCP.</p><p><strong>But, what are my options now?</strong></p><p>In order to reduce dimensionality, the variable with the highest number of levels can be grouped into clusters. In the case of my project, the origin and destination airport have the highest number of levels and can be grouped into clusters. But clustering will reduce resolution of the data and the model might not learn the nuances between different levels of the variable. As a result, the model will not perform as well as we would expect and will have a lower AUC which is undesirable.</p><p>Instead of training the model on the whole dataset, what if you trained a separate model for each airport? For each airport, the number of columns reduces to about 1 million and the average number of rows reduces to a manageable 2000. This is a huge improvement over 6 Million Rows and 550 Million Columns.</p><p><strong>Steps of Execution</strong></p><p><strong>Step 1</strong>: Subset the data based on the categorical variable that has the highest number of levels. In this case, the airport of origin can be used to subset the data into 322 different chunks of data.</p><p><strong>Step 2</strong>: Perform train-test splits for each chunk of data separately and train the classifier on the train split of each chunk.</p><p><strong>Step 3</strong>: Record predictions on the test split of each chunk with their corresponding classifiers. Combine all predictions into a single dataset.</p><p><strong>Step 4</strong>: Perform hyper-parametric tuning for the whole dataset or for individual classifiers depending on the number of levels in each category.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/1*fZUExkhlDojlMDwwVi7Z1A.png" /><figcaption>Layout for Training Multiple Classifiers</figcaption></figure><p><strong>Result</strong></p><p>The models, although there were 322 of them, ran on my MacBook Pro with a 8 GB of RAM without any problems. In addition, the Area Under Curve for the overall prediction was 0.65 which is a tad lesser than a model for the whole dataset (Run on AWS) which yielded 0.68. I tried using random forest, SVM and logistic regression and saw the best results from logisitic regression.</p><p>This can be easily applied to any problem where the regressor that is the most predictive of the response is a categorical variable with a large number of levels.</p><p>Click <a href="https://github.com/rakeshravidata/kaggle_flight_delay_prediction">here</a> to access the github repo.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=db555a6c0be7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ethics of Big Data in Cyber Security]]></title>
            <link>https://medium.com/@rakeshravi_14055/ethics-of-big-data-in-cyber-security-378c5cb08f9b?source=rss-77f5576baa39------2</link>
            <guid isPermaLink="false">https://medium.com/p/378c5cb08f9b</guid>
            <category><![CDATA[consent]]></category>
            <category><![CDATA[ethics]]></category>
            <category><![CDATA[online-privacy]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <dc:creator><![CDATA[Rakesh Ravi]]></dc:creator>
            <pubDate>Wed, 12 Dec 2018 22:09:08 GMT</pubDate>
            <atom:updated>2018-12-12T22:09:08.528Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/560/0*zVYvHSJYCIreI2Ln.jpg" /></figure><p><strong>Background Information</strong></p><p>I am a Master’s in Data Science student at the University of Virginia and I am working on a project with the School of Engineering that involves detecting cyber-crime activity within the University network using a semi-supervised machine learning algorithm. The dataset for the project is a set of data points including personally identifiable information such as IP addresses of the users that interact with the University of Virginia network. The goal is to train a machine learning algorithm to detect the patterns that help in deciding whether an IP address is malicious or not. The final outcome of the project is a blacklist of the IP addresses which will be used by the cyber security team to prevent cyber-attacks in the future. An important sub-goal of the project is minimizing the instances of false positives where an IP address is diagnosed as malicious but is actually benign. While the project is intended to serve the common good of the university, it involves some serious ethical trade-offs and possibly some collateral damage. As the data scientist working on building this algorithm, it’s my responsibility to ensure that I address the ethical concerns around online privacy and data security</p><p><strong>Ethical Issues</strong></p><p>Capturing personally identifiable information such as IP addresses is a necessity in pinpointing malicious activity with a substantial level of granularity. While this is important for the project as it will increase the model’s accuracy, there are risks involved in warehousing this information which could potentially invite more attacks or can be used for purposes that the user may or may not have consented to. When it comes to security of data, we do have guidelines from the Federal Trade Commission but are they good enough? Over the last few years, we have had massive data breaches in tech organizations like Equifax, Yahoo, eBay, Facebook etc., that invest a huge amount in money in putting together a state of the art cyber security infrastructure. Clearly, in the field of cyber security, no one or organization can do enough to protect themselves from an attack or a security breach. The bigger question is, is it worth storing personal information of individuals in an attempt to protect them from cyber-security threats when we do not have a complete understanding of possible threat?</p><p>One of the key aspects of the project involves characterizing IP addresses based on the patterns that machine learning algorithm detects before diagnosing them as malicious. Mathematically speaking, some IP addresses will be categorized as malicious based on a benchmarked statistic derived from a sample of the dataset. Based on this, a blacklist is created and deployed to the University of Virginia. There is a small chance that the sample (used for computing the benchmark statistic) failed to provide an accurate representation of the population and ended up denying access to users that were not engaging in malicious activity. Algorithms make decisions and resolve trade-offs in ways that are different from human and are often limited by the data that is fed to them. While the algorithm might be efficient at neutralizing cyber threats but in the blatant pursuit of accuracy of prediction, it can lose sight of the inherent discrimination it might be propagating and can be stuck in feedback loops. As a result of algorithmic bias or generalized and sometimes biased grouping of users by the algorithm, is there more potential harm than potential good?</p><p>For the purpose of building a database of possible malicious activity that could intrude the University’s network, we set up booby traps in the networks called honeypots to lure intruders or attackers and track their movement through the network. As soon as the the hackers are trapped in honeypot, everyone in the cybersecurity team at UVa is alerted and they could potentially stop them at their tracks. However, in order to get a complete understanding of how botnets or hackers operate, we let them perform operations which could involve stealing personal information. Tracking the hackers reveal invaluable insights into attacker’s techniques and network vulnerabilities and it also helps make the intrusion detection systems in the network better. The use of honeypots is a very controversial topic and although perfectly legal, how ethical is it to lure someone into stealing an object?</p><p>Like most Data Science problems, there is not enough information for us to predict malicious activity with complete certainty. An IP address (A section of the network) is considered the smallest unit of classification for the algorithm because that is level of resolution of information the server is able to capture with the current technical setup. A more granular unit of classification would be individual computers but unfortunately, we do not have this information. A smaller unit of classification could potentially reduce the number of incidents of benign users losing access to the internet. As a result, this is a minor design flaw in the methodology adopted for the project. The School of Engineering at UVa believes that in spite of minor flaws in the data, it is worthwhile building an algorithm to combat cyberthreats. While this does not solve any of the issues mentioned above, this information in crucial in formulating potential solutions for the ethical dilemmas highlighted in this paper.</p><p><strong>Potential Solutions</strong></p><p>Data security and privacy is an important part of the capstone project and it is given paramount importance by the School of Engineering at UVa. One of the methods in which cybersecurity team at UVa is ensuring security of personal information is by anonymizing the records so that it is not possible to connect the data to an actual person. The anonymization is performed by using another algorithm that scrambles the IP address (PII) so that it is decipherable only to the cybersecurity team that has the key to the decryption. Although anonymization might seem like an elegant solution to the issue but it does not provide a clear solution to the underlying problem</p><p>Helen Nissenbaum, in her paper “Big Data’s End Run around Anonymity and Consent” argues that anonymization of PII makes them non-identifiable but does not make them unreachable. A recent study found that students suffering from depression could be identified from their internet browsing patterns. In the context of this project, there is a deeper implication of Nissenbaum’s comments. The algorithm, in question, characterizes users based on their activity on the UVa network and not on the basis of their IP address. The IP address happens to be just an identifier and removing or anonymizing it is not going to affect the performance of the algorithm. In fact, the features that are used to characterize users remain in the database and the algorithm can certainly discriminate against a certain group without knowing any of their PII. If the algorithm that I am working on can potentially cause harm, there is no telling what cyber terrorists could if they get their hands on this information. The reason why anonymization was considered as a solution by the cyber security team is because most technological organizations treat data security and privacy as a legal compulsion rather than an ethical obligation. As per legal norms set by the governing body of cybersecurity, anonymization counts as meeting the required standards for data security of IP addresses. In addition to the anonymization, the School of Engineering should make it mandatory for all researchers that are interested in working with this data to go through a review from an Independent Ethics Committee or Research Ethics Board. This will ensure that the impact on data privacy and security is measured before commencing research on the data.</p><p>Algorithmic bias has been cited in cases ranging from election outcomes to the spread of online hate speech. Albeit this is an easier problem to comprehend, it is much harder problem to solve. In the field of Data Science, algorithmic bias is treated by ensuring that distribution between different groups of users or data is equitable and by constantly updating the algorithms with new information. On the surface, this may seem like a way out of algorithmic bias, but this works only in ideal scenarios where it is actually possible to have an equitable distribution in classes. In reality, there are only a handful of instances in the past where such techniques were successful to alleviating algorithmic bias. According to Sasha Eder, COO of NewtonX, there are three primary reasons why algorithmic bias is perpetuated. It is either due to diversity gaps in the dataset, personal preferences of the Data Analyst related to the definition of the outcome or due to the involvement of a sophisticated and uninterpretable algorithm. Diversity gaps and using a simpler algorithm can be solved by consulting an expert statistician but the inducing personal preferences (in the form of a benchmark statistic in the case of my capstone project) into the algorithm cannot be solved by a technologist. In my opinion, having an ethics committee of social scientists review such algorithms before they are deployed can help keep personal preferences at an arm’s length distance.</p><p>Honeypots remain a controversial topic and although they are generally seen as being a legal solution, not everyone sees them as ethical. Pelletier and Kabay, make the following statement, ‘As for entrapment of hackers using honeypot, although this is not a legal problem, this does not mean that the way a honeypot entices attackers is not unethical’. The stakeholders involved in this issue are researchers, malicious actors and the users who are a part of the network. Using honeypots benefits the researchers in studying malicious actors but could prove detrimental to the users. The primary argument against implementation of honeypots is that exposing network vulnerabilities in order to study a botnet specimen (attacker) is too much of a risk posed to the users belonging to the network. Little work has been done in this area to ensure that the damage inflicted due to the honeypot system is minimal or to find alternate approaches to study botnets. As researchers, while we are interested in learning about the techniques that a malicious actor might adopt, we need to respect those who we are trying to protect. We should communicate to the users our proposed research methods, help them understand the ramifications of implementing these methods and acquire their full consent. In addition to that, we should do our best as individuals to respect their autonomy and do everything in our power to minimize the harm they experience.</p><p><strong>Conclusion</strong></p><p>In my opinion, most ethical challenges faced in the field of technology and especially in the field of machine learning are due to the limited presence of social scientists. As the government agencies are unable to keep pace with the advancements in Silicon Valley, it falls on the tech organizations to ensure that they have an ethics review committee to perform risk assessment from an ethical perspective.</p><p>I have expert guidance from one of my professors for putting this together. If you are interested in working on such themes, I would highly recommend checking out his website <a href="https://www.samuellengen.net/">https://www.samuellengen.net</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=378c5cb08f9b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>